Neural structure learning with stochastic differential equations
Authors: Benjie Wang
Joel Jennings
Wenbo Gong
Venue: ICLR 2024 poster (2024)
Keywords: Structure Learning
Causal Discovery
Generative Model
Variational Inference
Differential Equation
Primary area: causal reasoning
TL;DR: We propose a novel structure learning method that leverages stochastic differential equations and variational inference to model continuous temporal process and infer posterior distributions over possible structures with theoretical guarantees.

Abstract: Discovering the underlying relationships among variables from temporal observations has been a longstanding challenge in numerous scientific disciplines, including biology, finance, and climate science. The dynamics of such systems are often best described using continuous-time stochastic processes. Unfortunately, most existing structure learning approaches assume that the underlying process evolves in discrete-time and/or observations occur at regular time intervals. These mismatched assumptions can often lead to incorrect learned structures and models. In this work, we introduce a novel structure learning method, SCOTCH, which combines neural stochastic differential equations (SDE) with variational inference to infer a posterior distribution over possible structures. This continuous-time approach can naturally handle both learning from and predicting observations at arbitrary time points. Theoretically, we establish sufficient conditions for an SDE and SCOTCH to be structurally identifiable, and prove its consistency under infinite data limits. Empirically, we demonstrate that our approach leads to improved structure learning performance on both synthetic and real-world datasets compared to relevant baselines under regular and irregular sampling intervals.

Published as a conference paper at ICLR 2024
NEURAL STRUCTURE LEARNING WITH STOCHASTIC DIFFERENTIAL EQUATIONS
Benjie Wang*
Joel Jennings DeepMind
Wenbo Gong" Microsoft Research Cambridge
University of California, Los Angeles
ABSTRACT
Discovering the underlying relationships among variables from temporal observa- tions has been longstanding challenge in numerous scientific disciplines, includ- ing biology, finance, and climate science. The dynamics of such systems often best described using continuous-time stochastic processes. Unfortunately, most existing structure learning approaches assume that the underlying process evolves in discrete-time and/or observations occur at regular time intervals. These mis- matched assumptions often lead to incorrect learned structures and models. In this work, we introduce a novel structure learning method, SCOTCH, which com- bines neural stochastic differential equations (SDE) with variational inference infer a posterior distribution over possible structures. This continuous saatinuous-tmee ap- proach can naturally handle both learning from and predicting observations arbitrary time points. Theoretically, we establish sufficient conditions for an SDE and SCOTCH to be structurally identifiable, and prove its consistency under infi- nite data limits. Empirically, demonstrate that approach leads improved structure learning performance on both synthetic and real-world datasets com- pared to relevant baselines under regular and irregular sampling intervals.
INTRODUCTION
Time-series data is ubiquitous in the real world, often comprising a series of data points recorded at varying time intervals. Understanding the underlying structures between variables associated with temporal processes is of paramount importance for numerous real-world applications (Spirtes al.
2000 Berzuini et al. 2012: Peters et al. 2017). ). Although randomised experiments are considered the gold standard for unveiling such relationships, they frequently hindered by factors such as cost and ethical concerns. Structure learning seeks to infer hidden structures from purely observa-
tional data, offering powerful approach for wide array of applications (Bellot et al. 2022 Löwe
et al.
2022 Runge 2018
Tank et al., (202T] Pamfil et al. 2020 Gong et al. 2022).
However, many existing structure learning methods for time series are discrete, assuming that the
underlying temporal processes are discretized in time and requiring uniform sampling intervals. Consequently, these models face limitations: (i) they may misrepresent the true underlying process when it is continuous, potentially leading to incorrect inferred relationships; and (ii) they struggle with handling irregular sampling intervals, which frequently arise many fields (Trapnell
et al. 2014 Qiu et al. 2017; Qian et al.) 2020) and climate science (Bracco et al. 201 8 Raia 2008).
To address these challenges, we introduce a novel framework, Structure learning with COntinuous- Time stoCHastic models (SCOTCH)) which employs stochastic differential equations (SDEs) for structure learning temporal processes. SCOTCH can naturally handle irregularly sampled time series and accurately represent and learn continuous-time processes. We make the key contributions:
1.
We introduce a novel latent Stochastic Differential Equation (SDE) formulation for mod- elling structure in continuous-time observational time-series data. To effectively train our
Eqqual contribution.
Correspondence benj jennjiewwaeeullle edu, werboogongem.crooooft.cco
Work done during an internship at Microsoft Research Cambridge.
ht : / / thub. com/mi croso http:///woooooooooooo//o////////
scotch

---

Published as a conference paper at ICLR 2024
proposed model, which we denote SCOTCH, we adapt the variational inference frame- work proposed in (Li et al. 2020 Tzen & Raginsky (019a) to approximate the posterior for both the underlying graph structure and the latent variables. We show that, in contrast to a prior approach using ordinary differential equations (ODEs) (Bellot et al. (022), our model is capable of accurately learning the underlying dynamics from trajectories exhibit- ing multimodality and with non non-Gaussian distribution.
2. We provide rigorous theoretical analysis to support our proposed methodology. Specif- ically, we prove that when SDEs are directly employed for modelling the observational process, the resulting SDEs are structurally identifiable under global Lipschitz and diago- nal noise assumptions. We also prove our model maintains structural identifiability under certain conditions, even when adopting the latent formulation; and that variational infer- ence, when integrated with the latent formulation, in the infinite data limit, can successfully recover the ground truth graph structure and mechanisms under specific assumptions.
3. Empirically, we conduct extensive experiments on both synthetic and real-world datasets showing that SCOTCH can improve upon existing methods on structure learning, including when the data is irregularly sampled.
2 PRELIMINARIES
In the rest of this paper, t, with Xr,a representing the th variable of the observation. A time series a set observations X = Y=I. where {ti)/ are the observation times. In the case where we have multiple (N) i..d. time series, we use (m) to indicate the nh time series.
X; E R to denote the D-dimensional observation vector time
Bayesian structure learning In structure learning, the aim is to infer the underlying graph repre- senting the relationships between variables from data. In the Bayesian approach, given time series data (X J==1. define joint distribution over graphs and data given by:
N
pG, x(N) = P()) II ( G)
(1)
where p(G) is the graph prior and p( p(X (n) ()) is the likelihood term. The goal then to compute the graph posterior p( ( (N)). However, analytic computation is intractable in high
dimensional settings. Therefore, variational inference (Zhang et al. (0018) and sampling methods (Welling& & Tch 2011 Gong et al. 2018: Annadani et al.) 2023) are commonly used for inference.
Structural equation models (SEMs) Given a time series X and graph G E (0, (0,1)DX DXD we can use SEMs describe the structural relationships between variables:
X1, = (Paa ( t), Er,d)
(2)
where (< t) specifies the lagged parents Xt,d at previous time and et,d is the mutually independent noise. Such model requires discrete time steps that are usually assumed follow regular sampling interval, i.e. f++1 - t; is a constant for all i 1,..., can be regarded as special case of this framework.
Most existing models
Itō diffusion A time-homogenous Itô diffusion stochastic process X, and has the form:
dx. = f(X))dt + ((W
where f RDD + R.D .R..g: RDD R DxD are time-homogeneous drift and diffusion functions, respectively, and W, is Brownian motion under the measure P. It is known that under global Lipschitz guarantees (Assumption T this has a unique strong solution (Oksendal Øksendal 2003).
Euler discretization and Euler SEM For most Itô diffusions, the analytic solution X, is in- tractable, especially with non-linear drift and diffusion functions. Thus, we often seek to simulate
2

---

Published as a conference paper at ICLR 2024
the trajectory by discretization. One common discretization method is the Euler-Maruyama (EM) scheme. With a fixed step size A, EM simulates the trajectory as
x - f(X
(4)
where X is the random variable induced by discretization and 7/- N/ A). Notice that eq. (4)
a special case of eq. 2 If we define the graph G as the following: X
1, X in G iff
( ax,,
+ or 3k, ( + O; and assume g only outputs diagonal matrix, then the above OX
EM induces a temporal SEM, called the Euler SEM (Hansen & Sokol 2014), which provides a
useful analysis tool for continuous time processes.
SCOTCH: BAYESIAN STRUCTURE LEARNING FOR CONTINUOUS TIME SERIES
We consider a dynamical system in which there is both intrinsic stochasticity in the evolution of the state, as well as independent measurement noise that is present the observed data. For example, in healthcare, the condition of patient will progress with randomness rather than deterministically. Further, the measurement of patient condition will also affected by the accuracy the equipment, where the noise is independent the intrinsic stochasticity. To account for this behaviour, propose to use the latent SDE formulation (Li et al. 2020 Tzen & Raginsky 2019a):
dz. = (Z,) + go Z,,,W, (latent process)
X; Z, + c (noisy observations)
(5)
where Z, RD is the latent variable representing the internal state of the dynamic system, X, E RD describes the observational data with the same dimension, Et is additive dimension- wise independent noise, fo T + R D is the drift function, go : R - Dx is the diffusion function and W, is the Wiener process. For SCOTCH, we require the following two assumptions:
Assumption (Global Lipschitz). We assume that the drift and diffusion functions in eq. 5 satisfy the global Lipschitz constraints. Namely, we have
fz(z) fo(y)) + 1go(x)) go(z)l Clz y1l
for some constant C, x, y E IRD and is the corresponding L2 norm for vector-valued functions and matrix norm for matrix- valued functions.
Assumption 2 (Diagonal diffusion). We assume that the diffusion function go outputs a non-zero diagonal matrix. That is, it can be simplified to vector-valued function go X.) RDD - RR.
The former is a standard assumption required by most SDE literature to ensure the existence of strong solution. The key distinction the latter assumption of nonzero diagonal diffusion function, go. rather than a full diffusion matrix, enabling structural identifiability section. Please refer to appendix A I for more detailed explanations. Now, in accordance with the graph defined in Euler SEMs (section
we show in the next
is (Z.) A 0 or o2., f O. Note that there no requirement for the
2.. we define the signature graph G follows: edge
present in G iff 3t s.t. either J2.,: graph to be acyclic. Intuitively, the graph G describes the structural dependence between variables.
(g.(
We now instantiate our Bayesian structure learning scheme with prior and likelihood components:
Prior over Graphs Leveraging Geffner et al. (2022); Annadani et al. (2023), our graph prior is designed as:
G(G) exp X₂ G
where X₂ is the graph sparsity coefficient, and || || is the Frobenius norm.
Prior process Since the latent process induces distribution over latent trajectories po Z)) before seeing any observations, also call it the prior process. We propose to use neural networks for drift and diffusion functions fo R² x {0, (0,1) + R go RR D x {0, 1 + RD, which explicitly take the latent state and the graph inputs. Though the signature graph is defined through
3

---

Published as a conference paper at ICLR 2024
the function derivatives, explicitly the graph G as input to enforce the constraint. We will in- terchangeably use the notation fc and ga to denote fo(.. G) and go(:, : G). For the graph-dependent drift and diffusion, we leverage the design of Geffner et al. 2022 and propose:
D Σ
fa.a Z+) -
Gral ei),
(8)
for both fc and gG, where Ç. are neural networks, and is trainable node embedding for the 1th node. The corresponding prior process
dz. = fo 2r G)dl + go 22 G)dW, (prior process)
Likelihood time series
Given a time series X = Xm, the likelihood is defined as where Pea is the observational noise distribution for the th dimension.
I D
p( Z1 (Z.) G)
IIII
Pea (Xu, - Zec.a)
(10)
(=1d=1
3.1 VARIATIONAL INFERENCE
Suppose that given multiple time series (x)) x= observed data from the system. The goal is then compute the posterior over graph structures p(G) (x(")) Jo= 1). which is intractable. Thus, we leverage variational inference to simultaneously approximate both the graph posterior, and a latent posterior process over (n) for every observed time series (n).
Given N i.i.d time series [X()) we propose to use a variational approximation (G) R ( ())). With the standard trick from variational inference, we have the following evidence lower bound (ELBO):
N log po
log p( (1)
x(x)) 2 (gy
Σ
X (("))) - DDL((s (G))P((G))
(11)
Unfortunately, the distribution pa X (n) G) remains intractable due to the marginalization of the latent Itô diffusion (n)) Therefore, we leverage the variational framework proposed in Tzen &
Raginsky 2019a); Li et al. 2020) to approximate the true posterior p(Z (x) [ ("), G). For each n
1,..., N, the variational posterior qo (n)) ( G) is defined as the solution to the following:
(n) (G, E. (G, (posterior initial state)
( = h (Z) (7
t; G. ( + ga (Z(") JdW, (posterior process)
(12)
For the initial latent state, p En are posterior mean and covariance functions implemented as neural networks. For the SDE, we the same diffusion function ga for both the prior and posterior processes, but train a separate neural drift function be for the posterior, which takes a time series ( as input. The posterior drift function differs from the prior in two key ways. Firstly, the posterior drift function depends time; this is necessary conditioning creates this dependence even when the prior process is time-homogenous. Secondly, while h. takes the graph G an input, the function design is not constrained to have a matching signature graph like fc. More details on the implementation of h2) phg Eg can be found in Appendix B
the observed data
Assume for each time series (n), we have observation times t, for = L..... I within the time range [0, T), then, we have the following evidence lower bound for log pl X (*)
Li et al. 2020
1-"
logp(X x (1))GT G)
Equ
log p(
| (0 (, 122dt
(13)
where 2(²) is the posterior process modelled by
12 and u (n) (2(a)) ) is given by:
u (1) ( ((("))) ga (0())
(n) G. ((1) fo(((11))
(14)
4

---

Published as a conference paper at ICLR 2024
Algorithm SCOTCH training
Input:
ii.d time series (== drift functions fc. has diffusion function ge. SDE solver Solver,
initial condition _ training iterations L for ...., L do
Sample time series mini-bateh {x ( with batch size S. for I...... do
Draw graph G ~ (G)
Draw initial latent state ((m)
(H (()),, En x Solve (sample from) the posterior process (z(m), , L) = Solver( 2o .0), fo, hv. gG)
end for
Maximize ELBO eq. 15 w.r.t. 1
end for
10
840
-10
-15
1216
-10
(a) Data
(b) NGM
(c) SCOTCH
Figure 1: Comparison between NGM and SCOTCH for simple SDE (note vertical axis scale)
By combining eq.
111 and eq.
13
derive overall ELBO:
N
/-/ [| (Z
log po (X (1)
X x >E
Ego
Ege
Σ logp( (n) (n)
G)
G)
||2
n=1
DKL (go (G)))(())
(15)
In practice, we approximate the ELBO (and its gradients) using a Monte-Carlo approximation. The inner expectation can be approximated by simulating from augmented version of eq. (12) where
an extra variable L is added with drift 1/ (m),") and diffusion zero (Li et al. 2020). rithm summarizes the training algorithm of SCOTCH.
Algo-
3.2 STOCHASTICITY AND CONTINUOUS- -IME MODELING
Stochasticity Bellot et al. 2022) proposed a structure learning method, called NGM, to learn from single time series generated by SDEs. NGM uses neural ODE to model the mean process fo. and extracts graphical structure from the first layer of fa- However, NGM assumes that the observed single series X follows a multivariate Gaussian distribution, which only holds for linear SDEs. If this assumption is violated, optimizin their proposed squared loss cannot recover the underlying system. SCOTCH does have this limitation and can handle more flexible state-dependent drifts and diffusions. Another drawback of NGM is its inability to handle multiple time series (N > 1). Learning from multiple series important when dealing with SDEs with multimodal behaviour. We propose simple bimodal 1-D failure case: dXX X + 0.01dW., Xo O, with the signature graph containing self-loop. Figure [ shows the bimodal trajectories (upwards and downwards) sampled from the SDE. The optimal ODE mean process in this case is the constant fa 0 with an empty graph, as confirmed by the learned mean process of NGM (black line in fig. [b). In contrast, SCOTCH can learn the underlying SDE and simulate the correct trajectories (fig. Ic).
Discrete vs Continuous- Gong et al. 2022 proposed a flexible discretised temporal SEM, called Rhino, that is capable of modelling (1) lagged parents; (2) instantaneous effect; and (3) history
5

---

Published as a conference paper at ICLR 2024
dependent noise. Rhino's SEM is given by = fa t), (t) + ga t)) We can clearly see its similarity to SCOTCH. If fa has a residual structure as fa(-) Ne.d+ra() (-)) A and we assume no instantaneous effect (Pag" (t) is empty), Rhino SEM is equivalent the Euler SEM of the latent process (eq. (9) with drift r, step size and diagonal diffusion g. Thus, similar to the relation of ResNet (He et al. 2016) to NeuralODE (Chen et al. 2018), SCOTCH can be interpreted as the continuous- analog of Rhino.
4 THEORETICAL CONSIDERATIONS OF SCOTCH
In this section, we aim to answer three important theoretical questions regarding the Itō diffusion proposed in section 3 For notational simplicity, we consider the single time series setting. First, we examine when a general Itô diffusion is structurally identifiable. Secondly, we consider structural identifiability in the latent formulation of eq. (5). Finally, we consider whether optimising ELBO (eq. (15) can recover the true graph and mechanism if we have infinite observations of single time series within a fixed time range [0, Tİ. All detailed proofs, definitions, and assumptions can be found in appendix A
4.1 STRUCTURE IDENTIFIABILITY
Suppose that the observational process is given as Itô diffusion:
dX: dx)dt + (X,)))
(16)
Then, we might ask what are sufficient conditions for the model be structurally identifiable? That is, there does not exist G' * G' that can induce the same observational distribution.
Theorem 4.1 (Structure identifiability of the observational process). Given ea. [66 let us define another process with X,, G o G. f&r 9G and W,. Then, under Assumptions 12 and with the same initial condition (0) X (0) = the solutions X, and X, will have different distributions.
Next, we show that structural identifiability is preserved, under certain conditions, even in the latent formulation where the SDE solution is not directly observed.
Theorem 4.2 (Structural identifiability with latent formulation). Consider the distributions p. de- fined by the latent model in eq. (5) with (G, Z, X, K, gG); (G, 2,8, ,, Dc) respectively, where G + G. Further, let .......ty be the observation times. Then, under Assumptions and
1. if l++1 where pd is the density generated by the Fuler discretized eq. for Z,;
A for all 1...,, - 1, then A AA(t....... Xx,) + 7
X,,))
2. if we have fixed time range 10, T) then the path probability o( ......... pl a p(....... X1,) under the limit of infinite data (I + o0).
X1,) +
4.2 CONSISTENCY
Building upon the structural identifiability, we can prove the consistency of the variational formula- tion. Namely, in the infinite data limit, one can recover the ground truth graph and mechanism by maximizing ELBO with a sufficiently expressive posterior process and correctly specified model.
Theorem 4.3 (Consistency of variational formulation). Suppose Assumptions 7 are satisfied for the latent formulation (eq. 5b). Then, for a fixed observation time range [0, TT. as the number of observations > oo, when ELBO (eq. (15 is maximised, qo (G) 8(G"), where G is the ground truth graph, and the latent formulation recovers the underlying ground truth mechanism.
5 RELATED WORK
Discrete time causal models The majority of the existing approaches are inherently discrete in time. Assaad et al. (2022) provides comprehensive overview. There are three types discov- ery methods: (1) Granger causality; (2) structure equation model (SEM); and (3) constraint-based

---

Published as a conference paper at ICLR 2024
methods. Granger causality assumes that no instantaneous effects are present and the causal di- (2020); : Shojaic & Michailidis (2010); Siggiridou
rection cannot flow backward in time. Ww et al.
& Kuuguuttzis 2013); Amombbnchhrrvve et al.
(2019) leverage the pector-autoregressin model to
predict future observations. Löwe et al. (2022); Tank et al. (202T): [usssann et al. (2021); ; Dang
et al.] (2019); Xu et al. (2019); Khanna & Tan (2019) utilise deep neural networks for prediction.
Recently, Cheng et al. 2023) introduced a deep-learning based Granger causality that can handle irregularly sampled data, treating it as a missing data problem and proposing joint framework for data imputation and graph fitting. SEM based approaches assume an explicit causal model associ- ated to the temporal process. [Hyvärinen et al. (2010) leverages the identifiability of additive noise models (Hoyer et al. 2008) to build linear auto-regressive SEM with non- Gaussiassi noise. Pamfil et al. (222)) utilises the NOTEARS framework (Zheng et al. 2018) to continuously relax the DAG constraints for fully differentiable structure learning. The recently proposed Gong et al. (2022) ex- tended the prior DECI (Geffner et al. (2022) framework to handle time series data and is capable of modelling instantancous effect and iistory-dependen noise. Constraint -aasdd approaches use con- ditional independence tests to determine the causal structures. Runge et al. (2019) combines the PC (Spirtes et al. 2000) and momentary conditional independence tests for the lagged parents. PCMCI+ [Runge| 2020) can additionally detect the instantaneous effect. LPCMCI (Reiser (022) can further handle latent confounders. CD-NOD (Zhang al. 2017) is designed handle non-stationary het- erogeneous time series data. However, all constraint based approaches can only identify the graph up to Markov equivalence class without the functional relationship between variables.
Continuous time causal models In terms of using differential equations to model the continu- ous temporal process, Hansen & Sokol (2014) proposed using stochastic differential equations to describe the temporal causal system. They proved identifiability with respect to the intervention distributions, but did not show how to learn a corresponding SDE. Penalised regression has been explored for linear models, where parameter consistency has been established (Ramsay et al.. 2007] Chen et al. 2017] Wu et al. 2014). Recently, NGM (Bellot et al., 2022) uses ODEs to model the temporal process with both identifiability and consistency results. As discussed in previous sections, SCOTCH is based on SDEs rather than ODEs, and can model the intrinsic stochasticity within the causal system, whereas NGM assumes deterministic state transitions.
EXPERIMENTS
Baselines and Metrics We benchmark our method against representative sample of baselines: () VARLiNGaM (Hyvärinen et al., 2010). a linear SEM based approach; (ii) PCMCI+ (Runge) 2018 2020 a constraint-based method for time series; (iii) CUTS, a Granger causality approach which can handle irregular time series; (iv) Rhino (Gong et al.] 2022). a non-linear SEM based approach with history-dependent noise and instantaneous effects: and (v) NGM (Bellot al. (022)) a continuous-time ODE based structure learner. Since most methods require threshold to determine the graph, we use the threshold-free area under the ROC curve (AUROC) as the performance metric. In appendix D we also report F1 score, true positive rate (TPR) and false discovery rate (FDR).
Setup Both the synthetic datasets (Lorenz-96, Glycolysis) and real-world datasets (DREAM3, Netsim) consist of multiple time series. However, it is not trivial to modify NGM and CUTS to support multiple time series. For fair comparison, we use the concatenation of multiple time series, which we found empirically improve performance. We also mimic irregularly sampled data by randomly dropping observations, which VARLiNGaM, PCMCI, and Rhino cannot handle; in these cases, for these methods we impute the missing data using zero-order hold (ZOH). For SCOTCHI we use pathwise gradient estimators with Euler discretization for solving the SDE (see appendix D. for discussion on this choice). Further experimental details can be found in Appendices B
D
6.1 SYNTHETIC EXPERIMENTS: LORENZ AND GLYCOLYSIS
First, we evaluate SCOTCH on synthetic benchmarks including the Lorenz-96 (Lorenz 1996) and
Glycolysis (Daniels & Nemenman 2015) datasets, which model continuous-ting time dynamical sys- tems. The Lorenz model is a knolnn example of chaotic systems observed in biology (Gold- berger & West 1987 Heltberg et al.) (2019). To mimic irregularly sampled data, we follow the setup of Cheng al. (023) and randomly drop some observations with missing probability p. We also
7

---

Published as a conference paper at ICLR 2024
simulate another dataset from biological model, which describes metabolic iterations that break down glucose in cells. This is called Clycolysis, consisting of an SDE with variables. As prepro- cessing step, standardised this dataset to avoid large differences in variable scales. Both datasets consist of N = 10 time series with sequence length = 100 (before random drops), and have dimensionality 10 and 7, respectively. Note that we choose large data sampling interval, want to test settings where observations are fairly sparse and the difficulty of correctly modelling continuous-time dynamics increases. The above data setup is different from [elllot al. (2022); Cheng et al. (2023) where they use single series with I 1000 observations, which is more informative compared to our sparse setting. Refer appendix [D3 and appendix (D4 details.
The left two columns in table I compare the AUROC of SCOTCH to baselines for Lorenz. We can see that SCOTCH can effectively handle the irregularly sampled data compared to other baselines. Compared to NGM and CUTS, we can achieve much better results with small missingness and performs competitively with larger missingness. Rhino, VARLiNGaM and PCMCI+ perform poorly in comparison as they assume regularly sampled observations and are discrete in nature.
From the right column table T SCOTCH outperforms the baselines by a large margin on Glycol- ysis. In particular, compared to the ODE-based NGM, SCOTCH clearly demonstrates the advantage of the proposed SDE framework in multiple time series settings. As we may have anticipated from the discussion in section 3.2. NGM can produce an incorrect model when multiple time series sampled from a given SDE system. Another interesting observation is that SCOTCH is more robust when encountering data with different scales compared to NGM (refer appendix(D.4.3) This bustness is due to the stochastic nature of SDE compared to the deterministic ODE, where ODE can easily overshoot with less stable training behaviour. We can also see that SCOTCH has significant advantage over both CUTS and Rhino, which do not model continuous- dynamics.
Lorenz-96
Glycolysis Full
p= 0.3
p = 0.6
VARLiNGaM
0.5102:10.025
0.487610.032
0.5082:10.009
PCMCI+
0.4990+1.000
0,4952::0.021
0.4607++0003
NGM
0,6788
0.6329110008
0.595310.018
CUTS
0.6042+000015
0.6418=0.012
0.580++0.007
Rhino
0.571411.026
0.5123110025
0.520:10.015
SCOTCH (ours)
0.7279
0.6453=0.014
0.7113+10.012
Table 1: AUROC of synthetic datasets from SCOTCH and baselines, represents missing probabil-
ity, and Full means complete data without missingness. Each number the average over 5 runs
6.2 DREAM3
We also evaluate SCOTCH performance on the DREAM3 datasets (Prill et al. 2010 Marbach et al. 2009). which have been adopted for assessing the performance of structure learning (Tank et al. 2021 Pamfil et al.) 2020; Gong et al.) 2022). These datasets contain in silico measurement of gene expression levels for different structures. Each dataset corresponds network, and contains N = 46 time series of 100 dimensional variables, with = 21 per series. The goal is to infer the underlying structures from each dataset. Following the same setup as (Gong al.) 2022 Khanna & Tan 2019), we ignore all the self-connections by setting the edge probability to O, and use AUROC as the performance metric. Appendix D.5 details the experiment setup, selected hyperparameters, and additional plots. We do not include VARLiNGaM since it cannot support time series where the dimensionality (100) is greater than the length (21). Also, due to the time series length, we decide not to test with irregularly sampled data. We use the reported numbers for Rhino and PCMCI+ in [oog et al.) (2022) as the experimental setup is identical. For CUTS, we failed to reproduce the reported number in their paper, but cite it for fair comparison.
particular gene expression
Table 2 shows the AUROC performances of SCOTCH and baselines. We can clearly observe that SCOTCH outperforms the other baselines with a large margin. This indicates the advantage of the SDE formulation compared to ODEs and discretized temporal models, even when we have complete and regularly sampled data. A more interesting observation is to compare Rhino with SCOTCH. As discussed before, as SCOTCH is the continuous version of Rhino, the advantage comes from the continuous formulation and the corresponding training objective eq. 15
8

---

Published as a conference paper at ICLR 2024
EColi1
Ecoli2
Yeastl
Yeast2
Yeast3
Mean
PCMCI+
0.53020.002
0.519=0.002
0.530 10.003
0.510=10.001
0.512 + 0
0.520=10.004
NGM
0.611
0.595110.005
0.507:10.006
0.563-10.006
0,535
0.580.10.007
CUTS
0,543-10.003
0.555-10.005
0.54510.003
0.518:10.007
0.51110.002
0.534:10.000 008 (0.591)
Rhino
0.685=0.003
0.680=10.007
0.664:0.000
0.585:10.004
0.567=0.003
0.636:10.022
SCOTCH (ours)
0.752
0.705
0.712:0.003
0.622 0.004
0.594: 0.001
0.677:1 0.026
Table 2: AUROC for SCOTCH on DREAM3 100-dimensional datasets. Results are obtained by averaging over 5 runs. We cite the reported CUTS performance in parentheses.
Full
p = 0.1
p 0.2
VARLiNGaM
0.84+0
0.723+00001
0.719+00003
PCMCI
0.83+0
0.81
0.79=0.006
NGM
0.89 + 0.009
0.86 0 0.009
0.85 +0.007
CUTS
0.89 0.010
0.87 0.008
0.87 10.011
Rhino+NoInst
0.95 10.001
0.93 0.005
0.90=0.012
SCOTCH (ours)
0.95# 0.006
0.9110.007
0.89=0.007
Rhino
0.99=0.001
0.981100004
0.97
Table AUROC on Netsim dataset (subjects 2-6). Results are obtained by averaging over runs.
6.3 NETSIM
Netsim consists of blood oxygenation level dependent imaging data. Following the same setup as subjects 2-6 to form the dataset, which consists of 5 time series. Each contains 15 dimensional observations with / 200. The goal is to infer the underlying connectivity between different brain regions. Unlike Dream3,
Gong et al. (2022),
include the self-connection edge for all meth- ods. To evaluate the performance under irregularly sampled data, we follow the same setup as in the Lorenz and Cheng al. (2023) to randomly drop observations with missing probability. Since is very important to model instantaneous effects in Netsim (Gong et al. 2022), which SCOTCH cannot handle, we replace Rhino with Rhino+NoInst and PCMCI+ with PCMCI for fair comparison.
Table 3 shows the performance comparisons. We observe that SCOTCH significantly outperforms the other baselines and performs on par with Rhino+NoInst, which demonstrates its robustness to- wards smaller datasets and balance between true and false positive rates. Again, this confirms the modelling power of our approach compared NGM and other baselines. Interestingly, Rhino- based approaches perform particularly well the Netsim dataset. We suspect that the underlying generation mechanism can be better modelled with discretized as opposed to continuous system.
7 CONCLUSION
We propose SCOTCH, a flexible continuous- time temporal structure learning method based on latent Itô diffusion. We leverage the variational inference framework to infer the posterior over latent states and the graph. Theoretically, we validate our approach by proving the structural identifiability the Itô diffusion and latent formulation, and the consistency of the proposed variational framework. Em pirically, we extensively evaluated our approach using synthetic and semi-synthetic datasets, where SCOTCH outperforms the baselines in both regularly and irregularly sampled data. There are three limitations that require further investigation. The first one is the inability to handle instantancous ef- fects, which can arise due data aggregation. Another computational drawback with the series length. This could be potentially fixed by incorporating an encoder network to infer latent states arbitrary time points. Last but least, the current formulation of SCOTCH cannot handle -tatioonn systems due to the homogeneous drift and diffusion function. However, direct incorporation of time embeddings may break the theoretical guarantees without additional assump- tions. Therefore, new theories and methodologies may be needed to tackle such a scenario. We leave these challenges for future work.
scales linearly

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank the members of the Causica team at Microsoft Research for helpful discussions. We thank Colleen Tyler, Maria Defante, and Lisa Parks for conversations on real-world use cases that inspired this work. This work was done in part while Benjie Wang was visiting the Simons Institute for the Theory of Computing.
REFERENCES
Chainarong Amombunchornvej, Elena Zheleva, and Tanya Y Berger- Wolf. Variable-lag granger causality for time series analysis. In 2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp. 21-30. IEEE, 2019.
Yashas Annadani, Nick Pawlowski, Joel Jennings, Stefan Bauer, Cheng Zhang, and Wenbo Gong. Bayesdag: Gradient-based posterior sampling for causal discovery. arXiv preprint 11X1: 2023.
Charles Assaad, Emilie Devijver, and Eric Gaussier. Survey and evaluation of causal discovery methods for time series. Journal of Artificial Intelligence Research, 73:767-819, 2022.
Alexis Bellot, Kim Branson, and Mihaela van der Schaar. Neural graphical modelling in continuous- time: consistency guarantees and algorithms. In International Conference on Learning Represen- lutions, 2022.
Carlo Berzuini, Philip Dawid, and Luisa Bernardinell. Causality: Statistical perspectives and ap- plications. John Wiley & Sons, 2012.
Michelle Boué and Paul Dupuis. A variational representation for certain functionals of brownian motion. The Annals of Probability, 26(4):1641-1659, 1998.
Annalisa Bracco, Fabrizio Falasca, Athanasios Nenes, Ilias Fountalis, and Constantine Dovrolis. Advancing climate science with knowledge-discovery through data mining. npj Climate and Atmospheric Science, 1(1):20174, 2018.
Bart Bussmann, Jannes Nys, and Steven Latré. Neural additive vector autoregression models for causal discovery in time series. In Discovery Science: 24th International Conference, DS 2021, Halifax, NS, Canada, October 11-13, 2021, Proceedings 24, pp. 446-460. Springer, 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Shizhe Chen, Ali Shojaie, and Daniela M Witten. Network reconstruction from high-dimensional ordinary differential equations. Journal of the American Statistical Association, 1127520):1697- 1707, 2017.
Yuxiao Cheng, Runzhao Yang, Tingxiong Xiao, Zongren Li, Jinli Suo, Kunlun He, and Qiong- hai Dai. Cuts: Neural causal discovery from irregular time-series data. arXiv preprint arXiv:2302.07458, 2023.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcchre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint 1406 2014.
Andrea Cini, Ivan Marisca, and Cesare Alippi. Filling the g-ap-s: Multivariate time series imputation by graph neural networks. In International Conference on Learning Representations, 2022. URL
httpss://penreview
Xuan- Xuan-Hong Dang, Syed Yousaf Shah, and Petros Zerfos. seg2graph: Discovering dynamic non- linear dependencies from multivariate time series. In 2019 IEEE International Conference on Big Data (Big Data), pp. 1774-1783. IEEE, 2019.
10

---

Published as a conference paper at ICLR 2024
Bryan C Daniels and Ilya Nemenman. Efficient inference of parsimonious phenomenological mod- els cellular dynamics using s-systems and alternating regression. PloS one, 1033):c011921,, 2015.
Paul Dupuis and Richard $ Ellis. A weak convergence approach to the theory of large deviations. John Wiley & Sons, 2011.
Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, et al. Deep end-to-end causal inference. arXiv preprint arXiv:2202.02/995 2022.
Ary L Goldberger and Bruce West. Applications of nonlinear dynamics to clinical cardiology. Annals of the New York Academy of Sciences, 504: 1987.
Wenbo Gong, Yingzhen Li, and José Miguel Hernández-Lobato. Meta-learning for stochastic gra- dient memc. arXiv preprint arXiv: 180.004522 2018.
Wenbo Gong, Joel Jennings, Cheng Zhang, and Nick Pawlowski. Rhino: Deep causal temporal relationship learning with history-dependent noise. arXiv preprint 17Xi::2210.14706, 2022.
Niels Hansen and Alexander Sokol. Causal interpretation stochastic differential equations. 2014.
Ali Hasan, Joao M Pereira, Sina Farsiu, and Vahid Tarokh. Identifying latent stochastic differential equations. IEEE Transactions on Signal Processing, 70:89-104, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Mathias L Heltberg, Sandeep Krishna, and Mogens H Jensen. On chaotic dynamics in transcription factors and the associated effects in differential gene regulation. Nature communications, 10(1): 71, 2019.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators, Neural networks, 25))335--666, 1989.
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances in neural information processing systems, 21, 2008
Aapo Hyvärinen, Kun Zhang, Shohei Shimizu, and Patrik O Hoyer. Estimation of a structural vector autoregression model using non non-gaussianity. Journal of Machine Learning Research, 11(5), 2010.
Saurabh Khanna and Vincent YF Tan. Economy statistical recurrent units for inferring nonlinear granger causality. arXiv preprint arXiv:1911.0990 2019.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International Conference Artificial Intel- ligence and Statistics, pp. 2207-2217. PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14826690, 2014.
Xuechen Li, Ting-Kam Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp. 3870-3882, PMLR, 2020.
Edward N Lorenz. Predictability: A problem partly solved. In Proc. Seminar on predictability, volume 1. Reading. 1996.
Sindy Löwe, David Madras, Richard Zemel, and Max Welling. Amortized causal discovery: Leam- ing to infer causal graphs from time-serice data. In Conference on Causal Learning and Reason- ing, pp. 509-525. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2024
Daniel Marbach, Thomas Schaffter, Claudio Mattiussi, and Dario Floreano. Generating realistic in silico gene networks for performance assessment of reverse engineering methods. Journal of computational biology, 16(2):229-233 2009.
Bernt Øksendal and Bernt Øksendal. Stochastic differential equations. Springer, 2003.
Roxana Pamfil, Nisara Sriwattanaworachai. Shaan Desai, Philip Pilgerstorfer, Konstantinos Geor- gatzis, Paul Beaumont, and Bryon Aragam. Dynotears: Structure learning from time-series data. In International Conference Artificial Intelligence and Statisfics, pp. 1595-1605. PMLR, 2020.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithmx. The MIT Press, 2017.
Robert. Prill, Daniel Marbach, Julio Saez-Rodriguez, Peter K Sorger, Leonidas G Alexopoulos, Xi- and Gustavo Stolovitzky. Towards a rigorous
aowei Xue, Neil Clarke, Gregoire assessment of systems biology models: the dream3 challenges. PloS one, 5(2)xe9202, 2010.
Zhaozhi Qian, Ahmed Alaa, Alexis Bellot, Mihaela Schaar, and Jem Rashbass. Learning dynamic and personalized comorbidity networks from event data using deep diffusion processes. In Inter- national Conference on Artificial Intelligence and Statistics, pp. 301332253320 -3005 PMLR, 2020.
Xiaojie Qiu, Qi Mao, Ying Tang, Li Wang, Raghav Chawla, Hannah A Pliner, and Cole Trapnell. Reversed graph embedding resolves complex single-cell trajectories. Nature methods, 14(10): 979-982, 2017.
Federica Raia. Causality in complex dynamic systems: A challenge in earth systems science educa- tion. Journal of Geoscience Education, 56(1):81-94 2008.
Jim O Ramsay, Giles Hooker, David Campbell, and Jiguo Cao. Parameter estimation for differential equations: generalized smoothing approach. Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(5):741-766, 2007.
Christian Reiser. Causal discovery for time series with latent confounders. arXiv preprint
arXiv:2209.03427, 2022.
Jakob Runge. Causal network reconstruction from time series: From theoretical assumptions to practical estimation. Chaos: An Interdisciplinary Journal of Nonlinear Science, 28(7), 2018.
Jakob Runge.
Discovering contemporaneous and lagged causal relations in autocorrelated nonlinear time series datasets. In Conference on Uncertainty in Artificial Intelligence, pp. 1388-1397. PMLR, 2020.
Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdinovic. Detecting and quantifying causal associations in large nonlinear time series datasets. Science advances, (1 (1))ceau4996, 2019.
Shohei Shimizu, Patrik Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7(10), 2006.
Ali Shojaie and George Michailidis. Discovering graphical granger causality using the truncating lasso penalty. Bioinformatics, 26(1 (6(18)55517-5523, 2010.
Elsa Siggiridou and Dimitris Kugiumtzis. Granger causality in multivariate time series using time- ordered restricted vector autoregressive model. IEEE Transactions on Signal Processing, 64(7): 1759-1775 2015.
Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2000.
Alex Tank, Ian Covert, Nicholas Foti, Ali Shojaie, and Emily B Fox. Neural granger causality. JEEE Transactions Pattern Analysis and Machine Intelligence, 44(8):42674427,, 2021.
12

---

Published as a conference paper at ICLR 2024
Cole Trapnell, Davide Cacchiarelli, Jonna Grimsby, Prapti Pokharel, Shugiang Li, Michael Morse, Niall J Lennon, Kenneth Livak, Tarjei Mikkelsen, and John L Rimn. The dynamics and regulators of cell fate decisions are revealed by pseudotemporal ordering of single cells. Nature biotechnology, 32(4))381-38) 2014.
Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv: 009883 2019a.
Belinda Tzen and Maxim Raginsky. Theoretical guarantees for sampling and inference in generative models with latent diffusions. In Conference on Learning Theory, pp. 3084-3114. PMLR, 2019b.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pp. 681-688. Citeseer, 2011.
Hulin Wu, Tao Lu, Hongqi Xue, and Hua Liang. Sparse additive ordinary differential equations for dynamic gene regulatory network modeling. Journal of the American Statistical Association, 109 (506):700-716, 2014.
Tailin Wu, Thomas Breuel, Michael Skuhersky, and Jan Kautz. Discovering nonlinear relations with minimum predictive information regularization. arXiv preprint arXiv:2001.01885, 2020.
Chenxiao Xu, Hao Huang, and Shinjae Yoo. Scalable causal graph learning through deep neural network. In Proceedings of the 28th ACM international conference on information and knowledge management, pp. 1853-1862, 2019.
Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt. Advances in variational inference. IEEE transactions on pattern analysis and machine intelligence, 41(8):2008-2) 2026, 2018.
Kun Zhang, Biwei Huang, Jiji Zhang, Clark Glymour, and Bernhard Schölkopf. Causal discovery from nonstationayyheeerrggaou data: Skeleton estimation and orientation determination. In IJCAI: Proceedings of the Conference, volume 2017, pp. 1347. NIH Public Access, 2017.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems, 31, 2018.
13


Reviews:
[{"review_id": "QZjybFPivo", "reviewer": "Reviewer_rM8f", "summary": "This paper introduces SCOTCH  a continuous-time stochastic model that combines stochastic differential equations (SDEs) and variational inference to model temporal processes and learn the underlying graph structure of the dynamics. The results include theory on sufficient conditions for SCOTCH to be structurally identifiable and empirical results on a variety of biomedical systems", "questions": "What is the effect of the sparsity prior on the structure? Can a different type of sparsity be used?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "-Continuous dynamics modeling is a good extension (over RHINO etc) because of the ability to incorporate irregularly sampled time series\n-The Ito diffusion model is very flexible for modeling a variety of types of dynamic \n-I think that the independent diagonal noise assumption is perfectly reasonable for the generation of the dynamics \n-It is impressive that this network both learns the dynamics as well as matches the flows \n-The identifiability results are important despite their assumptions", "weaknesses": "-I think the assumptions (1, 2) should be more clearly stated (I had to comb through the text to find them)\n-In specific homogenous drift and diffusion processes assumption may not be able to model some types of dynamics, some discussion on that would be interesting \n-Not sure if I believe the explanation of the improved performance of RHINO on the Netsim data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YgoPm3VxOm", "reviewer": "Reviewer_pC52", "summary": "The paper introduces a novel structure learning method, SCOTCH, which leverages neural stochastic differential equations (SDE) and variational inference to infer posterior distributions over possible structures in continuous-time data. Traditional structure learning methods assume discrete-time processes with regularly spaced observations, which can lead to incorrect models. SCOTCH, however, is capable of handling both learning from and predicting observations at arbitrary time points. The authors establish the structural identifiability and consistency of SCOTCH under certain conditions. Empirical evaluations on synthetic and real-world datasets demonstrate its superior performance compared to relevant baselines, even with irregularly sampled data.", "questions": "See Weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. One of the primary strengths of this paper is its originality in tackling the problem of structure learning in continuous-time data. While many existing methods focus on discrete-time processes with regular observations, SCOTCH introduces a novel approach that can handle irregularly sampled data in continuous time. \n\n2. The paper provides a strong theoretical foundation for the proposed method. The establishment of structural identifiability conditions and the proof of consistency under infinite data limits. \n\n3. The paper maintains a high level of clarity in explaining the methodology, making it accessible to a wide audience. Additionally, the empirical evaluations conducted on both synthetic and real-world datasets demonstrate the effectiveness of SCOTCH. The comparison with relevant baselines under different sampling conditions reinforces the credibility of the proposed approach.", "weaknesses": "1. SCOTCH relies on neural stochastic differential equation (SDE) methods, which means that its performance and computational cost are inherently linked to the accuracy of numerical SDE solvers. The paper would benefit from a more in-depth analysis and discussion regarding the sensitivity of the proposed method to the SDE solvers.\n\n2. The paper aims to sample the structure of G and obtain a sparse G. However, it falls short in providing a thorough discussion of the reasons behind the choice of priors for structure sampling. Additionally, there is no exploration of whether achieving sparsity in G leads to improved estimation performance, or if it primarily results in more interpretable results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LZMR4ZMSiq", "reviewer": "Reviewer_XYcj", "summary": "The paper presents a structure learning method which aims to learn a graph fused into drift and diffusion function in stochastic differential equations (SDEs). The variational inference is followed Li et al, 2020 with additional conditions over graphs. The paper then studies structure identifiability of the model using tools from stochastic calculus. Experiments are conducted in both synthetic data sets and real-world data sets, comparing the proposed methods with alternative approaches.", "questions": "Please see weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper gives an interesting connection between structural learning and SDE. The theory and practice from SDE literature builds a good foundation for this direction.\n- There is a strong empirical evidence that SCOTCH performs well across multiple tasks.", "weaknesses": "Although the paper provides empirical results compared to existing models, it does not provide any analysis about obtained graphs. For me, I am more curious about the quality of produced graphs from SCOTCH compared to other models, and how we understand their structures. I wonder if there is any way to visualize the graphs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YDbS6mEdaT", "reviewer": "Reviewer_gu2c", "summary": "The authors describe a type of latent variable model that can be identified from the high dimensional observations. Specifically, the authors describe a type of stochastic differential equation based model where the parameters of the drift and diffusion are given by some graph. The graph induces a particular type of interaction between the variables and one can then estimate some causal structures from this. The authors then provide a series of experiments demonstrating the applicability of their proposed method. They additionally theoretically analyze the structural identifiability in terms of the graph structure.", "questions": "What is the main difference with other latent SDE models that have been proposed? Is it that the drift/diffusion are factorized in terms of the graph implying a particular architecture for those functions?\n\nIf one uses existing methods for uncovering SDEs (that use, for example, neural networks to represent functions) is there a way to estimate the graph structure from the learned function (e.g. by computing the partial derivatives between the function and the input)? How would the methods compare if the authors applied this technique to existing methods that estimate latent SDEs?\n\nDo the authors have an idea of how the performance scales to higher dimensional observations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The topic is important and the authors consider an important class of stochastic processes to work on. The method empirically performs well in comparison to existing models.  The method is additionally well motivated with a few theoretical results on identifying the latent structure as well as the parameters of the diffusion. The interpretation of the drift and diffusion in terms of graphs provides an easier interpretation of the dependencies of the variables.", "weaknesses": "Aside from the graph interpretation if the drift/diffusion, the work is similar to a lot of existing works, so I wonder a bit how it is practically different from some of the others. For example, some of the theoretical results on latent identifiability are from the work Hasan et al but the authors do not consider that work as a baseline or mention it within the related work, though they estimate similar quantities.  Additionally the authors mention that this is a novel latent SDE formulation but most of the methodologies follow existing methods using a change of measure (e.g. Li et al among others). \n\nThe numerical results were applied to fairly low dimensional datasets though it seems like the work could be applied to higher dimensional settings (as some of the related work applied to higher dimensional settings).\n\nTo summarize, I think I'm mainly confused as to what is different in this work compared to existing works since it's methodologically similar and the theoretical results are also similar. From what I've understood, it's mainly that the graph structure dictating the interaction is explicit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)