CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery
Authors: Yuxiao Cheng
Ziqian Wang
Tingxiong Xiao
Qin Zhong
Jinli Suo
Kunlun He
Venue: ICLR 2024 poster (2024)
Keywords: Time series
Causal discovery
Neural networks
Benchmark
Dataset
Primary area: datasets and benchmarks
TL;DR: A novel pipeline capable of generating realistic time-series along with a ground truth causal graph that is generalizable to different fields.

Abstract: Time-series causal discovery (TSCD) is a fundamental problem of machine learning. However,  existing synthetic datasets cannot properly evaluate or predict the algorithms' performance on real data. This study introduces the CausalTime pipeline to generate time-series that highly resemble the real data and with ground truth causal graphs for quantitative performance evaluation. The pipeline starts from real observations in a specific scenario and produces a matching benchmark dataset. Firstly, we harness deep neural networks along with normalizing flow to accurately capture realistic dynamics. Secondly, we extract hypothesized causal graphs by performing importance analysis on the neural network or leveraging prior knowledge. Thirdly, we derive the ground truth causal graphs by splitting the causal model into causal term, residual term, and noise term. Lastly, using the fitted network and the derived causal graph, we generate corresponding versatile time-series proper for algorithm assessment. In the experiments, we validate the fidelity of the generated data through qualitative and quantitative experiments, followed by a benchmarking of existing TSCD algorithms using these generated datasets. CausalTime offers a feasible solution to evaluating TSCD algorithms in real applications and can be generalized to a wide range of fields. For easy use of the proposed approach, we also provide a user-friendly website, hosted on www.causaltime.cc.

Published as a conference paper at ICLR 2024
CAUSALTIME: REALISTICALLY GENERATED TIME- SERIES FOR BENCHMARKING OF CAUSAL DISCOVERY
Yuxiao Cheng Ziqian Wang Jinli Suol22
Tingxiong Xiao
Qin Zhong²
Kunlun He*
'Department of Automation, Tsinghua University
Innttitute for Brain and Cognitive Science, Tsinghua University (THUIBCS)
Chinese PLA General Hospital
cyx22, z i qi w2-m20 i 1s. ts i nighnng .ddu.cn
ABSTRACT
Time-series causal discovery (TSCD) is fundamental problem of machine learn- ing. However, existing synthetic datasets cannot properly evaluate or predict the algorithms' performance real data. This study introduces the CausalTime pipeline to generate time-series that highly resemble the real data and with ground truth causal graphs for quantitative performance evaluation. The pipeline starts from real observations in a specific scenario and produces a matching benchmark dataset. Firstly, we harness deep neural networks along with normalizing flow accurately capture realistic dynamics. Secondly, we extract hypothesized causal graphs by performing importance analysis the neural network or leveraging prior knowledge. Thirdly, we derive the ground truth causal graphs by splitting the causal model into causal term, residual term, and noise term. Lastly, using the fitted network and the derived causal graph, we generate corresponding versa- tile time-series proper for algorithm assessment. the experiments, we validate the fidelity of the generated data through qualitative and quantitative experiments, followed by a benchmarking of existing TSCD algorithms using these generated datasets. CausalTime offers a feasible solution to evaluating TSCD algorithms in real applications and can be generalized to a wide range of fields. For easy use of the proposed approach, also provide user-frient user-friendly website, hosted on www, causa t. me. CC.
INTRODUCTION
Inferring causal structures from time-series, i.e., time-series causal discovery (TSCD), a funda- mental problem in machine learning. It goes beyond prediction forecasting by revealing the complex interactions buried under multi-variate time-series. Recently, many algorithms have been proposed (Löwe et al. 2022 Li al.) 2020 Wu al.) 2022 Brouwer al. (221)) and achieved satisfactory performance, i.e., the discovered causal graphs are close to the ground-trutt -ruuh counter- parts. Under some settings, the causal discovery results are nearly perfect, with AUROC scores approaching 1.
However, the benchmarks for TSCD algorithms not suffice for the performance evaluation. First of all, for the statistical significance of the quantitative evaluation results, the datasets need to be improved in terms of quality and quantity. Next, the current datasets are limited several fields and do not cover wide application directions. More importantly, the datasets with ground-truth causal graphs are synthesized and might deviate from the true data-generating process, so the scores may not reflect the performance real data (Reisach et al. 2021)
Despite the fact that recent works also propose better benchmarks for time-series causal discovery Lawrence et al. 202I Runge et al. 2020), as well as static settings (Göbler et al. 2023 Cheval- ley et al. 2023a b), current TSCD algorithms often incorporate three types of datasets: Numerical
"quual Contribution Correspnniing author

---

Published as a conference paper at ICLR 2024
datasets, e.g., VAR (vector auto-regression) and Lorenz-96 (Karimi & Paul 2010). are simulated using closed-form equations. Although some of these equations (Lorenz-96) are inspired by real application scenarios, e.g., climate dynamics, they over-simplified and have very limited gen- eralizability to real-world applications (Runge et al. 2020). Quasi-real datasets are composed of time-series generated with manually designed dynamics that mimic real counterparts under certain scenario. For example, DREAM3 (Prill et al., 2010) is dataset simulated using gene expression and regulation dynamics, and NetSim (Smith et al., 2011) is generated by simulating interactions between human brain regions under observation of fMRI. The problem with this type of dataset is that it only covers few research areas with underlying mechanisms relatively clearly known. For fields such as healthcare or finance, it is hard or even impossible to generate realistic time-series with manually designed dynamics. Real datasets (such as MoCap (Tank et al. 2022). S&P 100 stock returns (Pamfil et al. 2020)) do not have the above-mentioned problem, but the dealbreaker is that the ground truth causal graph is mostly inaccessible, and we have to resort to some ad hoc explanations. As shown in Tab. D currently available benchmarking tools cannot support a com- prehensive evaluation of the time-series causal discovery algorithm. Therefore, an approach for generating benchmarks that highly mimic the real data in different scenarios and with true causal graphs is highly demanded.
Table Comparison of benchmarks for time-series causal discovery evaluation.
Datasets
Numerical
Quasi-real
Real
CausalTime (Ours)
Realistic Data
Low
Moderate
Very High
High
With True Causal Graph
X
Generalizable to Diverse Fields
In this work, we propose a novel pipeline capable of generating realistic time-series along with a ground truth causal graph and is generalizable to different fields, named CausalTime. The process of generating time-series with given causal graph can be implemented using the autoregression model, however, pursuing a causal graph that matches the target time-series with high accuracy is nontrivial, especially for the data with little prior knowledge about the underlying causal mecha- nism. To address this issue, we propose to use a deep neural network to fit the observed data with high accuracy, and then retrieve causal graph from the network from prior knowledge that holds high data fidelity. Specifically, we first obtain a hypothesized causal graph by performing importance analysis on the neural network or leveraging prior knowledge, and then split the func- tional causal model into causal term, residual term, and noise term. The split model can naturally generate time-series matching the original data observations well. It is worth noting that the re- trieval of the causal graph is not a causal discovery process and does not necessarily uncover the underlying causal relationship. but can produce realistic time-series serve as the benchmark of causal discovery algorithms. Our benchmark is open-source and user-friendly, we host our website at www. causal 1 t time..... dc. Specifically, our contributions include:
We propose CausalTime, a pipeline generate realistic time-series with ground truth causal graphs. which can be applied to diverse fields and provide new choices for evaluating TSCD algorithms.
We perform qualitative and quantitative experiments to validate that the generated time-series preserves the characteristics of the original time-series.
We evaluate several existing TSCD algorithms on the generated datasets, providing some guide- lines for algorithm comparison, choice, as well as improvement.
2 RELATED WORKS
Causal Discovery. Causal Discovery (or Causal Structural Learning), including static settings and dynamic time-series, has been a hot topic in machine learning and made big progress in the past decades. The methods can be roughly categorized into multiple classes. (i) Constraint-based approaches, such as PC (Spirtes& & Glymour] 1991). FCI (Spirtes et al. 2000), and PCMCI (Runge) et al., 2019b; Runge 2020 Gerhardus & Runge 2020), build causal graphs by performing con- ditional independence tests. (ii) Score-based learning algorithms which include penalized Neural
Ordinary Differential Eqn.s and acyclicity constraint (Bellot et al. 2022 (Pamfil et al.) 2020). (iii) Approaches based on Additive Noise Model (ANM) that infer causal graph based on additive noise assumption (Shimizu et al. 2006 Hoyer al. 2008). ANM is extended by Hoyer et al. (2008) to
2

---

Published as a conference paper at ICLR 2024
nonlinear models with almost any nonlincarities. (iv) Cranger-causality based approaches. Granger
causality was initially introduced by |Ganger (1969) who proposed to analyze the temporal causal relationships by testing the help of a time-series
predicting another time-series. Recently, Deep Neural Networks (NNs) have been widely applied to nonlinear Granger causality discovery. (Wu
et al. 2022 Tank al.) 2022 Khanna & Tan 2020; Löwe et al. 2022 Cheng et al.) 2023b) (V)
Convergent Cross Mapping (CCM) proposed by Sugihara et al. (2012) that reconstructs nonlinear
state space for nonseparable weakly connected dynamic systems. This approach is later extended to
situations of synchrony, confounding, or sporadic time-series (Ye et al. 2015 Benkō et al. 2020,
Brouwer et al., (021). The rich literature in this direction requires effective quantitative evaluation and progress in this direction also inspires designing new benchmarking methods. In this paper, we propose to generate benchmark datasets using causal models.
Benchmarks for Causal Discovery. Benchmarking is of crucial importance for algorithm design and applications. Researchers have proposed different datasets and evaluation metrics for causal discovery under both static and time-series settings. (i) Static settings. Numerical, quai-real, and real datasets are all widely used in static causal discovery. Numerical datasets include datasets simu- lated using linear, polynomial, triangular functions (Hoyer et al. 2008 Mooij et al. 2011 Spirtes & Glymour, 1991; Zheng et al.] 2018); Quai-real datasets are generated under physical laws (e.g. double pendulum (Brouwer et (021)) or realistic scenarios (e.g- synthetic twin birth datasets (Geffner et al.) 2022). alarm message system for patient monitoring (Scutaril [0100] Lippe et al., 2021). neural activity data (Brouwer et al. 202I), and gene expression data (Van den Bulcke al., 2006);; Real datasets are less frequently used. Examples include "Old Faithful" dataset on volcano cruptions (Hoyer et al. 2008), and expression levels of proteins and phospholipids in human im- mune system cell (Zheng et al. 2018). Recently, Göbler et al. (2023) proposes novel pipeline, i.e., causal/sseebbly, generating realistic and complex assembly lines in manufacturing scenario. Chevalley et al. (2023a) and Chevalley et al. (2023b) on the other hand, provides CausalBenchh set of benchmarks on real data from large-scale single-cell perturbation. Although causal Assembly and CausalBench are carefully designed, they are restricted in certain research fields where the dynam- ics can be easily replicated or the ground truth causal relationships can be acquired by performing interventions. (ii) Time-series settings.
In time-series settings. widely used numerical datasets include VAR and Lorenz-96 (Tank et al. 2022 Cheng et al., 2023b Khanna & Tan
2020) Bellot
et al. 2022); ; quasi-real datasets include NetSim Löwe et al. 2022), Dream-3/Dream ("Tnn et al.,
2022). and finance dataset simulated using Fama-French Three-Factor Model (Nauta et al.) 2019);
real datasets include MoCap dataset for human motion data (Tank et al. 2022). S&P 100 stock data
(Pamfil et al. 22020), tropical climate data (Runge et al.) (019b)), and complex ecosystem data (Sug- ihara et al.] al.]]_12111 2012). Other than these datasets. there are several works providing novel benchmarks with ground-truth causal graphs. CauseMe (Runge et al. 2020; 2019a) provides a platforn for nu- merical, quasi-real, as well as real datasets, which are mainly based on TSCD challenges on climate scenarios. However, although the platform is well-designed and user-friendly, it did not alleviate the tradcoff among fidelity, ground truth availability, and domain generalizability. For example, the nu- merical datasets in CauseMe are still not realistic, and the ground truth causal graphs for real datasets are still based on domain prior knowledge that may not be correct. (Lawrence et al. 202I) focuses on generating time-series datasets that go beyond CauseMe. Their framework allows researchers to generate numerous data with various properties flexibly. The ground-truth graphs for their gener- ated datasets are exact, but the functional dependencies in Lawrence al. (2021) are still manually designed and may not reflect real dynamics in natural scenarios. As a result, their generated datasets are still categorized into numerical datasets in Tab. [ although with far higher flexibility.
Recently, neural networks have been extensively studied for their capability of generating time-series Yoon et al. 2019 Jarrett et al. (202T): Pei et al. (202T): : Kang et al.) (2020); Zhang et al. (2018) 8); EE-
teban et al. (2017). However, the time-series generated with these methods are improper for bench- marking causal discovery, since causal graphs are not generated alongside the series. Therefore, we
propose a pipeline to generate realistic time-series along with the ground truth causal graphs.
causeme.net
3

---

Published as a conference paper at ICLR 2024
Real Time-series
Fit with NAR Model
Construct ACG and Generate Time-series
Netural Scenarios
Original Time series
LSTM
Normalizing Flow
Actual Crusal Graph (ACS)
Auto vegrssseesvvve Goncrotion
Generated Residuai
Timeseriz
Prior Graph
[Optional]
Quality Control
Prior Graph
Importance
Causal Dim. Visualization
Hypothesis
Discriminative Score
Qualitty
Generated
DeepSHNP Extraction
Traffic
Deloatss with Prioz Gruph
Clasttler
Figure I: Illustration of the Causal pipeline. We fit the observations from real scenarios with NAR model, then split the model and reorganize the components construct actual causal graph (ACG) that can generate time-series resembling the input observations. After visual and quantitative quality control, the synthesized time-series and corresponding ACG serve benchmark to evaluate the performance of TSCD algorithms in real applications.
3 THE PROPOSED TIME-SERIES GENERATION PIPELINE
Aiming generating time-series that highly resemble observations from real scenarios, and with ground truth causal graphs, propose general framework to generate causal graph built from the real observations and generate counterpart time-series that highly resemble these observations. The whole pipeline consists of several key steps, illustrated in Fig. TT Although the real causal graph for the original time-serics unknown, the built causal graph serves as the ground truth lying under the generated version, and they together serve benchmark for the TSCD algorithms, shown in Fig. 2 We would like to clarify that, our generation pipeline based on several assumptions that common in causal discovery literature: markovian condition, faithfulness, sufficiency, instantaneous effect, and stationarity. We place the detailed discussion of these assumptions in Supp. Section A.I.I due to page limits.
3.1 CAUSAL MODEL
Causal models in time-series are frequently represented graphical models (Vowels et al. 2021 Spirtes et al. 2000). However, different from the classic Pearl's causality (Pearl) (009)). spatio- be taken into account for time-series data. We denote uni- formly sampled observation of a dynamic system as X { J.N in which æ; is the sample vector time point and consists of N variables { with t [1, ..., T} and i fl..... N). The structural causal model (SCM) for time-series (Runge et al. (019b) is xg,5 fa (P(z),)) m.,s) i 1, 1, N. where /s any (potentially) nonlinear function, 7/t, denotes dynamic noise with mutual independence, and (xx²) denotes the causal parents of x.e- This model is generalizable to most scenarios, but may bring obstacles for our implementation. In this paper, we consider the nonlinear autoregressive model (NAR), a slightly restricted class of SCM.
temporal structural dependency
Nonlinear Autoregressive Model. We adopt the representation in many time-series causal discov- algorithms (Tan et al. 2022. Löwe et al.) 2022] Cheng et al.) (023b)), as well as Lawrence Nonlinear Autoregressive Model (NAR), the noise is assumed to be independent and additive, and each sampled variable æz, is generated by the
al. (202I's 's time-series generation pipeline.
following equation:
I2, fo (P(x.,)) + Th.i. i= 11,,, 1,2,..,,
(I)
where P(.) denotes parents in causal graph. We further assume that the maximal time lags for causal effects are limited. Then the model can be denoted as Tτ, fi (f sPP,,)) + M,,i- Here t r < Tmax Vær.i E P(rr.)). and Tima denotes the maximal time lag. In causal discovery, homo-eemygyneity (Gong et al. 2023) is often assumed, i.e., function fi and causal parents P is irrelevant to time. By summarizing temporal dependencies, the summary graph for causal models
11 ST,8 E P(ra,)) The can be denoted with binary matrix A, where its element aj.- 0. otherwise
dataset pair for causal discovery is (X, TSCD targets recover matrix A given time-series
4

---

Published as a conference paper at ICLR 2024
X. However, since for most real time-series X, causal graph A is unknown, benchmarking causal discovery algorithms with real time-series is generally inappropriate.
3.2 SEMEISERIES FITTING
After collecting real-time-seriet from diverse fields, fit the dynamic process of multivariate time- series with a deep neural network and normalizing flow.
Time-series Fitting with Causally Disentangled Neural Network (CDNN). To fit the ob- served time-series with a deep neural network and introduce casual graphs into the network's pre- diction of output series, Tank et al. (2022): Khanna & Tan (2020); Cheng et af. (2023b) separate the causal effects from the parents to each of individual output series using N separate MLPs / LSTMs, which is referred to as component-wise MLP / LSTM (cMLP cLSTM). In this pa- per, we follow (Cheng et al. 2023a)'s 's definition and refer to the component-wise neural net- works as "causally disentangled neural networks" (CDNN), i.e., neural networks taking the form (X, = [fee (X @ .)..)...., fe.. (X @xa,))] an,")", where A E {0, and the operator @ is defined as Soy [X @ a..))) Joj ((x1 aij..... XN (Nz)).
So far function fe, (.) acts as the neural network function used to approximate f-() in Eqn. ① Since we assume no prior on the underlying causal relationships, to extract the dynamics of the time- series with high accuracy, we fit the generation process with all historical variables (with maximal time lag Tmax, which is discussed in Supp. Section A.I.T) and obtain fully connected prediction model. Specifically, we assume that
F2,s fi (Xt-rit 1,1,
1,N) + Ma,s i 1,,,..., N.
(2)
In the following, we omit the time dimension of x --r-(-1.1 11 and denote it with x,, Using a CDNN, we can approximate f.). with fe, (-). (Tank et al., 2022) and (Cheng et al.) 2023b) implement CDNN with component-wise MLP / LSTM (cMLP7 cLSTM), but the structure is highly redundant because consists of N distinct neural networks.
Implementation of CDNN. The implementation of CDNN can vary. For example, Cheng et al. (2023a) explores enhancing causal discovery with a message-passing-bsse neural network, which special version of CDNN with extensive weight sharing. In this work, we utilize an LSTM- based CDNN with a shared decoder (with implementation details shown in Supp. Section [A.2). Moreover, perform scheduled sampling (Bengio al. 2015) to alleviate the accumulated error when performing autoregressive generation.
Noise Distribution Fitting by Normalizing Flow.
After approximating the functional term fe(-) with fe.()). we then approximate noise term T4, with Normalizing Flow (NF)
(Kobyzev al.] 2021 Papamakarios et al.) (202)) The main process is described îe, (u), where u ~ (u). in which Twx (-) is an invertible and differentiable transformation im- plemented with neural network, Pu (u) is the base distribution (normal distribution in our pipeline), and Pa: (M..a) = Pa. (TTp) (fr.)) à T-1 (fe..) Then, the optimization problem can be formulated as Dijes #1
a max /, MN logPa(TTT) dhe (nc.)) + log am.. v. (T))))
3.3 EXTRACTION OF HYPOTHETICAL CAUSAL GRAPH
In the fully connected prediction model, all N variables contribute to each prediction, which fits the observations quite well but is over-compleen complicaee than the latent causal graph. We now proceed to extract hypothetical causal graph (HCG) H by identifying the most contributing variables in the prediction model fe; (-). We would like to clarify that, extracting HCG is not causal discovery, and it instead targets to identify the contributing causal parents while preserving the fidelity of the fitting model. Two options are included in our pipeline: i) HCG extraction with DeepSHAP; ii) HCG extraction with prior knowledge.
HCG Extraction with DeePSHAP. Shapley values (Sundararajan Najmi 2020) are frequently used to assign feature importance for regression models. It originates from cooperative game the- ory (Kuhn Tucker 2016), and has recently been developed to interpret deep learning models (DeepSHAP) (Lundberg & Leee 2017 Chen et al. 2022). For each prediction model fe, (+), the
5

---

Published as a conference paper at ICLR 2024
calculated importance of each input time-series Xt-r:-11, by DeepSHAP is Øjs. By assigning im- portance values from each time-series j to i, we get the importance matrix b. After we set the be calculated with cumulative distribution, i... F (a),
sparsity of a HCG, threshold where (-) is the cumulative distribution of Φjt if assume all ΦJz are i.idd.
HCG Extraction with Prior Knowledge. Time-series in some fields, e.g., weather or traffic. relationships be- tween each variable are highly relevant to geometry dis- tances. For example, air quality or traffic flows in cer- tain area can largely affect that in nearby area. As a result, geometry graphs can serve as hypothetical causal graphs (HCGs) in these fields, we show the HCG calcu- lation process in Supp. Section A.2 for this case.
Original Timee-series
Real Causal Graph [Unknown)
Cousiity
Aporosimation Goncration
Generated Time
Time-series series
Hypothesis Causal Graph
(HCG)
Approdmane
The extracted HCGs is not the ground truth causal graph of time-series X. because time-series X not gen- erated by the corresponding NAR or SCM model (shown in Fig. 2 A trivial solution would be running auto- regressive generation by setting input of non-causal terms
spating
Generated - Residual
Actual Causal Graph (ACG)
Quueltyy
to zero, i.e.,
lime 3k
fr, =fe, (81- hii..... & 1 hNr) f/t... = 1.,...,,N N.
Figure 2: Illustration of how CausalTime gets 2N x 2N shaped sparse causal graph that exactly synthe- sizes the desired data.
where { } is the entries in the HCG H. However, the fidelity of the fitting model fe, (-) in Eq. Bis hampered by only including a subset of the variables. In the following, introduce another way to generate time-series with actual causal graph, and most importantly, without losing fidelity.
3.4 SPLITTING THE NAR MODEL TO ACQUIRE ACTUAL CAUSAL GRAPH AND REALISTIC TIME-SERIES
To acquire the Actual Causal Graphs (ACGs) and generate new time-series with high data fidelity, we propose to split the NAR model taking the form of fully connected prediction model into causal term, residual term, and noise term. In this manner, we do not have to identify exact causal relationships before generating new time-series, and hence avoid the dilemma of "using a TSCD algorithm to build synthetic dataset to test TSCD algorithms". Specifically,
fr,i fe. (81-1:t-111 hi...... & hNs)) +
M
(4)
causal term
residual term
moüse term
where the residual term #8-1,6 indicates the "causal effect" of non-parent time-series of time-series i in HCG H. In other words, causal terms represent the "major parts" of causal effects, and the residual term represents the remaining parts. Mathematically, 4R 12-1, is calculated as
f = fe. (8-- & - fe, (8t-rit-1,1 11 & hNi). (5)
and is then stored as an independent time-serics. When treated as a generation model, X, > x3 contains instantancous effects,, however, which does not affect the causal discovery result xf + x; for most existing TSCD approaches, as we discussed in Supp. Section A.1.2 Indeed, combining Eqn.s 4 and 5 all components of x,-1 However, the "dense" prediction models in our pipeline do not imply that the "natural" causal graphs dense. The exact form of the "natural" causal graph are unknown since we do not perform TSCD.
included to predict X4, i.e., "dense" prediction model.
After randomly selecting an initial sequence fo from the original time-series X, x; and x is generated via the rego-segressive model, i.c., the prediction results from the previous time step are used for generating the following time step. Our final generated time-series include all x, and xR i.e., a total of 2N time-series are generated, and the ACG Á is of size 2N x 2N:
x x= &easto +T: & BBooott]
A
H
JN
(6)
In
0
2N

---

Published as a conference paper at ICLR 2024
where JN is an all-one matrix, IN is identity matrix, T is the total length of the generated time- series. Although ground truth causal graphs for the original time-series are unknown, but for the generated version the causal graph is known and unique, shown in Fig. 2
With the prediction model fo.. normalizing flow model T and ACG A, we can obtain the final dataset X A X then feeded to TSCD algorithms to recover matrix given time-series X.
EXPERIMENTS
In this section, we demonstrate the CausalTime dataset built with the proposed pipeline, visual- ize and quantify the fidelity of the generated time-series, and then benchmark the performance of existing TSCD algorithms on CausalTime.
4.1 STATISTICS OF THE BENCHMARK DATASETS
Theoretically, the proposed pipeline is generalizable diverse fields. Here we generate 3 types of benchmark time-series from weather, traffic, and healthcare scenarios respectively, as illustrated in Fig. 3 As for the time-series of weather and traffic, relationships between two variables are highly relevant to their geometric distances, i.c., there exists prior graph, while there is no such prior in the healthcare series. The detailed descriptions are Supp. Section
AQI Dataset
Traffic Dataset
Medical Dataset
Figure 3: Visualization of three subsets in CausalTime For AQI and Traffic, overlay the ground truth causal graphs onto the map.
4.2 FIDELITY OF THE GENERATED TIME-SERIES
To qualitatively and guantitatively analyze the fidelity of the generated time-series, utilize PCA Bryant Yarnold (1955) and t-SNE van der Maaten & Hinton (2088 dimension reduction visu- alization, neural-network-baaie based discriminative score, and MMD score of real and synthetic feature vectors to evaluate whether our generated time-series is realistic.
Visualization via Dimension Reduction. To judge the fidelity of the generated time-series, project the time-series features to two-dimensional space, and assess their similarity by comparing the dimension reduction results. After splitting the original and generated time-series into short sequences (length of 5), we perform dimension reduction via linear (PCA) and nonlinear (t-SNE) approaches three generated datasets and visualize the difference explicitly, as shown in Fig. One can observe that the distributions of the original and generated series are highly overlapped, and the similarity especially prominent for AQI and Traffic datasets (i.e., the 1st, 2nd, 4th, and 5th columns). These results visually validate that our generated datasets are indeed realistic across a variety of fields.
(e)
(b)
Figure 4: Visualization of the similarity between generated and original time-series in a low dimen- sional (2D) space, where original and generated series are shown blue and red, respectively. (a) and (b) plots the two components of PCA and t-SNE on three datasets.
Discriminative Score MMD Score. Other than visualization in low dimensional space, further assess the generation quality, i.e., evaluate the similarity between the original and

---

Published as a conference paper at ICLR 2024
generated time-series, quantitatively using a neural-network--base based discriminator and the MMD score. For the neural-network- discriminator, by labeling the original time-series as pos- itive samples and the generated time-series then report the discriminative score in terms of |AUROC - 0.5| on the test set. MMD is frequently used metric to evaluate the similarity of two distributions (Gretton et al., 2006). 2
negative ones, we train an LSTM classifier and
It is estimated with MMD,
a(n-I) Σ Ej (2) mn Σ Ej=1 K (2 8/) +
mi De Eite K(Js; Vj), where K is the radial basis function (RBF) kernel. MMD gives another quantitative evaluation of the similarity without the need train another neural network, as listed in the bottom row of Tab. 2 It is observed that the generated dataset is similar to the origi- nal ones, since the discriminative score is very close to zero (i.e., neural networks cannot distinguish generated samples from original samples), and the MMD score is relatively low. Other than discrim- inative score and MMD, we also utilize cross-correlation scores and perform additive experiments, which is shown in Supp. Section A.3
Table 2:
Quantitative assessment of the similarity between the generated and original time-series in terms of discriminative score and MMD. We show the ablation study in the table as well.
Discriminative Score
MMD
Datascts
AQI
Traffic
Medical
AQI
Traffic
Medical
Additive Gaussian Noise w/o Noise Term
00,888 0.01 0.499 i 0.000 0.445 1 0.003
0.533 :: 0.091 0.716 0.11 0.480 1 0.07
0.361 0.0s 0.391
= 0.06 0.346
+ 0.011
0.454
= 0,25 0.717
000 0.453 + 0.229
Fit w/o Residual Term
0.309 0010 0.500
0.000
0,482
0.005
0.474
0,033
0,858
1 0.00 0.520 0.00
Generate w/o Residual Term
0.361 = 2014 0.371
= 0.003
0.348
+ 0014
0.431
= 0,05s
0.657 + 0.16 0.489 +: 0.077
Full Model
0.054 0,025 0.039 0,020 0.017 :: 0.027 0.246 0,029 0.215 0,013 0.461 0,33
Ablation Study. Using the above two quantitative scores, we also perform ablation studies to justify the effectiveness of our design. In the time-series fitting, we use normalizing flow to fit the noise distributions. To validate its effectiveness, we replace normalizing flow with (a) additive Gaussian noise (with parameters estimated from real series) and (b) no noise. Further, to validate that our pipeline reserves the real dynamics by splitting the causal model into causal term, residual term, and noise term, we add two alternatives that do not include the residual term when (c) fitting the NAR model or (d) generating new data, besides the above two settings.
The results are shown in Tab. which shows that the full model produces time-series that mimic the original time-series best, in termss of both discriminative score and MMD score. The only exception the slightly lower MMD under settings "w/o Noise" on medical datasets. It worth noting that our discrimination scores are close to zero, i.e., neural networks cannot discriminate almost all generated time-series from original versions.
4.3 PERFORMANCE OF THEE-AT CAUSAL DISCOVERY ALGORIHTMMS
To quantify the performances of different causal discovery algorithms, here we calculate their AU- ROC and AUPRC with respect to the ground truth causal graph. We do not evaluate the accuracy of the discovered causal graph A with respect to its ground-truth Å, because there exists instantaneous effects in x, + x (see Supp. Section A.1.2). Instead, we ignore the blocks IN Ju and 0 in Eqn. 6), and compare H with respect H,
Baseline TSCD Algorithms. We benchmarked the performance of 13 recent or representative causal discovery methods on our CausalTime datasets, including: i) Granger-causalitybbasse ap- proach: Granger Causality (GC, [Granger, 1969)). Neural Granger Causality (NGC, (Tank et al.) p. economy-SRU (eSRU, (Khanna & Tan 2020p. Scalable Causal Graph Learning (SCGL, Xu et al.] 2019). Temporal Causal Discovery Framework (TCDF, (Nauta et al.] 2019)). ). CUTS
Cheng et al., (2023)), CUTS+ (Cheng et al. 2023a) upgrading CUTS to high dimensional time- series. ii) Constraint-based approaches: PCMCI (Runge et al. 2019b), SVAR, NTS-NOTEARS (shown as N.NTS, (Sun et al. (023)), and Rhino (Gong et al. 2022), in) CCM-based approaches: Latent Convergent Cross Mapping (LCCM, (Brouwer et al. 2021)) and iv) Other approaches: Neu- ral Graphical Model (NGM, (Bellot et al.) (022)). which employs neural ordinary differential equa- tions to handle irregular time-series data. To ensure fairness, we searched for the best set of hyperpa-
8

---

Published as a conference paper at ICLR 2024
rameters for these baseline algorithms on the validation dataset, and tested performances testing sets for random seeds per experiment.
Table 3: Performance benchmarking of baseline TSCD algorithms on our CausalTime datasets. We highlight the best and the second best in bold and with underlining, respectively.
AUROC
AUPRC Traffic
Methods
AQI
Traffic
Medical
AQI
Medical
GC
0.4538 + 0.0771
0.4191
0.010
0.5737 = 0.3338
0.6347 + 0.1588
0.2789
+
0,4213 # 0.0811
SVAR
0.6225 + 0.0466
0.6329
0.042
0.7130 = 0.018
0.7903 + 0.0755
0,5845
00021
0,6774 + 0.05s8
N.NTS
0,5729 i: 0.0229
0,6329
0,335
0,5019
0.0682
0.7100 1: 0,0288
0,5770
1-02212
0,4567 1: 0.0162
PCMCI
0.5272 0.741
0,5422
0.7377 0,6991
0.0111
0.6734 4: 0.0372
0.3474
00681
0.5082 0.0177
Rhino
0.6700 +: 0,0933
0.6274
0.0U85
0.6520
:: 0.0212
0.7593 +: 0.015s
0.3772
0.0933
0.4897 1: 0.0211
CUTS
0.6013 = 0.00a8
0.6238
= 0:0179
0.3739
= 0.0297
0.5096
+ 0.1622
0.1525
+ 00226
0.1537 + 0.000
CUTS+
0.8928 + 8,0233
0.6175
0.0755
0.8202
0.0733
0,7983 + 0.0875
0.6367
0.1197
0,5481
= 0.149
NGC
0.7172 = 0.0076
0.6032
0.0056
0.5744
0.0096
0.7177
+ 0.0669
0.3583
0049 0.4637
0.02221
NGM
0.6728 = 0.0164
0.4660
0.0144
0.5551
= 0.0154
0.4786
+ 0.196
0.2826
0.098
0.4697 : 0.0166
LCCM
0.8565 + 0.0633
0,5545
0.0254
0.8013
+ 0,4218
0.9260 + 0.0246
0,5907
+ 0047
0.7554 + 0.0235
eSRU
0.8229 + 0:0017
0,5987
0.0922
0.7559
# 0.0365
0.7223 + 000117
0,4886
+ 000ER
0.7352 + 0.0000
SCGL
0.4915 + 0,0466
0.5927
0.0533
0.5019
0.0224
0.3584
+ 0u0281
0,4544
00815
0,4833 0.088
TCDF
0.4148 0.0007 0.5029
0.0011
0.6329
0.0344
0.6527 1 0.0087
0.3637
0,008 0.5544 0,0013
Results and Analysis. From the scores in Tab. 3 one can see that among these algorithms, CUTS+ and LCCM perform the best, and most of the TSCD algorithms do not get AUROC > 0.9. The low accuracies tell that current TSCD algorithms still have a long way to go before being put into practice and indicate the necessity of designing more advanced algorithms with high feasibility to real data. Interestingly. a few results demonstrate AUROC < 0.5, which means that we get inverted classifications. Constraint-based approaches such as PCMCI achieve high performance on synthetic datasets (especially linear cases, (Cheng et al. )0236) but perform worse on realistic data. This is probably due the difficulty of conducting conditional independence tests in nonlinear or realistic scenarios. Granger-causality- based approaches such as CUTS+ perform the best because of their neural network-based architecture. However, these methods, such as CUTS, CUTS+, NGC, eSRU, have limitations since they rely on the assumptions of causal sufficiency and no instantaneous effects. CCM-based approach LCCM performs surprisingly well although the performance is not that good in synthetic datasets (reported in Cheng al. (2023b)). This proves that learning Neural- ODE-based latent processes greatly helps the application of Convergent Cross Mapping. Besides, compared with the reported results in previous work (Cheng et al.) 2023b] Tank et al., (022), one can notice that the scores on our Causal dataset are significantly lower than those on synthetic datasets (e.g.. VAR and Lorenz-96), on which some TSCD algorithms achieve scores close to 1. This implies that the existing synthetic datasets are insufficient to evaluate the algorithm performance on
real data and calls for building new benchmarks to advance the development in this field.
Additional Information. We place theoretical analysis and assumptions in Supp. Section A.
implementation details along with hyperparameters for each of our key steps in Section A.2 ad-
ditional experimental results (including a comparison of various CDNN implementations, ablation study for scheduled sampling, and experimental results for cross-correlation scores and with missing entries or confounding) in Section A.3 and algorithmic representation in Section 1
5 CONCLUSIONS
We propose CausalTime, a novel pipeline to generate realistic time-series with ground truth causal graphs, which can be used assess the performance of TSCD algorithms in real scenarios and can also be generalizable to diverse fields. Our CausalTime contributes to the causal- causal-discovery community by enabling upgraded algorithm evaluation under diverse realistic settings, which would advance both the design and applications of TSCD algorithms. Our work can be further developed in multiple aspects. Firstly, replacing NAR with the SCM model can extend CausalTime to a richer set of causal models; secondly, we plan to take into account the multi-scale causal effects that widely exist in realistic time-series. Our future works include i) Creating a more realistic time- series by incorporating prior knowledge of certain dynamic processes, multi-scale associations, or instantaneous effects. ii) Investigation of an augmented TSCD algorithm with reliable results on real time-series data.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
For the purpose of reproducibility, we include the source code on GitHub (https://githui) com/ jan/jjarycyx (NN). The generated CausalTime datasets are shared at www. causaltime.
cc. Moreover, we provide all hyperparameters used for all methods in Appendix Section A.2 All experiments are deployed on a server with Intel Core CPU and NVIDIA RTX3090 GPU.
ACKNOWLEDGMENTS
This work jointly funded by Ministry of Science and Technology of China (Grant No. 2020AAA0108202), National Natural Science Foundation of China (Grant Nos. 61931012 and 62088102) 12), Beijing Natural Science Foundation (Grant No. 7200002), and Project of Medi- cal Engineering Laboratory of Chinese PLA General Hospital (Grant No. 2022SYSZ2KY21, No.2022YYSZKY1I1,
REFERENCES
Charles K. Assaad, Emilie Devijver, and Eric Gaussier. Survey and evaluation of causal discovery methods for time series. Journal of Artificial Intelligence Research, 73:767-819, February 2022.
ISSN 1076-9757. doi: (0.1613/jair..1133
Alexis Bellot, Kim Branson, and Mihaela van der Schaar. Neural graphical modelling in continuous- time: Consistency guarantees and algorithms. In International Conference on Learning Repre- sentations, February 2022.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. In Advances in Neural Information Processing Sys- tems, volume 28. Curran Associates, Inc., 2015.
Zsigmond Benkō, Ádám Zlatniczki, Marcell Stippinger, Dániel Fabó, András Sólyom, Loránd Bröss, András Telcs, and Zoltán Somogyvári. Complete Inference of Causal Relations between Dynamical Systems, February 2020.
Edward De Brouwer, Adam Arany, Jaak Simm, and Yves Moreau. Latent Convergent Cross Map- ping. In International Conference Learning Representations, March 2021.
Fred B. Bryant and Paul R. Yarnold. Principal-components analysis and exploratory and confirma- tory factor analysis. In Reading and Understanding Multivariate Statistics, pp. 99-136. American Psychological Association, Washington, DC, US, 1995. ISBN 178-1-55798-273-5.
Hugh Chen, Scott M. Lundberg, and Su-In Lee. Explaining series of models by propagating Shapley values. Nature Communications, 13(1):5512 August 2022. ISSN 2041-1723. doi: 10. 10.10088/44447-022--1333 -22-33222--
Yuxiao Cheng, Lianglogg Li, Tingxiong Xiao, Zongren Li, Qin Zhong, Jinli Suo, and Kunlun He. CUTS+: High-dimensional Causal Discovery from Irregular Time-series, August 2023a.
Yuxiao Cheng, Runzhao Yang, Tingxiong Xiao, Zongren Li, Jinli Suo, Kunlun He, and Qionghai Dai. CUTS: Neural Causal Discovery from Irregular Time-Series Data. In The Eleventh Interna- tional Conference on Learning Representations, February 2023b.
Mathieu Chevalley, Yusuf Roohani, Arash Mehrjou, Jure Leskovec, and Patrick Schwab. Causal- Bench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data, July 2023a.
Mathieu Chevalley, Jacob Sackett-Sanders, Yusuf Roohani, Pascal Notin, Artemy Bakulin, Dariusz Brzezinski, Kaiwen Deng, Yuanfang Guan, Justin Hong. Michael Ibrahim, Wojciech Kotlowski, Marcin Kowiel, Panagiotis Misiakos, Achille Nazaret, Markus Püschel, Chris Wendler, Arash Mehrjou, and Patrick Schwab. The CausalBench challenge: A machine learning contest for gene network inference from single-cell perturbation data, August 2023b.
10

---

Published as a conference paper at ICLR 2024
Andrea Cini, Ivan Marisca, and Cesare Alippi. Filling the G.ap.s: Multivariate time series im- putation by graph neural networks. In International Conference on Learning Representations, February 2022.
Cristóbal Esteban, Stephanie L. Hyland, and Gunnar Rätsch. Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs, December 2017.
Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, Miltiadis Allamanis, and Cheng Zhang. Deep End- to-end Causal Inference, June 2022.
Andreas Gerhardus and Jakob Runge. High-recal causal discovery for autocorrelated time series with latent confounders. In Advances in Neural Information Processing Systems, volume 33, pp. 12615-12625. Curran Associates, Inc., 2020.
Konstantin Göbler, Tobias Windisch, Tim Pychynski, Steffen Sonntag. Martin Roth, and Math- ias Drton. $ caunttttcauslllssmmblyy $: Generating Realistic Production Data for Benchmarking Causal Discovery, June 2023.
Chang Gong, Di Yao, Chuzhe Zhang. Wenbin Li. and Jingping Bi. Causal Discovery from Temporal Data: An Overview and New Perspectives, March 2023.
Wenbo Gong, Joel Jennings, Cheng Zhang, and Nick Pawlowski. Rhino: Deep Causal Temporal Relationship Learning with History-dependen. Noise. In The Eleventh International Conference on Learning Representations, September 2022.
C. W. Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica, 773)):444438 1969. ISSN 0012-9682. doi: 10.2307/1912791.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. Kernel Method for the Coo-Samlle-Probleen In Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008.
Stephanie Hyland, Martin Faltys, Matthias Hüser, Xinrui Lyu, Thomas Gumbsch, Cristóbal Es- teban, Christian Bock, Max Horn, Michael Moor, Bastian Rieck, Mare Zimmermann, Dean Bo- denham, Karsten Borgwardt, Gunnar Rätsch, and Tobias M. Merz. Early prediction circulatory failure in the intensive unit using machine learning. Nature Medicine, 26(3):364-377 March 2020. ISSN 1546-170X. doi: 10.
Daniel Jarrett, Ioana Bica, and Mihaela der Schaar. Time-series Generation by Contrastive Imitation. In Advances in Neural Information Processing Systems, volume 34, pp. 28968-28998 Curran Associates, Inc., 2021.
Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 10(1):1, January 2023. ISSN 2052-4463. doi: 10.
Yanfei Kang, Rob J. Hyndman, and Feng Li. GRATIS: GeneRAting Time Series with diverse and controllable characteristics. Statistical Analysis and Data Mining: The ASA Data Science Journal, 13(4):354-376, 2020. ISSN 1932-1872. doi: 10. 10.1002/samm 1461.
A. Karimi and M. R. Paul. Extensive chaos in the lorenz-96 model. Chaos: An Interdisciplinary Journal of Nonlinear Science, 20(4):043105, December 2010. ISSN 1054-1500. doi: 10. 10.1063/1. 3496397.
Saurabh Khanna and Vincent Y. F. Tan. Economy statistical recurrent units for inferring nonlinear granger causality. In International Conference on Learning Representations, March 2020.
11

---

Published as a conference paper at ICLR 2024
Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker. Normalizing Flows: An Introduc- tion and Review of Current Methods. IEEE Transactions on Pattern Analysis and Machine In- relligence, 43(11):396433979 November 2021. ISSN 1939-3539. doi: 10.1199/TPMML2000. 2992934.
Harold William Kuhn and Albert William Tucker. Contributions to the Theory of Games (AM-28), Volume II. Princeton University Press, March 2016. ISBN 000-1-4008-819--11
Andrew R. Lawrence, Marcus Kaiser, Rui Sampaio, and Maksim Sipos. Data Generating Process to Evaluate Causal Discovery Techniques for Time Series Data, April 2021.
Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal dis- covery in physical systems from videos. In Advances in Neural Information Processing Systems, volume 33, pp. 9180-9192. Curran Associates, Inc., 2020.
Phillip Lippe, Taco Cohen, and Efstratios Gavves. Efficient neural causal discovery without acyclic- ity constraints. In International Conference Learning Representations, September 2021.
Sindy Löwe, David Madras, Richard Zemel, and Max Welling. Amortized causal discovery: Learn- ing infer causal graphs from time-series data. In Proceedings of the First Conference on Causal Learning and Reasoning, pp. 509-525. PMLR, June 2022.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Joris M Mooij, Dominik Janzing, Tom Heskes, and Bernhard Schölkopf. On Causal Discovery with Cyclic Additive Noise Models. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
Meike Nauta, Doina Bucur, and Christin Seifert. Causal discovery with attention-based convo- lutional neural networks. Machine Learning and Knowledge Extraction, 1(1):312-340, March 2019. ISSN 2504-4990. doi: 10.3390/make111000
Roxana Pamfil, Nisara Sriwattanaworachai,, Shaan Desai, Philip Pilgerstorfer, Konstantinos Geor- gatzis, Paul Beaumont, and Bryon Aragam. DYNOTEARS: Structure learning from time-series data. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pp. 1595-1605. PMLR, June 2020.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1):57:2617-57-
January 2021. ISSN 1532-4435.
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, August 2009. ISBN 978-0-521-89560-6.
Hengzhi Pei, Kan Ren, Yuqing Yang. Chang Liu, Tao Qin, and Dongsheng Li. Towards Generating Real-Worle World Time Series Data. In 2021 IEEF International Conference on Data Mining (ICDM), pp. 469-478,
December 2021. doi: 10. 109/CCDM51129 220E112222222222222222222 1.00000
Robert J. Prill, Daniel Marbach, Julio Saez-Rodrigues Peter K. Sorger, Leonidas G. Alexopoulos, Xiaowei Xue, Neil D. Clarke, Gregoire AltnneBonnet, and Gustavo Stolovitzky. Towards a Rig- orous Assessment of Systems Biology Models: The DREAM3 Challenges. PLOS ONE, 5(2): e9202, 2010. ISSN 1932-6203. doi: 13. www.urnal..ooe..0099999
Alexander Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy to Game. In Advances in Neural Information Pro- cessing Systems, volume 34, pp. 27772-27784. Curran Associates, Inc., 2021.
Jakob Runge. Discovering contemporaneous and lagged causal relations in autocorrelated nonlinear time series datasets. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelli- gence (UAI), pp. 1388-1397. PMLR, August 2020.
12

---

Published as a conference paper at ICLR 2024
Jakob Runge, Sebastian Bathiany, Erik Bollt, Gustau Camps- Camps-Valls, Dim Coumou, Ethan Deyle, Clark Glymour, Marlene Kretschmer, Miguel D. Mahecha, Jordi Muñoz-Marí, Egbert H. van Nes, Jonas Peters, Rick Quax, Markus Reichstein, Marten Scheffer, Bernhard Schölkopf, Peter Spirtes, George Sugihara, Jie Sun, Kun Zhang, and Jakob Zscheischler. Inferring causation from time series in Earth system sciences. Nature Communications, 10(55:2553, June 2019a. ISSN 2041-1723. doi: 0011088841446-0011-0000-533
Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdinovic. Detecting and quantifying causal associations in large nonlinear time series datasets. Science Advances, (1) :e)aau49999 2019b. doi: 10.1 10.111 1266/ciadv..au44999
Jakob Runge, Xavier-Andoni Tibau, Matthias Bruhns, Jordi Muñoz-Mari, and Gustau Camps- Valls. The Causality for Climate Competition. In Proceedings of the NeurIPS 2019 Competition and Demonstration Track, pp. 110-120. PMLR, August 2020.
Jakob Runge, Andreas Gerhardus, Gherardo Varando, Veronika Eyring, and Gustau Camps- Valls. Causal inference for time series. Nature Reviews Earth & Environment, 4(7):487-505, July 2023. ISSN 2662- 138X. doi: 00058//44301-22-0044333333
Marco Scutari. Learning Bayesian Networks with the bulearn R Package, July 2010.
Shohei Shimizu, Patrik O. Hoyer, Aapo Hyv&#228, rinen, and Antti Kerminen. A Linear Non- Gaussian Acyelic Model for Causal Discovery. Journal of Machine Learning Research, 7(72): 2003-2030, 2006. ISSN 1533-7928.
Stephen M. Smith, Karla L. Miller, Gholamreza
Matthew Webster, Christian F. Beckmann, Thomas E. Nichols, Joseph D. Ramsey, and Mark W. Woolrich. Network modelling methods for FMRI. Neurolmage, (4(2):875589)) January 2011. ISSN 1053-8119. doi: 10.1016,
j.neuroimage.20 20100.8..6..
Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse causal graphs. Social science computer review, 91)::22-7, 1991.
Peter Spirtes, Clark N. Glymour, Richard Scheines, and David Heckerman. Causation, Prediction, and Search. MIT press, 2000.
George Sugihara, Robert May, Hao Ye, Chih-hao Hsieh, Ethan Deyle, Michael Fogarty, and Stephan Munch. Detecting Causality Complex Ecosystems. Science, 338(1106):496-500, October 2012. doi: 10. 1el11///eeneeee227000000
Xiangyu Sun, Oliver Schulte, Guiliang Liu, and Pascal Poupart. NTS-NOTEARS: Learning Non- parametric DBNs With Prior Knowledge. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, pp. 1942-1964. PMLR, April 2023.
Mukund Sundararajan and Amir Najmi. The Many Shapley Values for Model Explanation. In Proceedings of the 37th International Conference on Machine Learning, pp. November 2020.
PMLR,
Alex Tank, Ian Covert, Nicholas Foti, Ali Shojaie, and Emily B. Fox. Neural granger causality. IEEE Transactions Patter Analysis and Machine Intelligence, 44(8):4267-422) 4279, 2022. ISSN 1939-3539. doi: BLLL10//PAAM.2221136665652
Hans-Chrisste Thorsen-Meyer, Annelaura B Nielsen, Anna P Nielsen, Benjamin Skov Kaas- Hansen, Palle Toft, Jens Schierbeck, Thomas Strøm, Piotr J Chmura, Marc Heimann, Lars Dyb- dahl, Lasse Spangsege, Patrick Hulsen, Kirstine Belling, Søren Brunak, and Anders Perner. Dy- namic and explainable machine learning prediction of mortality in patients in the intensive care unit: A retrospective study of hhih-frreqenne data in electronic patient records. The Lancet Digital Health, 199)1119- April 2020. ISSN 3500. doi: 10. (50016555997755022)3300 8-2.
Tim Van den Bulcke, Koenraad Van Leemput, Bart Naudts, Piet van Remortel, Hongwu Ma, Alain Verschoren, Bart De Moor, and Kathleen Marchal. SynTReN: A generator of synthetic gene expression data for design and analysis structure learning algorithms. BMC Bioinformatics, (1):43, January 2006. ISSN 1471-2105. doi: 10.1186/1771-21055-2222
13

---

Published as a conference paper at ICLR 2024
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9(86):257 2605, 2008. ISSN 1533-7928.
Matthew J. Vowels, Necati Cihan Camgoz, and Richard Bowden. D'ya like dags? A survey on structure learning and causal discovery, March 2021.
Alexander P. Wu, Rohit Singh, and Bonnie Berger. Granger causal inference on dags identifies genomic loci regulating transcription. In International Conference on Learning Representations, March 2022.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph WaveNet for Deep TemporllT Graph Modeling, May 2019.
Chenxiao Xu, Hao Huang, and Shinjae Yoo. Scalable Causal Graph Learning through Deep Neural Network. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM '19, pp. 1853-1862, New York, NY, USA. November 2019. As- sociation for Computing Machinery. ISBN 978-1-4503-6976-
doi: 10.1 10.1145/35538843337666
Hao Ye, Ethan R. Deyle, Luis J. Gilarranz, and George Sugihara. Distinguishing time-delayed causal interactions using convergent cross mapping. Scientific Reports, 5(1):14750, October 2015. ISSN 2045-2322. doi: 10.1038/step14750
Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series Generative Adversarial Networks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Chi Zhang, Sanmukh Kuppannagari, Rajgopal Kannan, and Viktor Prasanna. Generative Adversarial Network for Synthetic Time Series Data Generation in Smart Grids. In 2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGriCCommm) pp. 1-6, October 2018. doi: 10.1 109/Smar(CrdCoom..... 0tt.:mmm..11..87744550
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. DAGs with NO TEARS:
Continuous optimization for structure learning. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
14

---

Published as a conference paper at ICLR 2024
A
APPENDIX
CONTENTS
A.1
Theory
15
A.1.1
Assumptions and Limitations
15
A. 1.2
Instantancous Eflect of Residual Term
16
A.1.3
Causality in The Generated Time-
16
A.2
Implementation Details
17
Additional Results
20
A.3.1
Time-series Fitting
20
A.3.2
Scheduled Sampling
20
A.3.3
Cross Correlation Scores for serics Generation
20
A.3.4
Variability of the Extracted HCG
20
A.3.5
TSCD Evaluation with Missing Entries and Latent Contounding
21
Algorithmic Representation for Causal Pipeline
A.1
THEORY
A.1.1 ASSUMPTIONS AND LIMITATIONS
CausalTime is a realistic benchmark to test various TSCD algorithms. As a result, we choose to include the most common assumptions, e.g., Markovian Condition, Causal Faithfulness, No instan- tancous effects, Causal Sufficiency, and Causal Stationarity.
Markovian Condition. The joint distribution can be factorized into P(x) = II, P(x.((())) i.c.
every variable is independent of all its nondescendants, conditional on its parents.
Causal Faithfulness. causal model that accurately reflects the independence relations present in the data.
Causal Sufficiency, (or no latent confounder) All common causes of all variables are observed. This assumption is potentially very strong since we cannot observe "all causes in the world" because it may include infinite variables. However, this assumption important for a variety of literature.
No Instantaneous Effect. (Or Temporal Priority (Assaad al. 2022)) The cause occurred before its effect the time-series. This assumption can be satisfied if the sampling frequency is higher than the causal effects. However, this assumption may be strong because the sampling frequency of many real time-series is not high enough. In Causal Time, testing methods that support instantaneous effects, like Rhino and DYNOTEARS Gong et al. 2022); Pammiil et al. (2020) is still possible by testing only the time-lagging parts. However, it is indeed a limitation that we cannot test the instantancous part.
Causal Stationarity. All the causal relationships remain constant throughout time. With this sumption, full time causal graph can be summarized into windowed causal graph (Assaad et al., 2022).
In AQI and Traffic datasets, causal relationships are highly relevant to geometry distances, since nearby stations have mutual influences, so the extracted HCG and the subsequent ACG are directly from distance graphs, which align with common sense and are widely used Cini et al.) (2022); Wu ct al. (2019). medical datasets, the HCG is extracted with DeepSHAP since is hard to build reliable graphs from only prior knowledge. Although Shapley values might not exactly match with causality, Shapley-value-ban based approaches are widely used in the field medicine and are shown to capture features with actual important relationships Hyland et al. (2020); Thorsen horsen-Meyer et al. 22020). Consequently, the extracted graphs in our three subsets are built on existing extensive studies and are expected to be close those found in nature.
15

---

Published as a conference paper at ICLR 2024
A.1.2 INSTANTANEOUS EFFECT OF RESIDUAL TERM
section 3.4 we propose to split the NAR model into causal term, residual term, and noise term, where the residual term w;_ 1 is generated with
R 1 1, = Se. (8t-ra-
& Sea. (Re-ra-1,) his, Re-ri- hNi)
As a result, instantancous effects exist in this generation equation. i.e., in X, + x but not in x, - x,, ji This is not a problem when tested on TSCD algorithms with compatibility with instantaneous effects. In the following, we discuss the consequences when tested on TSCD algo- rithms without compatibility Mapping, and PCMCI (Runge al. 2023). For most cases, this does not affect causal discovery results between all x,. Though they may draw a wrong conclusion of Xj + x this not the part compare to ground truth in the evaluation (Section 4.3)
instantaneous effects, e.g. Granger Causality, Convergent Cross
Granger Causality. Granger Causality determines causal relationships by testing if time-series helps the prediction of another time-series. In this case, the causal discovery results of x - x, not affected since instantaneous effects of xj + x do not affect whether the time-series X, helps to predict the time-series x,, Vi
Constraint-based Causal Discovery. These line of works is based conditional independence tests, i.c., if Xta.f + Xt. then 2oo.5 HK It, ( 1/ where t t S to < t (Runge et al.) (2019;; Rungel (020).. This relationship is unaffected when considering instantaneous effect of V1.j 22,1- since all paths from Lto.j to Ft.; through o are blocked by conditioning on xi Vi,j, k, as shown in Fig. 5
CCM-based Causal Discovery. CCM detects if time-series x causes time-series y by examine whether time indices nearby points on the y manifold can be used to identify nearby points on x (Sugihara et al. 2012 Brouwer et al. (2021). In this case, the examination of whether x, X, is not affected by x Vi,jkk
Hypothesis Causal Graph (HCG)
Actual Causal Graph
8,
X1
x!
Time Siice
Lagged Effects
Instaneous Effects
X
Figure 5: Illustration of instantaneous effects of Xj are blocked by conditioning on x Vi,j,, k.
xR
All paths from Tro to Tei through x.R
A.1.3 CAUSALITY IN THE GENERATED TIME-SERIES
The intention of CausalTime is to generate time-series (i) with unique ground truth causal graph for benchmarking and (ii) highly resembles the distributions of real data for evaluating the algorithm performance on real data. We achieve this in three steps:
1. Extracting HCG in Section 3.3 Here HCG is not the ground truth causal graph since it only reflects the "major part" of the causal relationship.
2. Split the NAR model into causal terms, residual terms, and noise terms. Here causal terms also represent the "major parts" of the causal relationships, and residual term represents the remaining parts.
3. Treat the residual terms R as independent time-series. As result, there exists 2N instead of the original N time-series. be visualized in Fig.
series. And the graph (ACG) becomes 2N x 2N, as Eqn. This process can
16

---

Published as a conference paper at ICLR 2024
In this manner, combining Equ.s 6 and we get the original NAR form. Also, ACG is the ground truth causal graph since treat the residual terms a as independent time-series. Hence, ground truth causal graphs for the original time-series are unknown, but for the generated version the causal graph known and unique, i.e., the ACG. As result, avoid facing the dilemma of "using TSCD algorithm to build synthetic dataset to test TSCD algorithms". We show this relationship in Fig. 2
assure that 24,5 is still generated by exactly the original NAR model since
A.2 IMPLEMENTATION DETAILS
Original Time-series. In CausalTime, we generate 3 types of benchmark seme-series from weather, traffic, and healthcare scenarios respectively. The original time-series are
1. Air Quality Index (AQI) is subset of several air quality features from 36 monitoring stations spread across Chinese citie with an hourly measurement over one year. We consider the PM2.5 pollution index in the dataset. The total length of the dataset is I. 8760 and the number of nodes N = 36. We acquire the prior graph by computing the pairwise distances between sensors (Supp. Section A.2)
Traffic subset built from time-series collected by traffic semsors in the San Francisco Bay A The total length of the dataset is L 52116 and we include 20 nodes, i.e., N = 20. The prior graph also calculated with the geographical distance (Supp. Section A..)
3. Medical subset is from MIMIC-4, which is database that provides critical care data for over 40,000 patients admitted to intensive care units (Johnson et al. 2023). We select 20 most fre- quently tested vital signs and "chartevents" from 1000 patients, which are then transformed into time-series where each time point represents a 2-hour interval. The missing entries are imputed using the nearest interpolation. For this dataset, prior graph is unavailable because of the ex- tremely complex dynamics.
Network Structures and Training. not the only choice. For example,,heel et al. (2023a) explores enhancing causal discovery with message-passing-aase neural network, which is a special version of CDNN with extensive weight sharing. However, the fitting accuracy of CDNN less explored. Sharing partial weights may alleviate the structural redundancy problem. Moreover, the performance with more recent structures unknown, e.g. Transformer. the following, this work investigates various implementations of CDNN when they are applied to fit causal models.
The implementation of CDNN can vary and cMLP / cLSTM
Specifically, three kinds of backbones combining three network sharing policies are applied, i.c., MLP, LSTM, Transformer combining no sharing, shared encoder, and shared decoder. For MLP, the encoder and decoder are both MI.. For LSTM, we assign an L.ST encoder and an MI.P decoder. For the Transformer, we assign Transformer encoder and MLP decoder. We show the structure for no sharing, shared encoder, and shared decoder Fig. 6 with a vhrialvariable example. The test results for prediction using these architectures are shown Tab.
No Weight Sharing
Masking
Encoder
Deuzdder
Encoder
Decoder
Encodier
Decoder
Share Encoder
Decoder
Masking
Encoder
Decoder
Masking
Encoder
Becoder
Encoder
Deunder
Enaoder
Decuder
Encoder
Decoder
Encoder
Decoder
Figure 6: Illustration of network structures of CDNN.
hhtp:///wwwwelooccccccceeeeeeeer httpps:/pemms.ot..ca. dot.
17

---

Published as a conference paper at ICLR 2024
Table 4: Hyper parameters for time-series fitting.
Module
Parameter
AQI
Traffic
Medical
Layers
2
LSTM Encoder
Hidden
128
128
128
Heads
4
4
Layers
MI.P Encoder
Hidden
128
128
128
Hidden
128
128
128
Transformer Encoder
Heads
4
Layers
3
3
Decoder
Hidden
128
128
128
Learning Rate
0.01
0.001
0.003
Training
Optimizer
Adam
Adam
Adam
Input Window
20
20
20
Layers
Normalizing Flow
Hidden
128
64
Normalizing Flow. We implement normalizing flow using its open-source repository" We use normal distributions base distributions. For transformations, we use simple combinations of linear and nonlinear layers, with parameters shown in Tab.
DeePSHAP. We use the official implementation of DeepSHAF Specifically, use its "DeepEx- plainer" module to explain the time-series prediction model which is trained with a fully connected graph. Note that this prediction model is CDNN, which enables pairwise explanation of feature i's importance to the prediction of j. The explained samples are randomly selected from train set of real time-series, and the final feature importance graph acquired by taking the average values of all samples. To convert to binarized HCG, we select a threshold to get a sparsity of 15% (i.e., 15% of the elements in HCG labeled as 1).
Prior Graph Extraction. For datasets with prior knowledge, e.g., AQI and Traffic, relationships between each variable are highly relevant geometry distances. Consequently, we extract HCG from the geographic distances between nodes using thresholded Gaussian kernel, i.e.,
L,
dist(i,,) S
(8)
0.
otherwise
we select a based on geographical distances (which is = 40 km for AQI dataset and for Traffic dataset, we use the "dist graph" from httts:// com wttpc////yoogggg//////////eeee master).
Dimension Reduction. For the implementation of t-SNE and PCA, we use scikit-learr package. saack-- To solve the dimension reduction in an acceptable time, we split the generated time-series into short sequences (with lengths of 5) and flattened them for the input of dimension reduction.
Autoregressive Generation. After fitting the time-series with neural networks and normalizing flow, and acquiring the ground-truth causal graph by splitting the causal model, we generate new time-series autoregressively. Although we utilize scheduled sampling avoid the accumulation of the generation error, the total time step must be limited to a relatively small one. Actually, our generated time length is 40 for these three datasets. For each of them, generate 500 samples, i.c., a total of 20000 time steps for each dataset.
hhttps://github... wwtpcauuuu..oomm/eeeeeeeeeeeeeee com/shap/shap hhtpss//scikit-eamm....
18

---

Published as a conference paper at ICLR 2024
Table 5: Hyperparameters settings of the baseline causal discovery and data imputation algorithms.
Methods
Params.
AQI
Traffic
Medical
PCMCI
Tiua
5
5
5
PCa
0.05
0.05
0.05
Learning rate
0.05
0.05
0.05
NGC
Arsdge
0.01
0.01
0.01
0.02 - 0.2
0.02 - 0.2
0.02 + 0.2
An
0.1
0.1
Learning rate
0.01
0.01
0.001
eSRU
Batch size
40
40
40
Epochs
50
50
50
10
10
10
SCGL
Batch size
32
32
32
Window
3
Epochs
50
50
50
LCCM
Batch size
10
10
Hidden size
20
Steps
200
200
200
NGM
Horizon
5
GI. reg
0.05
0.05
0.05
10
10
TCDF
Epoch num Learning rate
1000
1000
1000
0.01
0.01
0.01
Input step
20
20
20
CUTS
0.1
0.1
0.1
0.1
0.1
0.1
Input step
1
CUTS+
0.01
0.01
0.01
0.1 +11
0.1
+
0.1 +
TSCD Algorithm Evaluation. Since our generated time-series are relatively short and contain several samples, alter existing approaches by enabling TSCD from multiple observations (i.c. multiple time-series). For neural-networr--baeee or optimization-base CUTS+, NGC, and TCDF, alter their dataloader module prevent cross-sample data fetching. For PCMCI, we use its variant JPCMCI+ which permits the input of multiple time-series. For re- maining TSCD algorithms that do not support multiple time-series, we use zero-padding isolate each sample. We list the original implementations of our included TSCD algorithms in the follow- ing:
approaches such as CUTS,
PCMCI.
The code is from htmptppp///ttttuuoooo//////orrrr
NGC. The code is from https: fettps://gittae //g/thub.bb/////// com/ iancoverttttt. /Neural-GGGG We use the cMLP network because according the original paper Tank et al. 2022) cMLP achieves better performance, except for Dream-3 dataset.
eSRU. The code is from https: //gpt/ggttttt////
SCGL.
The code downloaded from the link shared in its original paper Xx et al. 2019).
LCCM.
The code is from finntttppp//aattt.cccc/////oooos
NGM. The code is from httpm//ttttrr.ggghooooc/////eeea continuous-time.
ing-
CUTS/
The code is from fomnhhttper//gthhh....////yyy/// The code is from httpr//nttt///ggght.aoo////////a
TCDE
Discriminative Network. To implement the discrimination score in time-series quality control, train separate neural networks for each dataset classify the original from the generated time-
19

---

Published as a conference paper at ICLR 2024
Table 6: Comparison of predictive MSE with different implementations of CDNN. The experiments are performed on AQI datasets.
Backbone
MLP
LSTM
Transformer
No Sharing
0.0511 0.0009
0.0017 0.0002
0.0061 1 0002
Weight Sharing
Shared Encoder
0.0248 + 0.000....0 0.0033
0.0048 + 0.0002
0.0056 = 0. 0.0001
Shared Decoder
0.0254 0.0024
0.0014 0.0002
0.0060 0.0002
series. We use a 2-layered LSTM with hidden size 8, the training is performed with a learning rate of le-4 and total of 30 epochs.
A.3 ADDITIONAL RESULTS
A.3.1 TIME-SERIES FITTING
we show that CDNN can be used to fit causal models with adjacency matrix A. By splitting each network into two parts, i.c., encoder and decoder, 9 combinations (MLP, LSTM, Transformer combining shared encoder, shared decoder, and no weight sharing) are considered in the experiments. By comparing fitting accuracy (or prediction accuracy) on the AQI dataset, we observe that LSTM with shared decoder performs the best among 9 implementations.
In Section 3.2
A.3.2 SCHEDULED SAMPLING
To validate if scheduled sampling is effective in the training process, we perform an ablation study on three datasets with different autoregressive prediction steps. We observe in Tab. 7 that, by incorporating scheduled sampling, the cumulative етrоr decreases, and the decrement is large with higher autoregressive steps. This demonstrates that scheduled sampling does decrease accumulative error for the fitting model, which is beneficial for the following generation process.
Table 7: Comparison of prediction MSE with and without scheduled sampling policy. The compar- isons are performed with different autoregressive prediction steps n in terms of MSE.
AQI
Traffic
Medical
Step
w/o
w/o
w/o
n 2
0.0129
0.0123
0.0124
0.0124
0.0082
0.0082
5
0.0348
0.0329
0.0279
0.0282
0.0136
0.0155
10
0.0351
0.0377
0.0312
0.0354
0.0181
0.0193
20
0.0331
0.034
0.0318
0.0331
0.0144
0.0174
A.3.3 CROSS CORRELATION SCORES FOR TIME-SERIES GENERATION
Despite the discriminative score and MMD score, we further compare the similarity generated data to original versions in terms of coosscoorreltton scores. Specifically, calculate the correla- tion between real and generated feature vectors, and then report the sum of the absolute differences between them, which is similar to Jarrett al. (202))ss calculation process. We show the results along with the ablation study in Tab. 8
A.3.4 VARIABILITY OF THE EXTRACTED HCG
For Medical dataset, the HCG is extracted with DeepSHAP and the result may vary with different seed. In Fig. we show the extracted HCG with 5 different seeds, which shows that the variability relatively small. The average standard deviation is 0.104. The adjacency matrices of three ACGs of CausalTTime are shown in Fig.
20

---

Published as a conference paper at ICLR 2024
Table 8: Quantitative assessment of the similarity between the generated and original time-series in terms of cross-correlation scores. We show the ablation study in the table as well.
Cross Correlation Score
Datasets
AQI
Traffic
Medical
Additive Gaussian Noise w/o Noise Term Fit w/o Residual Term
43.74 11.55
198.30 +.00
18.91
1:177
50.04 + 4.79
194.44
5.00
20.77
+ 4,40
49.04 70l
238.38
4:00
21.94
3:45
Generate w/o Residual Term Full Model
40.62 + 2473
164.14
+ 5004
23.53
3.12
39.75 + 5.44
60.37
2,88
22.37
1.59
18
D0
Figure 7: Variability of the extracted HCG,
A.3.5 TSCD EVALUATION WITH MISSING ENTRIES AND LATENT CONFOUNDING
Missing entries or latent confounding are common in real world TSCD. We simulate these issues on our synthetic data, by dropping some observations certain missing rates or leaving out certain nodes to create hidden confounding. Specifically, we
Set random missing rates for the generated dataset, i.e. missing completely at random (MCAR), which is the most common missing scenario (Geffner et al. 2022). Specifically, value observation mask or,i to label the missing entries. Each data point in the observations are missing with a certain probability pt, here in our experiments follows Bernoulli distribution ot, ~ Ber(1 (1- Pt). Then the time-series with missing data is tested on existing approaches that work with missing entries, i.c., CUTS. CUTS+, LCCM, NGM. We set missing rate to p 0.3 and the result is shown in Tab. 9
bi-
Leave out certain nodes to create hidden confounding, like Nauta et al. (2019)'s 's approach. And test existing approaches on the hidden confounding version of CausalTime, Specifically, we deleted node no. 1,17,27,23 for AQI dataset, 7,17 for Traffic dataset, and 6, 9 for Medical dataset. Then we generate new versions of the original ACG. The results are shown Tab. TO However, since almost all existing TSCD algorithms struggle to get accurate causal discovery results on CausalTime, is hard draw any conclusion by focusing on certain nodes in the confounding version of CausalTime.
Table 9: Performance of TSCD algorithms in the presense of missing data.
AUROC Traffic
AUPRC
Methods
AQI
Medical
AQI
Traffic
Medical
CUTS
0.5953 0.0018
0.5574 a 00185
0.7864 +: 0.0400
0.6547
0.005 0.7086 :: 0.0644 0.7672 4: 0.0074
CUTS+
0.8830 0.0688 0.5071
+ 0.374
0.7296
+ 0.015
0.7429
0.0150
0,5193
0.476 0.6774 = 0.0134
NGM
0.5341
0,6048
0.0266
0.4328
0.191
0.7159
0.4596
0.5657
0.0163
LCCM
0.7768
0.0054
0.5321 + 00054
0.7877 + 0.6611
0.8317
0.00ss
0.5296
0.0147
0.7699 = 0.177
A.4
ALGORITHMIC REPRESENTATION FOR CAUSALTIME PIPELINE
We show the detailed algorithmic representation of our proposed data generation pipeline in Algo- rithm A.4 where we exclude quality control and TSCD evaluation steps.
21

---

Published as a conference paper at ICLR 2024
Pm2.5
Traffic
Medical
10
10-
10
30
15
15
40
20
20
25
30
30
60-
35
35
10
60
10
20
30
10
30
Figure 8: Adjacency matrices of three ACGs in CausalTime.
Table 10: Performance benchmarking of baseline TSCD algorithms CausalTime datasets in the presense of hidden confounding.
AUROC Traffic
AUPRC
Methods
AQI
Medical
AQI
Traffic
Medical
GC
0.4680
0.0366
0.4414
0.0375
0,5997
0.0588
0.6664 100050
0.3160
0.004
0.5398
0.0470
SVAR
0.6207
0.63627
+ 0.891
0.7478
+00002
0.8053 + 0.0288
0.6186 + 00481
0.7733 + 0.066
N.NTS
0.4630
0.0881
0.5238
0.1268
0,4886
0.6524
0.1111
0.4165
00416
0.4786
PCMCI
0.5865
0.0673
0.6031
+ 0.0089
0.7877
0.0231
0.7385 + 0.0244
0.3988 + 01412
0.6248
0.0148
Rhino
0,7028
0,0816
0.6327
0.0233
0,6688
0.0386
0.7966 +: 0,054s
0,4286
00218
0.5926 0,0268
CUTS
0.6557
0.7026
0.8184
0.6550 + 0.7633
0.7958
0.7886
CUTS+
0.8944
п.006)
0.6596 + 0.0156
0.85507
+ 0.00069
0.8062 + 0.0245
+ 00052
0.7585 0.0066
NGC
0.6077
0.0067
0.6081
0.0107
0.5637
0.003
0.7460 0.0048
0.4356
00125
0.542 0.002
NGM
0.5059
0,0246
0.5540
+ 0.0278
0,5565
0.0068
0,7092 + 0.0166
0.3886
0018s
0,5552 0.0199
LCCM
0,9281
00153
0,6395
0.0432
0.8185
0.0399
0.9648 +: 0.0088
0.6018
00233
0.8465 0.0255
eSRU
0.7473
0.6792
0.8401
0.8634
0.5336
0.8360
SCGL.
0.5504
0,3755
0.6434
2.0949
0.4854
0.0194
0,3982 + 0.0297
0.6009
+ 00173
0,5107 0.0296
TCDF
0.5248
0.0133
0.5743
0.0271
0.6995
0.0243
0.7344
0.003
0.4553
0.039
0.6375 0.0145
22

---

Published as a conference paper at ICLR 2024
Algorithm Pipeline for CausalTime Generation (Excluding quality control and TSCD evaluation)
Input: Time-series dataset 11I:L,1,... F1:L,N) with length L; (Optional) graph H, generated with prior knowledge; generation length L.
Output: Paired time-series causal discovery (TSCD) dataset (8, A)
Initilize parameter set {O:) (V:)
Lime-series Fitting
for n1 Epochs do Update {O:) with Algorithm (Prediction Model Fitting).
end for
for n1 Epochs do Update { with Algorithm (Noise Distribution Fitting).
end for
HCG Extraction
if Exists prior knowledge Hp then
H <-<l Hp else
H [DeepSHAP (+))
end
H + [( > 7), where calculation of y is discussed in Section 3 Time-series Generation with ACG
Generate actual causal graph Å with Eqn. 6 select initial sequence & randomly from origi- nal time-series X. for t = top do
for i= = 1 top N do Generate R..; and aR. with Eqn. 4 and where fe, is sampled from Tw, (u).
end for end for return Generated TSCD dataset X. A)
Algorithm Prediction Model Fitting
Input: Time-series dataset (#1.L,1,,-.,, F1:L,N) with length L, parameter set {6.)
Output: Paired time-series causal discovery (TSCD) dataset (x, A)
for i = 1 top N do
Perform prediction with Lt,; < fe, (xt-r:t111- Xt-rit 1.N) + 1h.
Calculate loss function with MSE(X, (EE(XXX X)
Update {0 with Adam optimizer
end for
return Discovered causal adjacency matrix A where each elements is āz.--
Algorithm Noise Distribution Fitting
Input: Time-series dataset (II.L,1, .... $1.L,N] with length L; Parameter set (0,), (V.)..
Output: Paired time-series causal discovery (TSCD) dataset (x
for i = 1 top N do
Perform prediction with He,e <- fe:
Xt 1,N) + Mt.s-
Calculate real noise Th &- Iz,i Zz,5.
Calculate likelihood Pq, (mh,s) + Pa (T (n),s) Dn... Update { with Adam optimizer.
0_ T (m.s).
end for
return Discovered causal adjacency matrix where each elements is āz
23


Reviews:
[{"review_id": "3Ktm9LpeBh", "reviewer": "Reviewer_7uMr", "summary": "This paper introduces a pipeline to generate time series with know causal graphs. The models considered are nonlinear auto-regressive with instantaneous causal effects. The procedure taken is as follows\n - run a causally disentangled neural network, here based on LSTM, on a real dataset \n - fit normalising flows to the residuals\n - sparsify the node-to-node relationships, the resulting graph $H$ is the HCG. This can be done using Shapley values or using prior domain knowledge.\n - to maintain higher data fidelity, the generated time series have double the dimension of the original data. Each time step consists of $\\mathbf{x}$ and $\\mathbf{x}^\\text{R}$, where $\\mathbf{x}^\\text{R}$ represents the residual when predictions are made using the sparsified model with $H$. The advantage is that it is now possible to generate a new dataset that resembles the old as closely as possible and with a know, sparse causal graph.\n\nExperimentally, the authors run their algorithm on 3 datasets and generate new causal benchmark time series. These are evaluated for fidelity with the original. Existing methods struggle on the resulting datasets.", "questions": "- How does causal time fit in with methods that are designed to discover instantaneous relationships, like Rhino (Gong et al., 2022)? Could Rhino be applied to CausalTime datasets, and if so should it be included in the benchmarking? Based on your assumption of \"No Instantaneous Effect\", this would not be possible.\n- The methods in the paper suggest that including the residual term is important. This, in turn, implies that \"natural\" causal graphs may be dense. E.g. all components of $\\mathbf{x}_{t-1}$ affect $\\mathbf{x}_t$. However, in CausalTime, the causal graph is forcibly sparsened to produce $H$, so as to present a more interesting problem to causal discovery algorithms. In Table 2, the inclusion of the residual term seems to reduce discriminative scores by a factor of at least 10, implying it is very necessary to get good reconstruction. Thus- is the forced sparsification used in CausalTime actually contrary to natural time series?\n\nGong, Wenbo, et al. \"Rhino: Deep causal temporal relationship learning with history-dependent noise.\" arXiv preprint arXiv:2210.14706 (2022).", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- this paper presents a neat trick to generate a time series with a know causal graph that matches real data in distribution without knowing anything about the causal structure of the underlying data. This is achieved by doubling the dimension of the time series and having a \"residual\" stream as well as a real-variable stream\n- the generated datasets present problems for existing temporal causal discovery methods", "weaknesses": "- this paper is only \"causal\" in a very weak sense. Indeed, given the \"No instantaneous effects\" assumption, it is possible to write down a dense causal graph in which all prior variables are parents of $\\mathbf{x}_t$. Arguably, there is no true causality here, only the problem of learning a time series with *sparse* relationships, because the DAG constraint is automatically satisfied when instantaneous effects in the original time series are discounted.\n- the primary novelty is the $N \\rightarrow 2N$ trick to make a time series that fits the stated requirements. Whilst I think this is quite smart, the remainder of the paper is a concatenation of existing methods.\n- this key idea is not explained particularly well in Sec 3.4. \n- it is unknown how realistic the causal relationships used in CausalTime are. Indeed, the authors make no claim of doing causal discovery. Hence, the causal relationships used in CausalTime may be very different to those found in nature, implying that CausalTime datasets are not a good surrogate for causal discovery on real time series. Indeed, when using Shapley values, a correspondence between feature importance and causality is suggested which may be incorrect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1MiA7fG27U", "reviewer": "Reviewer_jMsi", "summary": "The paper introduces a time series simulator that utilizes neural networks to closely match a real dataset. This simulator allows its users to manipulate the causal graph of the process by altering the neural network pathways, or by setting less significant inputs to zero. An individual neural network is employed for each generated time series, thereby encoding a stationary \"family\" within the causal graph of the Dynamic Bayesian Model-like structure. The simulator's construction and justification draw from various existing methodologies. The paper demonstrates that the datasets generated in this manner do indeed share the traits of the original time series, especially when evaluated through their nonlinear embeddings such as t-SNE. Furthermore, the paper benchmarks nine recent deep learning-based models against each other, using data produced by the simulator trained on three different real datasets.\n\n*I raised my score after rebuttal*", "questions": "1. Could you please clarify the notation for the ACG? Your A is represented as a $2N\\times 2N$ matrix. The $2N\\times N$ portion seemingly represents the \"causal\" and residual term mixing in the previous time step. However, the actual causal graph likely has an adjacency matrix of a $2N \\tau_{max}\\times N$ dimension to model edges from parents up to a lag of $\\tau_{max}$. Does this notation imply that in this paper you were only considering Markov order 1 models?\n\n2. I would appreciate if you could use more classical approaches, in addition to purely neural models, for benchmarking time-series causation. I suspect that the experiments may favor neural models over others (as noted in the weaknesses). It would be beneficial to include comparisons with the SVAR model, Granger Causality, the PC algorithm modified to work with time series, and similar models.\n\n3. Could you please provide plots of the graphs that your simulator generates for each of the three real data test-case datasets?\n\n4. Could you characterize the variability of your ground truth ACGs as a function of training your simulator model starting with different seeds?\n\n5. I would appreciate an explanation of how the ground truth causal graphs in Figure 2 are combined.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written, with concepts and methodologies clearly explained, making it accessible and comprehensible.\n2. It addresses a significant need in the field of causal time series modeling, offering a solution to a complex problem.\n3. The evaluation of the proposed approach is relatively comprehensive, even though it is somewhat one-sided (as addressed in the paper's weaknesses). It includes benchmarking against several competing methods, which adds a comparative dimension to the analysis. Furthermore, an earnest attempt is made to measure the goodness of fit of the data generated by the proposed method, adding a quantitative validation to the study.", "weaknesses": "1. The only sanity check of performance of the simulator is the quality of the fit but not demonstration of the learned graphs and whether they are reasonable. Without an investigation into the graphs learned by the simulator we have a situation of severe under-determination of the system. What if, there is enough information in any reasonably sized subset of the prior time series to autoregressively fit any signal. What kind of graph the methods tested on the proposed simulated time series supposed to reconstruct? Table 3 does show non-random performance, so there's truth to it, but are the ACG graphs sparse, dense, high or low Markov order and do they even make sense from the \"organic\" perspective of the domain they have been generated for? These questions are left unanswered.\n2. The focus on neural models in simulation and estimation is limiting. It would be best to show how other models are benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2WsBS4gc08", "reviewer": "Reviewer_3QnF", "summary": "This paper proposes a methodology for producing realistic time series data with ground truth, which can be used to evaluate algorithms for causal structure learning in time series.  For most domains, it's impossible to have realistic data with a known ground truth causal structure, since we don't know the ground truth dynamics of most domains, so most data for evaluation of causal modeling algorithms is at least semi-synthetic.  The authors' approach builds on this tradition by learning a model of realistic data, treating that model as the ground truth, and using it to generate a new data set that can be used for evaluation.  The authors detail the methodology for their approach and perform a series of experiments, both to assess how realistic the generated data looks and to compare the performance of multiple time series causal discovery algorithms.", "questions": "I don't see this mentioned anywhere - will code be provided upon publication?\n\nIn Definition 1, what is Y?  Is it X at time t? (What is the output of the neural network? I'm not seeing Y referenced after this section)\n\nWhat was the motivation for using |AUC - 0.5|, rather than just AUC?\n\n\nEdit post author response: Updating score from 6 to 8.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "While the overall idea of this paper (fitting a model on realistic data, and then using it to generate data that has a known ground truth) isn't novel, I haven't seen it applied to time series data, and I think the authors' treatment of it is well-described and motivated.  The authors describe their methodology well and provide a reasonable technical foundation.\n\nThe experiments cover a useful breadth.  The comparison of the data distribution between the original and generated time series was interesting, and I appreciated how the ablation study, highlighting the importance of each piece of the equation.  For the comparison of causal discovery algorithms, I thought the authors picked a reasonable set of algorithms to compare, providing a nice demonstration of the application of CausalTime.", "weaknesses": "My biggest confusion/concern about this work is the inclusion of the residual term.  Clearly, from Table 2, the residual term plays a massive role in producing data that looks like the original data.  However, it seems as though the presence of the residual term simply means that every time series depends on every other time series.  While it makes sense that including more variables would allow for more accurate model fitting, it's not clear to me that the resulting data actually reflects the causal dynamics of the supposed ground truth graph.  Looking at Equations 6 and 7, it doesn't look like the residual term is down-weighted or anything to reduce its contribution.  Substituting Equation 7 into Equation 6, it actually looks as though the primary f_{theta_i} terms cancel, leaving us with the first term in Equation 7 plus the noise term, which essentially means that we're just ignoring which variables are actually parents of x_i and just including everything.  Am I misunderstanding or misinterpreting something?  It looks to me like a ground truth graph is generated, and realistic-looking data is generated, but the realistic-looking data doesn't actually come from that graph.  This is the main reason my score is a 6 rather than an 8, and if this is cleared up satisfactorally, I'd be happy to raise my score.\n \nA bit more analysis for the final evaluation would be helpful.  The authors point out that, when evaluated on synthetic data in prior work, the scores are higher across the board.  While that's interesting, I'm much more interested in if the conclusions we would draw as a result of using CausalTime data differ from those we would draw using synthetic data.  If I were trying to figure out which method performs best using synthetic data, but the method I chose would actually perform worse on realistic data, that would be a very convincing argument for the value of CausalTime.  It also appears as though a citation is missing (\"Besides, compared with the reported results from previous work (), ...\"), so I'm unable to assess how the results actually do compare.\n \n3.4 describes its purpose as describing how \"to acquire the Actual Causal Graphs with high data fidelity\".  However, the following paragraphs, up into the equation for the ACG in Equation 8, concerns generating the time series X, not the ACG. (the ACG is defined based on H, I_N, and J_N, none of which are defined in 3.4 prior to Equation 8) By the time you get to Equation 6, don't you already have the ACG? (since it relies on H) So it seems like the first sentence should instead read something like \"To generate data from the Actual Causal Graph (ACG) with high data fidelity\", rather than \"To acquire the Actual Causal Graph (ACG) with high data fidelity\"\n\nThis is minor, but there are some grammatical issues - for example, the first sentence of the 3rd paragraph of the introduction is a fragment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qkfBMX2HED", "reviewer": "Reviewer_YwkA", "summary": "The paper introduces a pipeline to produce time series that are resembling real world data while being synthetic and analytical enough to serve as benchmarks for TSCD algorithms", "questions": "It is unclear how the method overcomes its main limitation in performance of extracting the causal graph from raw data. It appears that the proposed pipeline uses a  time series causal discovery algorithm, to build a synthetic dataset , to test other  time series causal discovery algorithms, making this an unorthodox loop. How are we guaranteeing the accurate extraction of the underlying DAG to produce the synthetic data. \n\nReal world observational data are rarely well behaved and suffer from , missing entries, confounding, and other biases. A successful  time series causal discovery algorithm needs to be able to disentangle all these from the true causal features. How are the synthetic data taking these biases into account and guarantee not making a too easy task for a TSCD algorithm\n\n\n\n### EDIT POST REBUTTAL \n\nI have updated the score from a 5->6", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Based on the evaluation of the method, the produced time series appear to be reasonably realistic.\nThe method appears to be fairly simple conceptually\nThere is extensive analysis of the related literature\nThere is an ablation study - a very welcome addition to the paper.\nI believe the community will stand to benefit from using this paper", "weaknesses": "The analysis of the method, and the caption of Fig 1 could be improved. Too large emphasis has been given to sounding and appearing mathematical, this makes the  true contribution and impact of the paper, its incorporation in the analysis frameworks of other algorithms, harder to realize as it obfuscates details. \nThe assumption of stationarity, albeit common, remains very restrictive and should be treated as a limitation. There is a significant amount of real world problems that are non stationary and a method like this would not fare well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)