DyCAST: Learning Dynamic Causal Structure from Time Series

Authors: Yue Cheng, Bochen Lyu, Weiwei Xing, Zhanxing Zhu
Primary area: causal reasoning
Keywords: dynamic causal discovery; time series

Abstract:
Understanding the dynamics of causal structures is crucial for uncovering the underlying processes in time series data. Previous approaches rely on static assumptions, where contemporaneous and time-lagged dependencies are assumed to have invariant topological structures. However, these models fail to capture the evolving causal relationship between variables when the underlying process exhibits such dynamics. To address this limitation, we propose DyCAST, a novel framework designed to learn dynamic causal structures in time series using Neural Ordinary Differential Equations (Neural ODEs). The key innovation lies in modeling the temporal dynamics of the contemporaneous structure, drawing inspiration from recent advances in Neural ODEs on constrained manifolds. We reformulate the task of learning causal structures at each time step as solving the solution trajectory of a Neural ODE on the directed acyclic graph (DAG) manifold. To accommodate high-dimensional causal structures, we extend DyCAST by learning the temporal dynamics of the hidden state for contemporaneous causal structure. Experiments on both synthetic and real-world datasets demonstrate that DyCAST achieves superior or comparable performance compared to existing causal discovery models.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 3

Reviews and discussion:
<thread>
creation date:
1729666835621

thread_type:
official_review

<thread_object>
creator: reviewer_e4ie

<content>
summary:
This manuscript investigates dynamic causal structure in time series using Neural ODEs, a topic of broad interest. The authors reformulate the problem of learning causal structures at each time step as finding the trajectory of a Neural ODE on a directed acyclic graph. Finally, simulations were performed on both synthetic and dynamical models to validate the proposed approach.

strengths:
This paper proposes a method for identifying dynamic causal structure by learning their trajectory using L1 normalization. Both intra-slice and inter-slice structures are learned simultaneously. The method demonstrates higher performance compared to other approaches it is evaluated against.

weaknesses:
The concept of using activity trajectories to identify causal structure seems intriguing. However, due to a lack of clarity, the novelty of the proposed method remains uncertain. The method is not well explained. There are insufficient numerical experiments to support the proposed approach. Several key details are missing. The writing and certain claims need refinement.

questions:
1)	The rationale of Eq.10. What would happen if the concatenation was removed? An ablation study addressing this would be helpful.
2)	Is this approach applicable to large-scale systems? The results presented in the manuscript only cover 5/50/100 nodes. 
3)	Line 182, is the correct phrasing "neural underlying ODE" or "underlying Neural ODE"?
4)	No information about the $\lambda$ parameter in Eq. 18. 
5)	Why do negative weights appear in Figure 3? This seems unreasonable and counterintuitive. Using a BA synthetic network might be more appropriate.
6)	How does the noise strength affect the performance of the proposed method?
7)	It is unclear why time steps are discussed, but time points are not mentioned. The causal structure detection largely depends on factors like the length of the time series considered, particularly the information at each time point. However, these key details are missing from the manuscript, which raises concerns about the accuracy and applicability of the proposed method.
8)	More rigorous numerical experiments are needed to substantiate the claims made. The current experiments are insufficient to demonstrate the robustness of the proposed method.
9)	I am confused why the authors mentioned time steps, but not mentioned time points at all. Causal structure detection is related to how much information, e.g., the length of time sequences you considered. All these important information is lacking. 
10)	I am surprised that the NetSim dataset includes 100 nodes. Can you provide more details about this dataset?
11)	Figure 5 requires sublabels, and in part (b), there seems to be an error (0, 30, 80, 00). Please review and correct it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730195182920

thread_type:
official_review

<thread_object>
creator: reviewer_gcbx

<content>
summary:
This work presents DyCAST, a novel framework leveraging Neural Ordinary Differential Equations (Neural ODEs) to learn dynamic causal structures, showing superior performance compared to existing models across various datasets.

strengths:
1. The algorithm effectively supports dynamic contemporaneous structures.

2. The experimental results are compelling, though there are inconsistencies in the choice of baseline methods across different sections (see weaknesses).

weaknesses:
1. While DyCAST claims to support dynamic intra-slice graphs, the ground truth causal graphs in the Netsim and CausalTime datasets do not contain such dynamics. Therefore, although I appreciate the results in Figure 6, I am concerned that the claim is not fully validated in the quantitative experiments.

2. Several baseline methods are included in the CausalTime experiments but not in the Netsim experiments. Additionally, the baseline choices differ between the Netsim and Synthetic experiments. Is there a specific reason for this? If not, more baseline comparisons should be included. If time is limited during the rebuttal period, incorporating the most recent works would be acceptable.

questions:
1. As someone who is not an expert in Neural ODEs, I find the model presented in Equation (4) somewhat unconvincing. Why not learn $W_t$ based on the data at each time step? Are there theoretical justifications or ablation studies to support this modeling choice? If this matter is cleared up, I will be happy to raise my score.

2. Are there additional examples that demonstrate the necessity of dynamic intra-slice graphs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730322095331

thread_type:
official_review

<thread_object>
creator: reviewer_ayt4

<content>
summary:
This paper introduces DyCAST, a method for determining time-varying causal graphs using constrained neural ODEs. In particular, DyCAST determines the causal graph for a set of variables $\mathbf{X}_t$ as a linear combination of dynamic "intra-slice" causes, $\mathbf{X}_t \mathbf{W}_t$, and static "inter-slice" causes, $\sum\_{k=1}^P \mathbf{X}\_{t-k} \mathbf{A}\_k$, plus additive noise. The dynamics of $W_t$ are modeled as a constrained neural ODE, which is extended to a latent ODE using the encoder-process-decoder framework. The latent state models an autonomous system, but may be concatenated with the time $t$ in decoding for systems with periodic components. Finally, a sparsity penalty is added, and DyCAST is combined with CUTS(+), a Granger-causal method for time-varying inter-slice graphs. Extensive experiments are performed on some limited synthetic data and several real or simulated data sets, where DyCAST with CUTS(+) is typically the best-performing method in terms of AUROC and AUPRC.

strengths:
- The issue of time-varying causal structure is under-addressed and practical problem. Using constrained neural ODEs is an elegant solution, and the results are impressive.
- The paper is well-written and introduces concepts from neural ODEs very well. The included illustrations are helpful, and the figures are instructive.
- Incorporating time-varying structures can be extremely interpretable in some scenarios. For example, I found Figure 6 very compelling, with causal influences smoothly following the time of day.

weaknesses:
**On the Combination With Other Methods**

The combination of DyCAST with CUTS+ is one of the more important contributions in the experiments section of the paper, but is not discussed much. This is especially difficult to me because CUTS+ is a Granger-causal method, and to the best of my knowledge does not actually learn what are referred to as "inter-slice" graphs, except in the trivial $p=1$ case.

It would improve the presentation of the paper to include more details of how DyCAST can be combined with other methods like CUTS+, and also how this changes the computational requirements and potential interpretations. Moreover, as an aside, the paper inconsistently refers to DyCAST-CUTS as using CUTS or CUTS+.

**A Lack of Ablation on Most Datasets**

As mentioned in the strengths, the performance of DyCAST is impressive, and it is apparent that it is beneficial in dynamic problems. That said, I find the ablation study in Section 4.1 very interesting, and would like to see a similar experiment in the NetSim and CausalTime experiments. In particular, from Table 1, it is clear that the latent state provides at least some benefit, even in static problems, but this isn't explored outside of the simplistic synthetic case. I view this as critical to understanding *why* DyCAST is successful.

**Minor Typos** 

There are several typos throughout the manuscript; ones I noticed include:
- (Page 2, Line 085) NOTEATS should be NOTEARS.
- (Page 2, Line 087) DYNOTEATS should be DYNOTEARS.
- (Page 3, Line 124) I think $X_{t-k}^j$ should be bolded.
- (Equation 18) In the first summand, there is an $\rvert$ instead of $\rVert$.
- (Page 8, Line 398) NTS-NOTEATS should be NTS-NOTEARS.
- (Page 8, Line 398) TEDCI should be TECDI.
- (Page 10, Line 404) "In future" should be "In the future".
- (Page 13, Line 657) Erdős–R'enyi should be Erdős–Rényi.

**A Few Other Minor Points**

- (Related Work) The approach of CCM is of course much different than the proposed approach -- nevertheless, latent CCM (Brouwer et al., 2021) seems like a relevant application of neural differential equations to causal learning to be mentioned in related work. 
- (Implementation and Code) While I think the details in Appendix A would be sufficient to reproduce most of the paper's experiments, I hope that the authors consider releasing their code with a finalized version of the paper, as I think it strengthens the impact and reproducibility of the paper.

questions:
- My main question, outlined in weaknesses, is **to what extent can the performance of DyCAST be attributed to the latent state**? For example, in Table 1, the latent state is shown to significantly improve performance; could similar ablation studies be performed in the other experiments?
- There is a brief mention of running time on page 7 ("moreover, DyCAST reduces the running time by more than 20% across various variable dimensions"). This claim is not particularly clear to me: **with respect to what is the running time reduced? At what variable dimension is this noticeable?** Similar claims are made in Appendix A.2.1, but without any specific figures. In general, I think it would improve the manuscript to report the run time of DyCAST with respect to the baselines.
- **What are implementation details for DyCAST-CUTS?** And how does this affect runtime? How are dynamic inter-slice graphs handled outside of the $p=1$ case?
- It seems like the most straightforward approach in the current framework for inter-slice dynamics would be another (independent?) neural ODE on $\mathbf{A}$. Is there a reason why this doesn't work?
- It seems surprising to me that the $F_1$ score is so insensitive to $r$ -- **how would the authors interpret this? Would they expect very small $r$ to be more detrimental in problems with faster dynamics?**
- The methods for comparison are different in NetSim and CausalTime datasets; for example, TECDI is included in NetSim but not CausalTime, and vice versa for LCCM. **Is there a reason for the differing benchmark methods in these sections?** Since both of these methods appear to be the closest competitors to DyCAST, it would be great to standardize the benchmarks, and include both TECDI and LCCM in all comparisons.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730727681918

thread_type:
official_review

<thread_object>
creator: reviewer_4eyh

<content>
summary:
The authors introduced the DyCAST algorithm to infer time-variant DAGs from nodal time series, considering both intra-slice and inter-slice dependencies among variables. They demonstrated its performance using both synthetic and real-world datasets and compared it with other causal discovery methods.

strengths:
1. The authors addressed limitations of previous methods by considering both intra-slice and inter-slice dependencies among variables. 
2. They offered clear and plausible theoretical motivation for their DyCAST algorithm. 
3. The authors tested DyCAST's performance and compared it with previous methods.

weaknesses:
1. The authors tested their DyCAST algorithm on datasets with a limited number of variables, which restricts its applicability in real-world scenarios.
2. While incorporating inter-slice dependencies is beneficial, the inference requires more data to be effective.
3. The algorithm relies on multiple independent realizations (runs) of system dynamics (i.e., nodal time series). For instance, in the first synthetic dataset, there are only d=5 variables and T=7 time steps (i.e. 7 snapshots of DAG), yet it requires N=500 independent realizations for inference. This is unrealistic, as many real-world situations may only provide one trial or a single trajectory of each variable.

questions:
1. The authors used traffic as a motivation and test example; however, traffic networks are not DAGs. Similarly, brain networks also contain many bidirectional edges and feedback cycles. How would the algorithm address situations where the underlying network is a directed graph rather than a DAG?  
2. How should the value of p (i.e., the order of time lag) be set in the algorithm? Is the ground truth for p assumed to be known for the inference process?  
3. How much additional data is needed for inference when considering both intra- and inter-slice dependencies? If unlimited data were available, many other factors could be incorporated into causal discovery. Thus, the amount of required data is critical for addressing general inverse problems. The authors should specify how many realizations are necessary to achieve a given performance for a specific problem defined by the number of variables and time steps. Additionally, if p is unknown during the inference process, the algorithm may require even more data (i.e., independent realizations).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732560188289

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers and Summary of Changes of the Revision

comment:
We thank all reviewers a lot for the valuable comments and insightful questions and suggestions.

To answer questions and address the concerns of the reviewers, we make the corresponding changes in the rebuttal revision, which will be summarized as follows: additional experiments for scalability, efficiency, and robustness; extension of DyCAST. For clarity, all changes in the revised manuscript are highlighted in blue.

***
### Additional Experiments for Scalability, Efficiency, and Robustness

As suggested by Reviewer **4Eyh**, **AYt4**, **gcbX** and **e4ie**, we supplement experiments to examine the scalability, efficiency, and robustness of the DyCAST. These additional experiments are presented in **Appendix C** (pages 15 to 22) and **Appendix E**(page 23). Specifically:

1. Scalability analysis over datasets with large-scale variables (in **Table 5, Appendix C.1, page 15**). We show that DyCAST still displays an average **F1 score> 0.5** when the number of variables **d=300**, which is much higher than the number of variables that other baseline methods can handle, as shown in **Table 5**. Correspondingly, DYNOTEARS and its variants fail to identify causal relationships when the variable count reaches $d = 50$ and does not converge for $d > 50$. Thus, the potential of DyCAST to effectively handle large-scale dynamic systems has been thoroughly demonstrated. 

2. Sample efficiency analysis over datasets with different sample sizes (in **Figure 7, Appendix C.2, page 16**). We show that DyCAST still displays an average **F1 score> 0.4** and an average **SHD < 30** when the number of sample sizes **N=20**, as shown in **Figure 7**. In contrast, with a sample size below 300, DYNOTEARS and its variants exhibit a sharp decline in causal recognition, with an average F1 score around 0.1 and SHD exceeding 200, rendering effective recognition infeasible. Thus, the potential of DyCAST to effectively handle limited trajectories has been thoroughly demonstrated.

3. Temporal generalization analysis over datasets with different lengths (in **Figure 8, Appendix C.3, page 17**). We show that DyCAST displays an average **F1 score> 0.5** and an average **SHD < 20** when the length of time steps **T=128**, as shown in **Figure 8**. In contrast, DYNOTEARS and its variants fail to converge when time steps exceed 64, with SHD surging to 100 (out of 100 total edges in this experiment). Thus, the potential of DyCAST to effectively handle long temporal dynamics has been thoroughly demonstrated.


5. Robustness to noise analysis over datasets with different strengths and types of noise (in **Figure 9, Appendix C.4, page 17**). We show that DyCAST exhibits robustness to noise intensity and type, as their effects are primarily limited to the initial intra-slice and inter-slice learning phases, with **minimal impact** on subsequent intra-slice accuracy due to the Neural ODE's ability to capture underlying dynamics, as shown in **Figure 9**. This demonstrates the model's resilience in learning robust temporal dependencies under noisy conditions.

6. Adaptability analysis over datasets with unknown autoregressive order (in **Figure 10, Figure 11, Appendix C.5, page 18**). We show that DyCAST can effectively estimate inter-slice entries even without prior knowledge of the autoregressive order $p$, highlighting its adaptability to unknown temporal dependencies.

7. More ablation studies over synthetic and real-world datasets (in **Table 1 (Section 4.1, page 8), Table 8, Table 9, (Appendix D, pages 22 to 23)**). We conducted extensive ablation studies on real and synthetic datasets, analyzing the impact of $\boldsymbol{S}_0$ and latent states to uncover the underlying factors driving DyCAST's performance.  Each row of $\boldsymbol{S}_0$ encodes how each intra-slice variable is influenced by others and, in turn, influences them. Without the concatenation operation in $\boldsymbol{S}_0$, DyCAST remains incomplete after the encoder, making this step crucial for its performance. Additionally, leveraging $\boldsymbol{S}_0$ to derive a more compact hidden state representation can further enhance DyCAST's computational efficiency and effectiveness.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper performs causal discovery for time-varying dynamic causal structures. In particular, it employs a constrained latent neural ODE to model the dynamics and its evolution. Most of the reviewers evaluated the paper positively. There was a question about the clarity of the presentation, which I hope can be addressed in the camera ready version of the paper. I would also hope that the authors can discuss whether and when the causal structure model considered in this paper is identifiable.