Gradient based Causal Discovery with Diffusion Model

Authors: Furui Liu, Yuanhang Feng, Huiyang Wang, Zhouhan Lin, Bowen Zhou
Primary area: causal reasoning
Keywords: causal discovery, generative models

Abstract:
Causal discovery from observational data is an important problem in many applied sciences. Incorporating a recently proposed smooth characterization of acyclicity, gradient-based causal discovery approaches search for a Directed Acyclic Graph (DAG) by optimizing various neural models. Although they show some inspiring results given certain assumptions satisfied, their capability of modeling complex nonlinear causal generative functions is still unsatisfactory. Motivated by recent advances in deep generative models, we propose to use diffusion models for causal discovery,  and search for the DAG under continuous optimization frameworks. The underlying nonlinear causal generative process is modeled with diffusion process, and with flexible parameter configurations, it has the ability to represent various functions, and the proposed causal discovery approach are able to generate graphs with satisfactory accuracy on  observational data generated by either linear or nonlinear causal models. This is evidenced by empirical results on both synthetic and real data.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730153999321

thread_type:
official_review

<thread_object>
creator: reviewer_cbsw

<content>
summary:
This paper introduces a diffusion model (denoised diffusion probabilistic model) for causal discovery and demonstrates its effectiveness in this task.

strengths:
The paper highlights the significant advantages of using a diffusion model for causal discovery. In nonlinear models, it consistently outperforms many other baseline methods.

weaknesses:
* Small typo: Line 309-310, "deep leyers"-> "deep layers". Line 183-184, "node i to node j"->"node $i$ to node $j$"


* The current form of Proposition 1 lacks rigor, raising concerns about its accuracy and possible need for revision. Here, $A$ is the weighted adjacency matrix. Notably, a sequence $A_n$ can always be constructed such that $\inf_{n} h(A_n) = 0$ (using $\inf$ here is more appropriate than $\min$), as $A$ is a weighted adjacency matrix and $h$ is continuous in $A$, implying that $h_* = 0$. Additionally, the phrase “all direct cyclic graphs in the solution space” is unclear and seems incorrect. The solution space should consist of only directed acyclic graphs, so there should be no cyclic graphs in this space. Please revise this accordingly.

* The experimental results omit the algorithm’s runtime, and results are only reported for graphs with up to 50 nodes and for ER2. Given that this paper employs a diffusion model for causal discovery, it would be helpful to include [1] as a baseline. Additional experiments would strengthen the case for the effectiveness of the diffusion model in causal discovery.

[1]Pedro Sanchez, Xiao Liu, Alison Q O’Neil, and Sotirios A Tsaftaris. Diffusion models for causal discovery via topological ordering. arXiv preprint arXiv:2210.06201, 2022.

questions:
* I noticed that CAM predicts many edges that do not exist in the true causal graphs. This could be due to the CausalDiscovery package having the tuning step turned off by default, which is crucial for the CAM algorithm. Could this be a contributing factor to CAM’s degraded performance?

* Could you clarify why DAG-diffusion’s performance is lower in linear models? Is it possibly because the diffusion model is more effective at capturing nonlinear relationships?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674332416

thread_type:
official_review

<thread_object>
creator: reviewer_eogv

<content>
summary:
This paper proposed an approach to combine the causal discovery with the diffusion model. In particular, the author proposed to adopt a DAG-GNN approach for structural causal model but use diffusion approach to model two of the non-linear function $f$, $g$. For DAGness of the graph, they propose to use NOTEARS approach for soft but differentiable constraints. Empirically, the author evaluates on 4 synthetic datasets and two semi-synthetic ones to evaluate the performance, showing that DAG-Diffuser can outperform the baselines on some datasets.

strengths:
This paper is clearly written and easy to follow. The proposed approach is indeed sound and correct. Personally, although I was expecting the proposed method is to use diffusion for graph modelling (i.e. diffusion model to directly model the graph distribution), it is still interesting to see the performance if one uses diffusion model only for function modelling.

weaknesses:
However, I have several concerns regarding this paper. First, methodology-wise, I think the contribution is not significant. The core idea of this paper is to use diffusion model to model function $f$, $g$. Due to the generality of diffusion model (i.e. not much technical requirements if one wants to use diffusion model), it does not pose significant technical challenges of simply plug-and-play. Therefore, in terms of methodology, it is of little difference compared to using normalizing flow, invertible neural network for $f$ and $g$, which has been done before. 

Another question I have is the motivation of using diffusion model for $f$ and $g$. Is it because diffusion is flexible, and the author think that this may be helpful? If that is the case, another set of causal inference experiments can provide stronger evidence of choosing diffusion model. Since the proposed method in fact learns a structural causal model. it is straightforward to extend the framework for average treatment effect and individual treatment effect on synthetic data. Under this case, DAG-Diffuser may demonstrate stronger performance compared to baselines. 

In addition, for empirical evaluation on linear Gaussian synthetic dataset, do you use the same variance for the Gaussian noise variable? If not, the model is not identifiable, and invalidate your statements. 

Some minor comments:
1. For eq.2, is the $Y_0$ and $Y_T$ are in wrong order?

2. You should add some reference on line192. For example, [1].

3. In line 196, you mentioned that "without loss of generality, we assume....", but the invertibility of $f$ and $g$ will hurt the capacity of $f,g$. 

4. For line 201, if f is non-linear, do you still need $Z$ to be non-Gaussian? 

5. Line 261, what is $S_G$? Shouldn't it be $R_G$?

6. For related work, you should also cite some related work on discovery with soft constraints and diffusion model, like [2,3,4,5].

7. Consider introduce the method name DAG-Diffuser before the experiment section. 

8. what is $(10\alpha)/\alpha$ in line 403?


Reference

[1] Hoyer, Patrik, et al. "Nonlinear causal discovery with additive noise models." Advances in neural information processing systems 21 (2008).

[2] Rolland, Paul, et al. "Score matching enables causal discovery of nonlinear additive noise models." International Conference on Machine Learning. PMLR, 2022.

[3] Lachapelle, Sébastien, et al. "Gradient-based neural dag learning." arXiv preprint arXiv:1906.02226 (2019).

[4] Geffner, Tomas, et al. "Deep end-to-end causal inference." arXiv preprint arXiv:2202.02195 (2022).

[5] Sanchez, Pedro, et al. "Diffusion models for causal discovery via topological ordering." arXiv preprint arXiv:2210.06201 (2022).

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698314122

thread_type:
official_review

<thread_object>
creator: reviewer_6nj7

<content>
summary:
This paper adopts diffusion models for differentiable causal discovery. Specifically, a specific function class is considered, where diffusion models are used to model the nonlinear causal relations. Empirical studies on synthetic and real data are provided.

strengths:
Causal discovery with nonlinear relations is an important task, because it is often unrealistic to assume linear relations in practice.

weaknesses:
- The specific nonlinear functional class considered/assumed is highly restrictive. The method cannot handle more general functional causal model, such as nonlinear additive noise models.
- The baselines considered are not adequate.

questions:
- I would suggest the authors to be clear about what type of functional causal model the method can handle, which is not clear from Eq. (10), (11), (12). Specifically, the paper should give a precise formulation in the form of a structural causal model $X=f_i(PA(X_i),Z_i)$.
- Is $f$ and $g$ in Eq. (10) and (11) a variable/element wise function? If so, this should be stated explicitly.
- Only CAM and DAG-GNN are compared for nonlinear models. Several other differentiable nonlinear methods could be included to strengthen the empirical studies, such as those for more general nonlinear causal models (https://arxiv.org/abs/1909.13189, https://arxiv.org/abs/1906.02226), and those for the specific causal model in Eq. (11) (https://arxiv.org/abs/1911.07420, https://arxiv.org/abs/2004.08697)

Minor:
- Proposition 1 follows similar spirit as Zhu et al. (2020), which should be made more explicitly in the main paper. Also, Proposition 1 does not provide any new insight into the optimization procedure because a typical way in differentiable causal discovery is to use augmented Lagrangian, which the paper does, so I would suggest removing this proposition.
- L56: Zheng et al. (2020) did not propose a nonparametric score function but a nonparametric DAG constraint.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730954503339

thread_type:
official_review

<thread_object>
creator: reviewer_fcqq

<content>
summary:
The authors propose using diffusion models for causal discovery and searching for the DAG under continuous optimization frameworks.  The authors claim that the diffusion model has the ability to represent various functions, and the proposed causal discovery approach is able to generate graphs with satisfactory accuracy on observational data generated by either linear or nonlinear causal models.  Experiments on synthetic and real-world datasets were conducted to test the proposed method.

strengths:
The authors present different experiments on synthetic and real-world datasets. The writing is clear and easy to follow.

weaknesses:
a. The method proposed in the paper is incremental, and it heavily relies on NoTears regularization.  The method is just a combination of diffusion model and NoTears constraint. The contribution of the paper is limited. 

b. The benefit of the method is limited. As shown in Table 1 and Table 2. 

c. The abstract is too general and does not provide sufficient information to summarize the content of the article.  It should provide insights into the proposed method.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732342744146

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of revision

comment:
Dear all,

We express our sincere thanks for the time you spent on our manuscript. We
revised our manuscript based on your comments, with the revisions summarized below.

1. Abstract:  abstract is revised so that the insight is more clear. (Reviewer FcqQ)

2. Section 2: more related work including GAE and CausalVAE, causal score matching are discussed. (Reviewer 6nj7, eoGv and CBSw)

3. Section 3: several revisions on statements related to invertibility of functions (line 197) and proposition 1. (Reviewer 6nj7, eoGv, CBSw)

4. Section 4: more experiments are added (GAE, DiffAN, NOTEARS-MLP) with discussions revised acordingly. Pruning method of CAM is also added. (Reviewer 6nj7, eoGv, CBSw)

5. Appendix: open source links (section C) and discussions (section D) are added about the type of nonlinear models and several new baselines. (Reviewer 6nj7)

For details, please kindly see our responses to each reviewer.

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a diffusion-based approach to continuous causal discovery, using a DAG-GNN approach that models the non-linear function with diffusions. This is an interesting direction with a few prior works already. Reviewers unanimously rated the paper as "weak reject" with no one strongly in favor of acceptance.