SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1): To the best of our knowledge, our results present the first upper bound on the explicit sampling complexity of score matching for topological ordering Algorithm 1 in non-linear additive Gaussian noise causal models. This novel contribution provides valuable insights into the efficiency and effectiveness of utilizing score matching for topological ordering in non-linear additive Gaussian noise causal models.
2): By choosing  $\varepsilon^2 = \frac{1}{\sqrt{n}}$ , the bound is modified to  $J_{\mathrm{ESM}}(\hat{s},p(\boldsymbol {x}))\lesssim \frac{\sigma^2d\log nd}{\sqrt{n}}\log \frac{\mathcal{N}_c(1 / n,\mathcal{S})}{\delta}$ . This expression demonstrates that the  $\ell_2$  estimation error converges at a rate of  $\frac{\log n}{\sqrt{n}}$  when the sample size  $n$  is significantly larger than the number of nodes  $d$ .
3): The bound is also related to the number of nodes  $d$ , the variance of the noise in denoising score matching  $\sigma$  and causal model  $\sigma_{i}$ , the covering number of the function space  $\mathcal{N}_c(\frac{1}{n},\mathcal{S})$ , and the upper bound of the data  $C_d$ . If these quantities increase, it is expected that the error of explicit score matching will also increase. This is due to the increased difficulty in accurately estimating the score function.
4): Theorem 1 is rooted in the generalization by sampling complexity bound. It is independent of the specific training algorithm used. The results are broadly applicable and can be seamlessly extended to encompass larger batch GD.
Next, we will establish a connection between score matching and the precise identification of the topological ordering.
3.3 Error bound for topological order in causal discovery
Based on the previously mentioned sample complexity bound of score matching, we establish an upper bound on the error rate of the topological ordering of the causal model obtained through Algorithm 1.
Theorem 2. Given a DNN defined by Eq. (3) trained by SGD with a step size  $\eta = \mathcal{O}\left(\frac{1}{poly(n,L)m\log^2m}\right)$  for minimizing empirical score matching objective. Then under Assumption 2, for  $m\geq poly(n,L)$ , with probability at least:
$$
1 - \exp (- \Theta (d)) - (L + 1) \exp (- \Theta (m)) - 2 n \exp (- \frac {n C _ {m} ^ {2} d ^ {2}}{2 ^ {4 L + 5} (\log m) ^ {2} (m ^ {2} + d ^ {2})}),
$$
over the randomness of initialization  $\mathbf{W}$  and training data that Algorithm 1 can completely recover the correct topological order of the non-linear additive Gaussian noise model.
1): The foundation of Theorem 2 rests upon Theorem 1, it can be seen as an embodiment of applying the upper bound of score matching for causal discovery. To the best of our knowledge, our results provide the first upper bound on the error rate of topological ordering in non-linear additive Gaussian noise causal models using Algorithm 1.
2): Considering that when  $m \sim d$  and  $L \sim 1$  the probability degenerates to:
$$
1 - \Theta (e ^ {- m}) - 2 n \exp \left(- \Theta \big (\frac {n C _ {m} ^ {2}}{(\log m) ^ {2}} \big)\right).
$$
The first term of the error arises due to the initialization of the neural network. As for the second term of the error, if the number of training data  $n$  satisfies  $\frac{n}{\log n} \gtrsim (\log m)^2$ , then it will have that  $2n \exp \left( -\Theta \left( \frac{nC_m^2}{(\log m)^2} \right) \right) \lesssim 1$ . This implies that the second term of the error probability exhibits linear convergence towards 0 when  $n$  is sufficiently large. Therefore, when the sample size  $\frac{n}{\log n} \gtrsim (\log m)^2$ , the contribution of the second term to the full error becomes negligible.
3): The theorem reveals that a smaller value of the constant  $C_m$  increases the probability of algorithm failure. This observation further confirms our previous statement that a smaller average second derivative of the nonlinear function makes it more challenging to identify the causal relationship in the model. Additionally, when the causal relationship is linear, our theorem does not provide any guarantee for the performance of Algorithm 1.
4): Consider the two variables case. If a child node is almost a deterministic function of its parents, the constant  $C_m$  can take on arbitrarily large values, according to Assumption 2. Consequently, the second term of the error probability,  $2n\exp \left(-\Theta \left(\frac{nC_m^2}{(\log m)^2}\right)\right)$ , tends to zero. This implies that the errors in Algorithm 1 are primarily caused by the random initialization of the neural network. The identifiability of this setting is consistent with classical results Daniusis et al. [2010], Janzing et al. [2015]. Intuitively, as long as the non-linearity is chosen independently of the noise of the parent variable<sup>2</sup>, the application of the non-linearity will increase the distance to the reference distribution of the parent variable (in our case Gaussian). Note that for the derivative in Assumption Assumption 2 to be defined, the parent node cannot be fully deterministic.
5): Instead of focusing on the kernel regime, we directly cover the more general neural network training. The kernel approach of Rolland et al. [2022] is a special case of our analysis. The basis of Theorem 2 lies in the proof of SGD/GD convergence of the neural network. These convergence outcomes also apply to BatchGD, as demonstrated in Jentzen and Kröger [2021]. Hence, Theorem 2 can naturally be expanded to incorporate Batch GD as well.
Proof sketch: The proof of Theorem 2 can be divided into three steps. The first and most important step is to derive the upper bound of  $\frac{\partial s_i(\boldsymbol{x})}{\partial x^{(i)}}$ . Here, we utilize the properties of deep ReLU neural networks to derive the distribution relationship between features of adjacent layers, then accumulate them and combine it with the properties of Gaussian initialization, yielding the upper bound for  $\frac{\partial s_i(\boldsymbol{x})}{\partial x^{(i)}}$ . The second step is to use the upper bound of  $\frac{\partial s_i(\boldsymbol{x})}{\partial x^{(i)}}$  obtained in the first step combined with the concentration inequality to derive the upper bound of the error of  $\mathrm{Var}\left(\frac{\partial s_i(\boldsymbol{x})}{\partial x^{(i)}}\right)$ . The third step is to compare the upper bound in the second step with Lemma 1 to obtain the probability of successfully selecting leaf nodes in each step. After accumulation, we can obtain the probability that Algorithm 1 can completely recover the correct topological order of the non-linear additive Gaussian noise model.
4 Theoretical results for score-based generative modeling (SGM)
In this section, we present the additional assumption required for the theoretical analysis of score matching in score-based generative modeling. Then, we provide the sample complexity bound associated with score matching in this framework. The full proof in this section is deferred to Appendix G.
Assumption 3 (Bounded data). We assume that the input data satisfy  $\| \pmb {x}\| _2\leq C_d$  ，  $\pmb {x}\sim p_0$
Remark: Bounded data is standard in deep learning theory and also commonly used in practice [Du et al., 2019b,a, Allen-Zhu et al., 2019, Oymak and Soltanolkotabi, 2020, Malach et al., 2020].
$^{2}$  Danius et al. [2010], Janzing et al. [2015] have formalized independence of distribution and function via an information geometric orthogonality condition that refers to a reference distribution (e.g., Gaussian)
Theorem 3. Given a DNN defined by Eq. (3) trained by SGD for minimizing empirical denoising score matching loss Eq. (8). Suppose Assumption 1 and 3 are satisfied. For any  $\varepsilon \in (0,1)$  and  $\delta \in (0,1)$ . Then with probability at least  $1 - 2\delta - 2L\exp(-\Omega(m))$  over the randomness of initialization  $\mathbf{W}$  and noise  $\epsilon$  in denoising score matching, it holds:
$$
\frac {1}{T - t _ {0}} \int_ {t _ {0}} ^ {T} \| \nabla \log p _ {t} (\cdot) - \hat {\boldsymbol {s}} (\cdot , t) \| _ {\ell^ {2} (p _ {t})} ^ {2} \mathrm {d} t \lesssim \frac {1}{n \varepsilon^ {2}} \left(\frac {d (T - \log (t _ {0}))}{T - t _ {0}} + C _ {d} ^ {2}\right) \log \frac {\mathcal {N} _ {c} (\frac {1}{n} , \mathcal {S})}{\delta} + \frac {1}{n} + d \varepsilon^ {2},
$$
where the  $\mathcal{N}_c\left(\frac{1}{n},S\right)$  is the covering number of the function space  $s$  for deep ReLU neural network.
1): Theorem 3 and Theorem 1 study similar problems between causal discovery and score-based generative modeling and share similar techniques drawn from statistical learning theory and deep learning theory. These two domains are connected by a common theoretical foundation centered on the upper bound of score matching.
2): Our result extends the results for score matching in diffusion models presented in Chen et al. [2023a] which rested on the assumption of low-dimensional data structures, employing this to decompose the score function and engineer specialized network architectures for the derivation of the upper bound. Our work takes a distinct route. Our conclusions are based on the general deep ReLU neural network instead of a specific encoder-decoder network and do not rely on the assumptions of low-dimensional data used in Chen et al. [2023a]. We harness the inherent traits and conventional techniques of standard deep ReLU networks to directly deduce the upper error bound. This broader scope allows for a more comprehensive understanding of the implications and applicability of score-based generative modeling in a wider range of scenarios.
3): Similar to Theorem 1, by choosing  $\varepsilon^2 = \frac{1}{\sqrt{n}}$ , we can obtain the best bound  $\frac{1}{T - t_0}\int_{t_0}^T\|\nabla\log p_t(\cdot) - \hat{s}(\cdot,t)\|_{\ell^2(p_t)}^2\mathrm{d}t \lesssim \frac{1}{\sqrt{n}}\left(\frac{d(T-\log(t_0))}{T-t_0} + C_d^2\right)\log\frac{\mathcal{N}_c(\frac{1}{n},S)}{\delta}$ . This expression demonstrates that the  $\ell_2$  estimation error converges at a rate of  $\frac{1}{\sqrt{n}}$  when the sample size  $n$  is significantly larger than the dimensionality  $d$  and time steps  $T$ .
4): The bound is also related to the data dimension  $d$ , the variance of the noise in denoising score matching  $\sigma$ , the covering number of the function space  $\mathcal{N}_c(\frac{1}{n},\mathcal{S})$ , and the upper bound of the data  $C_d$ . If these quantities increase, it is expected that the error of explicit score matching will also increase. This is due to the increased difficulty in accurately estimating the score function.
5): When  $t_0 = 0$ , the theorem lacks meaning. However, when  $T \gg t_0 \sim 1$ , the bound simplifies to  $\frac{d + C_d^2}{\sqrt{n}} \log \frac{\mathcal{N}_c(\frac{1}{n}, S)}{\delta}$ . This indicates that when  $T$  is sufficiently large, the loss estimated by the score function in the diffusion model becomes independent of time steps  $T$ .
6): Similar to Theorem 1, the result of Theorem 3 is also broadly applicable and can be seamlessly extended to encompass larger batch GD.
5 Numerical evidence
We conducted a series of experiments to validate the theoretical findings presented in the paper. We took inspiration from the code provided inRolland et al. [2022] and employed the structural Hamming distance (SHD) between the generated output and the actual causal graph to assess the outcomes. The ensuing experimental outcomes for SHD, vary across causal model sizes  $d$ , sample sizes  $n$ , and  $C_m$ . The experimental results are shown in Tables 1 to 3
Table 1: Fixed model size  $d = 100$  and the number of sampling  $n = 100$ , SHD results of causal discovery using Algorithm 1 for different  $C_m$  values (10 runs).
Cm	1	2	4	8	16
SHD	2941.0 ± 29.5	2905.7 ± 50.8	2900.6 ± 80.8	2637.1 ± 200.4	1512.4 ± 283.6
Cm	32	64	128	256	512
SHD	413.9 ± 93.4	55.0 ± 16.0	23.9 ± 4.6	21.2 ± 5.0	13.8 ± 1.8
Table 2: Fixed model size  $d = 10$  and  $C_m = 1$ , SHD results of causal discovery using Algorithm 1 for the different number of sampling  $n$  (10 runs).
n	5	10	20	40	80	100	160
SHD	31.7 ± 2.1	27.8 ± 4.1	23.3 ± 2.7	23.0 ± 4.0	18.4 ± 3.3	16.5 ± 3.4	13.0 ± 4.0
Table 3: Fixed the number of sampling  $n = 10$  and  $C_m = 1$ , SHD results of causal discovery using Algorithm 1 for the different model size  $d$  (10 runs).
d	5	10	20	40	80	100
SHD	4.5 ± 2.0	29.6 ± 2.2	124.3 ± 4.6	522.8 ± 11.6	1965.4 ± 18.7	2923.7 ± 38.5
Analyzing the experimental outcomes, we find a notable pattern: higher values of  $C_m$ , augmented sample sizes  $n$ , and reduced model size  $d$  all contribute to the performance of Algorithm 1 which is consistent with the insights from Theorem 2.
Score matching: Score Matching was initially introduced by Hyvarinen [2005] and extended to energy-based models by Song and Ermon [2019]. Subsequently, Vincent [2011] proposed denoising score matching, which transforms the estimation of the score function for the original distribution into an estimation for the noise distribution, effectively avoiding the need for second derivative computations. Other methods, such as sliced score matching [Song et al., 2020], denoising likelihood score matching [Chao et al., 2022], and kernel-based estimators, have also been proposed for score matching. The relationship between score matching and Fisher information [Shao et al., 2019], as well as Langevin dynamics [Hyvarinen, 2007], has been explored. On the theoretical side, Wenliang and Kanagawa [2020] introduced the concept of "blindness" in score matching, while Koehler et al. [2023] compared the efficiency of maximum likelihood and score matching, although their results primarily focus on exponential family distributions. Our paper, for the first time, analyzes the sample complexity bounds of the score function estimating in causal inference.
Causal discovery: The application of score methods for causal inference for linear additive models began with Ghoshal and Honorio [2018], which proposed a causal structure recovery method based on topological ordering from the precision matrix (equivalent to the score in that setting). Under certain noise variance assumptions, their method can reliably recover the DAG in polynomial time and sample complexity.
In recent years, there have been numerous algorithms developed for causal inference in non-linear additive models. GraNDAG [Lachapelle et al., 2021] aims to maximize the likelihood of the observed data under this model and enforces a continuous constraint to ensure the acyclicity of the causal graph Rolland et al. [2022] proposed a novel approach for causal inference which utilize score matching algorithms as a foundation for topological ordering and then employ sparse regression techniques to prune the DAG. Subsequently, Montagna et al. [2023a] extended the method to non-Gaussian noise, Sanchez et al. [2023] proposed to use diffusion models to fit the score function, and Montagna et al. [2023b] proposed a new scalable score-based preliminary neighbor search techniques.
Although advances have been achieved in leveraging machine learning for causal discovery, there is generally a lack of further research on error bounds. Other studies concentrate on broader non-parametric models but depend on various assumptions like faithfulness, restricted faithfulness, or the sparsest Markov representation [Spirtes et al., 2000, Raskutti and Uhler, 2018, Solus et al., 2021]. These approaches employ conditional independence tests and construct a graph that aligns with the identified conditional independence relations [Zhang, 2008].
Theoretical analysis of score-based generative modeling: Existing work mainly focuses on two fundamental questions: "How do diffusion models utilize the learned score functions to estimate the data distribution?" [Chen et al., 2023b, De Bortoli et al., 2021, De Bortoli, 2022, Lee et al., 2022,
[2023] and "Can neural networks effectively approximate and learn score functions? What are the convergence rate and bounds on the sample complexity?" [Chen et al., 2023a].
Specifically, De Bortoli et al. [2021] and Lee et al. [2022] studied the convergence guarantees of diffusion models under the assumptions that the score estimator is accurate under the  $\ell_1$  and  $\ell_2$  norms. Concurrently Chen et al. [2023b] and Lee et al. [2023] extended previous results to distributions with bounded moments. De Bortoli [2022] studied the distribution estimation guarantees of diffusion models for low-dimensional manifold data under the assumption that the score estimator is accurate under the  $\ell_1$  or  $\ell_2$  norms.
However, these theoretical results rely on the assumption that the score function is accurately estimated, while the estimation of the score function is largely untouched due to the non-convex training dynamics. Recently, Chen et al. [2023a] provided the first sample complexity bounds for score function estimation in diffusion models. However, their result is based on the assumption that the data distribution is supported on a low-dimensional linear subspace and they use a specialized Encoder-Decoder network instead of a general deep neural network. As a result, a complete theoretical picture of score-based generative modeling is still lacking.
In this work, we investigate the sample complexity error bounds of Score Matching using deep ReLU neural networks under two different problem settings: causal discovery and score-based generative modeling. We provide a sample complexity analysis for the estimation of the score function in the context of causal discovery for nonlinear additive Gaussian noise models, with a convergence rate of  $\frac{\log n}{n}$ . Furthermore, we extend the sample complexity bounds for the estimation of the score function in the ScoreSDE method to general data and achieve a convergence rate of  $\frac{1}{n}$ . Additionally, we provide an upper bound on the error rate of the state-of-the-art causal discovery method SCORE [Rolland et al., 2022], showing that the error rate of this algorithm converges linearly with respect to the number of training data.
A core limitation of this work is limiting our results to the Gaussian noise assumption. In fact, non-linear mechanisms with additive non-gaussian noise are also identifiable under mild additional assumptions [Peters et al., 2014] and Montagna et al. [2023a] already extended the score-matching approach of Rolland et al. [2022] to that setting. Relaxing this assumption would also allow us to apply our bounds to interesting corner cases, such as linear non-gaussian [Ghoshal and Honorio, 2018], and non-gaussian deterministic causal relations [Daniusis et al., 2010, Janzing et al., 2015]. It may be possible for this assumption to be relaxed in future work, but we argue that the added challenge, the significant difference in algorithms, and the standalone importance of the non-linear Gaussian case justify our focus.
In addition, we make other assumptions that limit the general applicability of our bounds. In particular, the assumption of the Lipschitz property for the score function imposes a strong constraint on the model space. Further investigating the relationship between the noise, the properties of the nonlinear functions in the causal model Eq. (4), and the resulting Lipschitz continuity of the score function would be an interesting extension of this work.
We are thankful to the reviewers for providing constructive feedback and Kun Zhang and Dominik Janzing for helpful discussion on the special case of deterministic children. This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043). This work was supported by the Swiss National Science Foundation (SNSF) under grant number 200021_205011. Francesco Locatello did not contribute to this work at Amazon. Corresponding author: Zhenyu Zhu.
P. Spirtes, C. N. Glymour, and R. Scheines. Causation, prediction, and search. MIT press, 2000.
M. Teyssier and D. Koller. Ordering-based search: A simple and effective algorithm for learning bayesian networks, 2012.
H. R. Varian. Causal inference in economics and marketing. Proceedings of the National Academy of Sciences, 2016.
R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Taylor & Francis, 2018.
P. Vincent. A connection between score matching and denoising autoencoders. Neural computation, 2011.
X. Wang, Y. Du, S. Zhu, L. Ke, Z. Chen, J. Hao, and J. Wang. Ordering-based causal discovery with reinforcement learning. In International Joint Conferences on Artificial Intelligence (IJCAI), 2021.
L. K. Wenliang and H. Kanagawa. Blindness of score-based methods to isolated components and mixing proportions, 2020.
J. Zhang. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence, 2008.
Z. Zhu, F. Liu, G. Chrysos, and V. Cevher. Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization). In Advances in neural information processing systems (NeurIPS), 2022.