Title: Sample Complexity Bounds for Score-Matching: Causal Discovery and Generative Modeling
Venue: NeurIPS 2023
Accepted: yes
Keywords: Causal discovery

Score matching

Score-based generative modeling
Abstract: This paper provides statistical sample complexity bounds for score-matching and its applications in causal discovery. We demonstrate that accurate estimation of the score function is achievable by training a standard deep ReLU neural network using stochastic gradient descent. We establish bounds on the error rate of recovering causal relationships using the score-matching-based causal discovery method of Rolland et al. [2022], assuming a sufficiently good estimation of the score function. Finally, we analyze the upper bound of score-matching estimation within the score-based generative modeling, which has been applied for causal discovery but is also of independent interest within the domain of generative models.

Sample Complexity Bounds for Score-Matching: Causal Discovery and Generative Modeling

Abstract

This paper provides statistical sample complexity bounds for score-matching and its applications in causal discovery. We demonstrate that accurate estimation of the score function is achievable by training a standard deep ReLU neural network using stochastic gradient descent. We establish bounds on the error rate of recovering causal relationships using the score-matching-based causal discovery method of Rolland et al. [2022], assuming a sufficiently good estimation of the score function. Finally, we analyze the upper bound of score-matching estimation within the scorebased generative modeling, which has been applied for causal discovery but is also of independent interest within the domain of generative models.

1 Introduction

Score matching Hyvärinen [2005], an alternative to the maximum likelihood principle for unnormalized probability density models with intractable partition functions, has recently emerged as a new state-of-the-art approach that leverages machine learning for scalable and accurate causal discovery from observational data Rolland et al. [2022]. However, the theoretical analysis and guarantees in the finite sample regime are underexplored for causal discovery even beyond score-matching approaches.

Contributions: In this work, we give the first sample complexity error bounds for score-matching using deep ReLU neural networks. With this, we obtain the first upper bound on the error rate of the method proposed by Rolland et al. [2022] to learn the topological ordering of a causal model from observational data. Thanks to the wide applicability of score-matching in machine learning, we also discuss applications to the setting of score-based generative modeling. Our main contributions are:

We provide the analysis of sample complexity bound for the problem of score function estimation in causal discovery for non-linear additive Gaussian noise models which has a convergence rate of log n/n with respect to the number of data. Importantly, our results require only mild additional assumptions, namely that the non-linear relationships among the causal variables are bounded and that the score function is Lipschitz. To the best of our knowledge, this is the first work to provide sampling complexity bounds for this problem.

We provide the first analysis of the state-of-the-art topological ordering-based causal discovery method SCORE [Rolland et al., 2022] and provide a correctness guarantee for the obtained topological order. Our results demonstrate that the algorithm's error rate converges linearly with respect to the number of training data. Additionally, we establish a connection between the algorithm's error rate and the average second derivative (curvature) of the non-linear relationships among the causal variables, discussing the impact of the causal model's inherent characteristics on the algorithm's error rate in identification.

We present sample complexity bounds for the score function estimation problem in the standard score-based generative modeling method, ScoreSDE [Song et al., 2021]. In contrast to previous results [Chen et al., 2023a], our bounds do not rely on the assumption of low-dimensional input data, and we extend the applicability of the model from a specific encoder-decoder network architecture to a general deep ReLU neural network.

High-level motivation and background: Causal discovery and causal inference refer to the process of inferring causation from data and reasoning about the effect of interventions. They are highly relevant in fields such as economics [Varian, 2016], biology [Sachs et al., 2005], and healthcare [Sanchez et al., 2022]. In particular, some causal discovery methods aim to recover the causal structure of a problem solely based on observational data.

The causal structure is typically represented as a directed acyclic graph (DAG), where each node is associated with a random variable, and each edge represents a causal mechanism between two variables. Learning such a model from data is known to be NP-hard [Chickering, 1996]. Traditional approaches involve testing for conditional independence between variables or optimizing goodness-offit measures to search the space of possible DAGs. However, these greedy combinatorial optimization methods are computationally expensive and difficult to extend to high-dimensional settings.

An alternative approach is to reframe the combinatorial search problem as a topological ordering task [Teyssier and Koller, 2012, Solus et al., 2021, Wang et al., 2021, Rolland et al., 2022, Montagna et al., 2023b,a, Sanchez et al., 2023], where nodes are ordered from leaf to root. This can significantly speed up the search process in the DAG space. Once a topological ordering is found, a feature selection algorithm can be used to prune potential causal relations between variables, resulting in a DAG.

Recently, Rolland et al. [2022] proposed the SCORE algorithm, which utilizes the Jacobian of the score function to perform topological ordering. By identifying which elements of the Jacobian matrix of the score function remain constant across all data points, leaf nodes can be iteratively identified and removed. This approach provides a systematic way to obtain the topological ordering and infer the causal relations within the entire model. This method has achieved state-of-the-art results on multiple tasks Rolland et al. [2022] and has been extended to improve scalability Montagna et al. [2023b] also using diffusion models Sanchez et al. [2023] and to non-Gaussian noise Montagna et al. [2023a]. Interestingly, these approaches separate the concerns of statistical estimation of the score function from the causal assumption used to infer the graph (e.g., non-linear mechanisms and additive Gaussian noise). This opens an opportunity to study the convergence properties of these algorithms in the finite data regime, which is generally under-explored in the causal discovery literature. In fact, if we had error bounds on the score estimate without additional complications from causal considerations, we could study their downstream effect when the score is used for causal discovery.

Unfortunately, this is far from trivial as the theoretical research on score matching lags behind its empirical success and progress would have far-reaching implications. Even beyond causal discovery, error bounds on the estimation of the score functions would be useful for score-based generative modeling (SGM) [Song and Ermon, 2019, Song et al., 2021]. These have achieved state-of-the-art performance in various tasks, including image generation [Dhariwal and Nichol, 2021] and audio synthesis [Kong et al., 2021]. There has been significant research investigating whether accurate score estimation implies that score-based generative modeling provably converges to the true data distribution in realistic settings [Chen et al., 2023b, Lee et al., 2022, 2023]. However, the error bound of score function estimation in the context of score-based generative modeling remains an unresolved issue due to the non-convex training dynamics of neural network optimization.

Notations: We use the shorthand [ n ] := { 1 , 2 , . . . , n } for a positive integer n . We denote by a ( n ) ≲ b ( n ) : there exists a positive constant c independent of n such that a ( n ) ⩽ cb ( n ) . The Gaussian distribution is N ( µ, σ 2 ) with the µ mean and the σ 2 variance. We follow the standard Bachmann-Landau notation in complexity theory e.g., O , o , Ω , and Θ for order notation. Due to space constraints, a detailed notation is deferred to Appendix A.

2 Preliminaries

As this paper concerns topics in score matching estimation, diffusion models, neural network theory, and causal discovery, we first introduce the background and problem setting of our work.

2.1 Score matching

For a probability density function p ( x ) , we call the score function the gradient of the log density with respect to the data x . To estimate the score function ∇ log p ( x ) , we can minimize the ℓ 2 loss over the function space S .

The corresponding objective function to be minimized is the expected squared error between the true score function and the neural network:

We refer to this formulation as explicit score matching (ESM).

Denoising score matching (DSM) is proposed by Vincent [2011] to convert the inference of the score function in ESM into the inference of the random noise and avoid the computing of the second derivative. For the sampled data x , ˆ x is obtained by adding unit Gaussian noise to x . i.e. ˆ x = x + ϵ , ϵ ∼ N (0 , σ 2 I ) . We can derive the conditional probability distribution and its score function:

Then the DSM is defined by:

Vincent [2011] have proven that minimizing DSM is equivalent to minimizing ESM and does not depend on the particular form of p (ˆ x | x ) or p ( x ) .

2.2 Neural network and function space

In this work, we consider a standard depthL widthm fully connected ReLU neural network. Formally, we define a DNN with the output s l ( x ) in each layer

where the input is x ∈ R d , the output is s L ( x ) ∈ R d , the weights of the neural networks are W 1 ∈ R m × d , W l ∈ R m × m , l = 2 , . . . , L -1 and W L ∈ R d × m . The neural network parameters formulate the tuple of weight matrices W := { W i } L i =1 ∈ { R m × d × ( R m × m ) L -2 × R d × m } . The S denotes the function space of Eq. (3).

The ϕ = max(0 , x ) is the ReLU activation function. According to the property ϕ ( x ) = xϕ ′ ( x ) of ReLU, we have s l = D l W l s l -1 , where D l is a diagonal matrix defined as below.

Definition 1 (Diagonal sign matrix) . For l ∈ [ L -1] and k ∈ [ m ] , the diagonal sign matrix D l is defined as: ( D l ) k,k = 1 { ( W l s l -1 ) k ≥ 0 } .

Initialization: We make the standard random Gaussian initialization [ W l ] i,j ∼ N (0 , 2 m ) for l ∈ [ L -1] and [ W L ] i,j ∼ N (0 , 1 d ) .

2.3 Causal discovery

In this paper, we follow the setting in Rolland et al. [2022] and consider the following causal model, a random variable x ∈ R d is generated by:

where f i is a non-linear function, ϵ i ∼ N (0 , σ 2 i ) and PA i ( x ) represent the set of parents of x ( i ) in x . Then we can write the probability distribution function of x as:

For such non-linear additive Gaussian noise models Eq. (4), Rolland et al. [2022] provides Algorithm 1 to learn the topological order by score matching as follows:

Algorithm 1 SCORE matching causal order search (Adapted from Algorithm 1 in Rolland et al. [2022])

Get the final DAG by pruning the full DAG associated with the topological order π .

2.4 Score-based generative modeling (SGM)

In this section, we give a brief overview of SGM following Song et al. [2021], Chen et al. [2023b].

2.4.1 Score-based generative modeling with SDEs

Forward process: The success of previous score-based generative modeling methods relies on perturbing data using multiple noise scales, and the proposal of the diffusion model is to expand upon this concept by incorporating an infinite number of noise scales. This will result in the evolution of perturbed data distributions as the noise intensity increases, which will be modeled through a stochastic differential equation (SDE).

The expression describes x t , where the standard Wiener process (also known as Brownian motion) is denoted as w , the drift coefficient of x t is represented by a vector-valued function called f , and the diffusion coefficient of x t is denoted as g t , a scalar function. In this context, we will refer to the probability density of x t as p t , and the transition kernel from x s to x t as p st ( x t | x s ) , where 0 ≤ s < t ≤ T . The Ornstein-Uhlenbeck (OU) process is a Gaussian process that is both timehomogeneous and a Markov process. It is distinct in that its stationary distribution is equivalent to the standard Gaussian distribution γ d on R d .

Reverse process: We can obtain samples of x 0 ∼ p SDE 0 by reversing the process starting from samples of x T ∼ p SDE T . An important finding is that the reversal of a diffusion process is a diffusion process as well. It operates in reverse time and is described by the reverse-time SDE:

When time is reversed from T to 0 , w is a standard Wiener process with an infinitesimal negative timestep of d t . The reverse diffusion process can be derived from Eq. (7) once the score of each marginal distribution, ∇ log p t ( x t ) , is known for all t . By simulating the reverse diffusion process, we can obtain samples from p SDE 0 .

Some special settings: In order to simplify the writing of symbols and proofs, in this work we choose that f ( x t , t ) = -1 2 x t and g ( t ) = 1 which has been widely employed in prior research [Chen et al., 2023a,b, De Bortoli et al., 2021] for theoretical analysis in Ornstein-Uhlenbeck process in score-based generative modeling.

2.4.2 Score matching in diffusion model

We aim to minimize the equivalent objective for score matching:

The transition kernel has an analytical form ∇ x t log p 0 t ( x t | x 0 ) = -x t -α ( t ) x 0 h ( t ) , where α ( t ) = e -t 2 and h ( t ) = 1 -α ( t ) 2 = 1 -e -t .

The empirical score matching loss is:

where the loss function ℓ ( x ( i ) ; s ) is defined as:

Here we choose w ( t ) = 1 T -t 0 , and we define the expected loss L ( · ) = E x ∼ p 0 [ ˆ L ( · )] .

3 Theoretical results for causal discovery

In this section, we state the main theoretical results of this work. We present the assumptions on nonlinear additive Gaussian noise causal models in Section 3.1. Then, we present the sample complexity bound for score matching in causal discovery in Section 3.2. In Section 3.3 we provide the upper bound on the error rate for causal discovery using the Algorithm 1. The full proofs of Theorem 1 and 2 are deferred to Appendix E and F, respectively.

3.1 Assumptions

Assumption 1 (Lipschitz property of score function) . The score function ∇ log p ( · ) is 1 -Lipschitz.

Remark: The Lipschitz property of the score function is a standard assumption commonly used in the existing literature [Block et al., 2020, Lee et al., 2022, Chen et al., 2023b,a]. However, for causal discovery, this assumption limits the family of mechanisms that we can cover.

Assumption 2 (Structural assumptions of causal model) . Let p be the probability density function of a random variable x defined via a non-linear additive Gaussian noise model Eq. (4) . Then, ∀ i ∈ [ d ] the non-linear function is bounded, | f i | ≤ C i . And ∀ i, j ∈ [ d ] , if j is one of the parents of i , i.e. x ( j ) ⇒ x ( i ) , then there exist a constant C m that satisfy:

Remark: This is a novel assumption that we introduce, relating the average second derivative of a mechanism (related to its curvature) to the noise variance of the child variable. This will play a crucial yet intuitive role in our error bound: identifiability is easier when there is sufficient non-linearity of a mechanism with respect to the noise of the child variable. Consider the example of a quadratic mechanism, where the second derivative is the leading constant of the polynomial. If this constant is small (e.g., close to zero), the mechanism is almost linear and we may expect that the causal model should be harder to identify. Similarly, if the child variable has a very large variance, one may expect it to be more difficult to distinguish cause from effect, as the causal effect of the parent is small compared to the noise of the child. According to Assumption 2, we can derive the identified ability margin for leaf nodes and parent nodes.

Lemma 1. If a non-linear additive Gaussian noise model Eq. (4) satisfies Assumption 2. Then, ∀ i, j ∈ [ d ] , we have:

This lemma intuitively relates our identifiability margin with the decision rule of SCORE Rolland et al. [2022] to identify leaves. Non-leaf nodes should have the variance of their score Jacobian sufficiently far from zero. As one may expect, we will see in Theorem 2 that the closer C m is to zero, the more likely it is that the result of the algorithm will be incorrect given finite samples.

3.2 Error bound for score matching in causal discovery

We are now ready to state the main result of the score matching in causal discovery. We provide the sample complexity bounds of the explicit score matching Eq. (1) that using denoising score matching Eq. (2) in Algorithm 1 for non-linear additive Gaussian noise models Eq. (4).

Theorem 1. Given a DNN defined by Eq. (3) trained by SGD for minimizing empirical denoising score matching objective. Suppose Assumption 1 and 2 are satisfied. For any ε ∈ (0 , 1) and δ ∈ (0 , 1) , if σ i ≂ σ and C i σ i ≂ 1 , ∀ i ∈ [ d ] . Then with probability at least 1 -2 δ -4 exp( -d 32 ) -2 L exp( -Ω( m )) -1 nd over the randomness of initialization W , noise ϵ and ϵ i , it holds that:

where the N c ( 1 n , S ) is the covering number of the function space S for deep ReLU neural network.

Remark:

1): To the best of our knowledge, our results present the first upper bound on the explicit sampling complexity of score matching for topological ordering Algorithm 1 in non-linear additive Gaussian noise causal models. This novel contribution provides valuable insights into the efficiency and effectiveness of utilizing score matching for topological ordering in non-linear additive Gaussian noise causal models.

2): By choosing ε 2 = 1 √ n , the bound is modified to J ESM (ˆ s , p ( x )) ≲ σ 2 d log nd √ n log N c (1 /n, S ) δ . This expression demonstrates that the ℓ 2 estimation error converges at a rate of log n √ n when the sample size n is significantly larger than the number of nodes d .

3): The bound is also related to the number of nodes d , the variance of the noise in denoising score matching σ and causal model σ i , the covering number of the function space N c ( 1 n , S ) , and the upper bound of the data C d . If these quantities increase, it is expected that the error of explicit score matching will also increase. This is due to the increased difficulty in accurately estimating the score function.

4): Theorem 1 is rooted in the generalization by sampling complexity bound. It is independent of the specific training algorithm used. The results are broadly applicable and can be seamlessly extended to encompass larger batch GD.

Next, we will establish a connection between score matching and the precise identification of the topological ordering.

3.3 Error bound for topological order in causal discovery

Based on the previously mentioned sample complexity bound of score matching, we establish an upper bound on the error rate of the topological ordering of the causal model obtained through Algorithm 1.

Theorem 2. Given a DNN defined by Eq. (3) trained by SGD with a step size η = O ( 1 poly ( n,L ) m log 2 m ) for minimizing empirical score matching objective. Then under Assumption 2, for m ≥ poly ( n, L ) , with probability at least:

over the randomness of initialization W and training data that Algorithm 1 can completely recover the correct topological order of the non-linear additive Gaussian noise model.

Remark:

1): The foundation of Theorem 2 rests upon Theorem 1, it can be seen as an embodiment of applying the upper bound of score matching for causal discovery. To the best of our knowledge, our results provide the first upper bound on the error rate of topological ordering in non-linear additive Gaussian noise causal models using Algorithm 1.

2): Considering that when m ≂ d and L ≂ 1 the probability degenerates to:

The first term of the error arises due to the initialization of the neural network. As for the second term of the error, if the number of training data n satisfies n log n ≳ (log m ) 2 , then it will have that

2 n exp ( -Θ ( nC 2 m (log m ) 2 )) ≲ 1 . This implies that the second term of the error probability exhibits linear convergence towards 0 when n is sufficiently large. Therefore, when the sample size n log n ≳ (log m ) 2 , the contribution of the second term to the full error becomes negligible.

3): The theorem reveals that a smaller value of the constant C m increases the probability of algorithm failure. This observation further confirms our previous statement that a smaller average second derivative of the nonlinear function makes it more challenging to identify the causal relationship in the model. Additionally, when the causal relationship is linear, our theorem does not provide any guarantee for the performance of Algorithm 1.

4): Consider the two variables case. If a child node is almost a deterministic function of its parents, the constant C m can take on arbitrarily large values, according to Assumption 2. Consequently, the

2

second term of the error probability, 2 n exp ( -Θ ( nC m (log m ) 2 )) , tends to zero. This implies that the errors in Algorithm 1 are primarily caused by the random initialization of the neural network. The identifiability of this setting is consistent with classical results Daniušis et al. [2010], Janzing et al. [2015]. Intuitively, as long as the non-linearity is chosen independently of the noise of the parent variable 2 , the application of the non-linearity will increase the distance to the reference distribution of the parent variable (in our case Gaussian). Note that for the derivative in Assumption Assumption 2 to be defined, the parent node cannot be fully deterministic.

5): Instead of focusing on the kernel regime, we directly cover the more general neural network training. The kernel approach of Rolland et al. [2022] is a special case of our analysis. The basis of Theorem 2 lies in the proof of SGD/GD convergence of the neural network, These convergence outcomes also apply to BatchGD, as demonstrated in Jentzen and Kröger [2021]. Hence, Theorem 2 can naturally be expanded to incorporate Batch GD as well.

Proof sketch: The proof of Theorem 2 can be divided into three steps. The first and most important step is to derive the upper bound of ∂s i ( x ) ∂x ( i ) . Here, we utilize the properties of deep ReLU neural networks to derive the distribution relationship between features of adjacent layers, then accumulate them and combine it with the properties of Gaussian initialization, yielding the upper bound for ∂s i ( x ) ∂x ( i ) . The second step is to use the upper bound of ∂s i ( x ) ∂x ( i ) obtained in the first step combined with the concentration inequality to derive the upper bound of the error of Var ( ∂s i ( x ) ∂x ( i ) ) . The third step is to compare the upper bound in the second step with Lemma 1 to obtain the probability of successfully selecting leaf nodes in each step. After accumulation, we can obtain the probability that Algorithm 1 can completely recover the correct topological order of the non-linear additive Gaussian noise model.

4 Theoretical results for score-based generative modeling (SGM)

In this section, we present the additional assumption required for the theoretical analysis of score matching in score-based generative modeling. Then, we provide the sample complexity bound associated with score matching in this framework. The full proof in this section is deferred to Appendix G.

Assumption 3 (Bounded data) . We assume that the input data satisfy ∥ x ∥ 2 ≤ C d , x ∼ p 0 .

Remark: Bounded data is standard in deep learning theory and also commonly used in practice [Du et al., 2019b,a, Allen-Zhu et al., 2019, Oymak and Soltanolkotabi, 2020, Malach et al., 2020].

Theorem 3. Given a DNN defined by Eq. (3) trained by SGD for minimizing empirical denoising score matching loss Eq. (8) . Suppose Assumption 1 and 3 are satisfied. For any ε ∈ (0 , 1) and δ ∈ (0 , 1) . Then with probability at least 1 -2 δ -2 L exp( -Ω( m )) over the randomness of initialization W and noise ϵ in denoising score matching, it holds:

where the N c ( 1 n , S ) is the covering number of the function space S for deep ReLU neural network.

Remark:

1): Theorem 3 and Theorem 1 study similar problems between causal discovery and score-based generative modeling and share similar techniques drawn from statistical learning theory and deep learning theory. These two domains are connected by a common theoretical foundation centered on the upper bound of score matching.

2): Our result extends the results for score matching in diffusion models presented in Chen et al. [2023a] which rested on the assumption of low-dimensional data structures, employing this to decompose the score function and engineer specialized network architectures for the derivation of the upper bound. Our work takes a distinct route. Our conclusions are based on the general deep ReLU neural network instead of a specific encoder-decoder network and do not rely on the assumptions of low-dimensional data used in Chen et al. [2023a]. We harness the inherent traits and conventional techniques of standard deep ReLU networks to directly deduce the upper error bound. This broader scope allows for a more comprehensive understanding of the implications and applicability of score-based generative modeling in a wider range of scenarios.

3): Similar to Theorem 1, by choosing ε 2 = 1 √ n , we can obtain the best bound

demonstrates that the ℓ 2 estimation error converges at a rate of 1 √ n when the sample size n is significantly larger than the dimensionality d and time steps T .

4): The bound is also related to the data dimension d , the variance of the noise in denoising score matching σ , the covering number of the function space N c ( 1 n , S ) , and the upper bound of the data C d . If these quantities increase, it is expected that the error of explicit score matching will also increase. This is due to the increased difficulty in accurately estimating the score function.

5): When t 0 = 0 , the theorem lacks meaning. However, when T ≫ t 0 ≂ 1 , the bound simplifies to d + C 2 d √ n log N c ( 1 n , S ) δ . This indicates that when T is sufficiently large, the loss estimated by the score function in the diffusion model becomes independent of time steps T .

6): Similar to Theorem 1, the result of Theorem 3 is also broadly applicable and can be seamlessly extended to encompass larger batch GD.

5 Numerical evidence

We conducted a series of experiments to validate the theoretical findings presented in the paper. We took inspiration from the code provided inRolland et al. [2022] and employed the structural Hamming distance (SHD) between the generated output and the actual causal graph to assess the outcomes. The ensuing experimental outcomes for SHD, vary across causal model sizes d , sample sizes n , and C m . The experimental results are shown in Tables 1 to 3

Analyzing the experimental outcomes, we find a notable pattern: higher values of C m , augmented sample sizes n , and reduced model size d all contribute to the performance of Algorithm 1 which is consistent with the insights from Theorem 2.

6 Related Work

Score matching: Score Matching was initially introduced by Hyvärinen [2005] and extended to energy-based models by Song and Ermon [2019]. Subsequently, Vincent [2011] proposed denoising score matching, which transforms the estimation of the score function for the original distribution into an estimation for the noise distribution, effectively avoiding the need for second derivative computations. Other methods, such as sliced score matching [Song et al., 2020], denoising likelihood score matching [Chao et al., 2022], and kernel-based estimators, have also been proposed for score matching. The relationship between score matching and Fisher information [Shao et al., 2019], as well as Langevin dynamics [Hyvarinen, 2007], has been explored. On the theoretical side, Wenliang and Kanagawa [2020] introduced the concept of "blindness" in score matching, while Koehler et al. [2023] compared the efficiency of maximum likelihood and score matching, although their results primarily focus on exponential family distributions. Our paper, for the first time, analyzes the sample complexity bounds of the score function estimating in causal inference.

Causal discovery: The application of score methods for causal inference for linear additive models began with Ghoshal and Honorio [2018], which proposed a causal structure recovery method based on topological ordering from the precision matrix (equivalent to the score in that setting). Under certain noise variance assumptions, their method can reliably recover the DAG in polynomial time and sample complexity.

In recent years, there have been numerous algorithms developed for causal inference in non-linear additive models. GraNDAG [Lachapelle et al., 2021] aims to maximize the likelihood of the observed data under this model and enforces a continuous constraint to ensure the acyclicity of the causal graph Rolland et al. [2022] proposed a novel approach for causal inference which utilize score matching algorithms as a foundation for topological ordering and then employ sparse regression techniques to prune the DAG. Subsequently, Montagna et al. [2023a] extended the method to non-Gaussian noise, Sanchez et al. [2023] proposed to use diffusion models to fit the score function, and Montagna et al. [2023b] proposed a new scalable score-based preliminary neighbor search techniques.

Although advances have been achieved in leveraging machine learning for causal discovery, there is generally a lack of further research on error bounds. Other studies concentrate on broader nonparametric models but depend on various assumptions like faithfulness, restricted faithfulness, or the sparsest Markov representation [Spirtes et al., 2000, Raskutti and Uhler, 2018, Solus et al., 2021]. These approaches employ conditional independence tests and construct a graph that aligns with the identified conditional independence relations [Zhang, 2008].

Theoretical analysis of score-based generative modeling: Existing work mainly focuses on two fundamental questions: "How do diffusion models utilize the learned score functions to estimate the data distribution?" [Chen et al., 2023b, De Bortoli et al., 2021, De Bortoli, 2022, Lee et al., 2022,

2023] and "Can neural networks effectively approximate and learn score functions? What are the convergence rate and bounds on the sample complexity?" [Chen et al., 2023a].

Specifically, De Bortoli et al. [2021] and Lee et al. [2022] studied the convergence guarantees of diffusion models under the assumptions that the score estimator is accurate under the ℓ 1 and ℓ 2 norms. Concurrently Chen et al. [2023b] and Lee et al. [2023] extended previous results to distributions with bounded moments. De Bortoli [2022] studied the distribution estimation guarantees of diffusion models for low-dimensional manifold data under the assumption that the score estimator is accurate under the ℓ 1 or ℓ 2 norms.

However, these theoretical results rely on the assumption that the score function is accurately estimated, while the estimation of the score function is largely untouched due to the non-convex training dynamics. Recently, Chen et al. [2023a] provided the first sample complexity bounds for score function estimation in diffusion models. However, their result is based on the assumption that the data distribution is supported on a low-dimensional linear subspace and they use a specialized Encoder-Decoder network instead of a general deep neural network. As a result, a complete theoretical picture of score-based generative modeling is still lacking.

7 Conclusion and Limitations

In this work, we investigate the sample complexity error bounds of Score Matching using deep ReLU neural networks under two different problem settings: causal discovery and score-based generative modeling. We provide a sample complexity analysis for the estimation of the score function in the context of causal discovery for nonlinear additive Gaussian noise models, with a convergence rate of log n n . Furthermore, we extend the sample complexity bounds for the estimation of the score function in the ScoreSDE method to general data and achieve a convergence rate of 1 n . Additionally, we provide an upper bound on the error rate of the state-of-the-art causal discovery method SCORE [Rolland et al., 2022], showing that the error rate of this algorithm converges linearly with respect to the number of training data.

A core limitation of this work is limiting our results to the Gaussian noise assumption. In fact, non-linear mechanisms with additive non-gaussian noise are also identifiable under mild additional assumptions [Peters et al., 2014] and Montagna et al. [2023a] already extended the score-matching approach of Rolland et al. [2022] to that setting. Relaxing this assumption would also allow us to apply our bounds to interesting corner cases, such as linear non-gaussian [Ghoshal and Honorio, 2018], and non-gaussian deterministic causal relations [Daniušis et al., 2010, Janzing et al., 2015]. It may be possible for this assumption to be relaxed in future work, but we argue that the added challenge, the significant difference in algorithms, and the standalone importance of the non-linear Gaussian case justify our focus.

In addition, we make other assumptions that limit the general applicability of our bounds. In particular, the assumption of the Lipschitz property for the score function imposes a strong constraint on the model space. Further investigating the relationship between the noise, the properties of the nonlinear functions in the causal model Eq. (4), and the resulting Lipschitz continuity of the score function would be an interesting extension of this work.

References

Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning (ICML) , 2019.

Block, Y. Mroueh, and A. Rakhlin. Generative modeling with denoising auto-encoders and langevin sampling, 2020.

C.-H. Chao, W.-F. Sun, B.-W. Cheng, Y.-C. Lo, C.-C. Chang, Y.-L. Liu, Y.-L. Chang, C.-P. Chen, and C.-Y. Lee. Denoising likelihood score matching for conditional score-based data generation. In International Conference on Learning Representations (ICLR) , 2022.

Chen, W. Liao, H. Zha, and T. Zhao. Distribution approximation and statistical estimation guarantees of generative adversarial networks, 2020.

Chen, K. Huang, T. Zhao, and M. Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning (ICML) , 2023a.

Chen, S. Chewi, J. Li, Y. Li, A. Salim, and A. Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In International Conference on Learning Representations (ICLR) , 2023b.

M. Chickering. Learning bayesian networks is np-complete. Learning from data: Artificial intelligence and statistics V , 1996.

Daniušis, D. Janzing, J. Mooij, J. Zscheischler, B. Steudel, K. Zhang, and B. Schölkopf. Inferring deterministic causal relations. In Uncertainty in Artificial Intelligence , 2010.

De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis, 2022.

De Bortoli, J. Thornton, J. Heng, and A. Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. In Advances in neural information processing systems (NeurIPS) , 2021.

Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. In Advances in neural information processing systems (NeurIPS) , 2021.

Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning (ICML) , 2019a.

S. Du, X. Zhai, B. Poczos, and A. Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations (ICLR) , 2019b.

Ghosh. Exponential tail bounds for chisquared random variables. Journal of Statistical Theory and Practice , 2021.

Ghoshal and J. Honorio. Learning linear structural equation models in polynomial time and sample complexity. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2018.

Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 2005.

Hyvarinen. Connections between score matching, contrastive divergence, and pseudolikelihood for continuous-valued variables. IEEE Transactions on neural networks , 2007.

Janzing, B. Steudel, N. Shajarisales, and B. Schölkopf. Justifying information-geometric causal inference. Measures of Complexity: Festschrift for Alexey Chervonenkis , 2015.

Jentzen and T. Kröger. Convergence rates for gradient descent in the training of overparameterized artificial neural networks with biases, 2021.

Koehler, A. Heckett, and A. Risteski. Statistical efficiency of score matching: The view from isoperimetry. In International Conference on Learning Representations (ICLR) , 2023.

Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations (ICLR) , 2021.

Lachapelle, P. Brouillard, T. Deleu, and S. Lacoste-Julien. Gradient-based neural dag learning. In International Conference on Learning Representations (ICLR) , 2021.

Lee, J. Lu, and Y. Tan. Convergence for score-based generative modeling with polynomial complexity. In Advances in neural information processing systems (NeurIPS) , 2022.

Lee, J. Lu, and Y. Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory , 2023.

Malach, G. Yehudai, S. Shalev-Schwartz, and O. Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning (ICML) , 2020.

Montagna, N. Noceti, L. Rosasco, K. Zhang, and F. Locatello. Causal discovery with score matching on additive models with arbitrary noise. In CLeaR , 2023a.

Montagna, N. Noceti, L. Rosasco, K. Zhang, and F. Locatello. Scalable causal discovery with score matching. In CLeaR , 2023b.

Nguyen, M. Mondelli, and G. F. Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference on Machine Learning (ICML) , 2021.

Oymak and M. Soltanolkotabi. Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory , 2020.

Peters, J. M. Mooij, D. Janzing, and B. Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research , 2014.

Raskutti and C. Uhler. Learning directed acyclic graph models based on sparsest permutations. Stat , 2018.

Rolland, V. Cevher, M. Kleindessner, C. Russell, D. Janzing, B. Schölkopf, and F. Locatello. Score matching enables causal discovery of nonlinear additive noise models. In International Conference on Machine Learning (ICML) , 2022.

Sachs, O. Perez, D. Pe'er, D. A. Lauffenburger, and G. P. Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science , 2005.

Sanchez, J. P. Voisey, T. Xia, H. I. Watson, A. Q. O'Neil, and S. A. Tsaftaris. Causal machine learning for healthcare and precision medicine. Royal Society Open Science , 2022.

Sanchez, X. Liu, A. Q. O'Neil, and S. A. Tsaftaris. Diffusion models for causal discovery via topological ordering. In International Conference on Learning Representations (ICLR) , 2023.

Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Shao, P. E. Jacob, J. Ding, and V. Tarokh. Bayesian model comparison with the hyvärinen score: Computation and consistency. Journal of the American Statistical Association , 2019.

Solus, Y. Wang, and C. Uhler. Consistency guarantees for greedy permutation-based causal inference algorithms. Biometrika , 2021.

Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in neural information processing systems (NeurIPS) , 2019.

Song, S. Garg, J. Shi, and S. Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence , 2020.

Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR) , 2021.

Spirtes, C. N. Glymour, and R. Scheines. Causation, prediction, and search . MIT press, 2000.

Teyssier and D. Koller. Ordering-based search: A simple and effective algorithm for learning bayesian networks, 2012.

R. Varian. Causal inference in economics and marketing. Proceedings of the National Academy of Sciences , 2016.

Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Taylor & Francis, 2018.

Vincent. A connection between score matching and denoising autoencoders. Neural computation , 2011.

Wang, Y. Du, S. Zhu, L. Ke, Z. Chen, J. Hao, and J. Wang. Ordering-based causal discovery with reinforcement learning. In International Joint Conferences on Artificial Intelligence (IJCAI) , 2021.

K. Wenliang and H. Kanagawa. Blindness of score-based methods to isolated components and mixing proportions, 2020.

Zhang. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence , 2008.

Zhu, F. Liu, G. Chrysos, and V. Cevher. Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization). In Advances in neural information processing systems (NeurIPS) , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focus on theoretical analysis on two main score-based approaches. The first one is the score-based casual discovery, where the authors give the sample complexity error bounds for score matching using ReLU NNs as well as an upper bound of the error rate. The second one is the score-based generative modeling, where the authors give a sample complexity bound for the score estimation.\n\nSTRENGTHS:\nThis work has strong theoretical results and has provided sufficient backgrounds for the readers to understand the theoretical problems and include a comprehensive discussion on related work.\n\nWEAKNESSES:\n- It is unclear why the analysis on score-based casual discovery and that on score-based generative modeling are put together in this work; it seems to me they are unrelated work. The authors might want to put some insights on the relationship between these two work, for example, some shared techniques for proving the sample complexity bounds.\n- The definition of covering number should be included to make this work more self-contained.\n- For Assumption 2, since this is an original assumption proposed by this work, the authors should give more justification to how much this assumption holds in practice. An specific example, even a simple one, showing when this assumption holds, would make this work more convincing.\n- To provide some proof sketch might make this work more convincing and help reader better understand what are the technical contributions of this work. Now it is unclear to me why it provides better analysis than the previous work; specifically, why it does not rely on the assumption of low-dimensional data while the previous work Chen et al. [2023a] does.\n\nQUESTIONS:\n- It seems that the theoretical results of score-based casual discovery are highly specific to the algorithm in Rolland et al. [2022]. I wonder how these results generalize to other related approaches?\n- Does Lemma 1 play a role in the later analysis? I wonder why we need Lemma 1.\n- The assumption in Line 148-149 is not properly justified.\n- Can the authors provide a specific case on when Assumption 2 holds and analysis on how practical this assumption?\n- As mentioned above, why this work does not rely on the assumption of low-dimensional data while the previous work Chen et al. [2023a] does?\n\nMinor issue:\n- It seems that the case l = 1 in Equation (3) can be merged into the case 2<=l<=L-1.\n- Line 107: donate -> denotes\n- The property mentioned in Line 108 seems unrelated to the Definition 1.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides sample complexity bounds on score function estimation when (1) the score function is estimated by using SGD to minimize a denoising score matching objective, (2) the probability distribution is induced by a structural causal model (SCM) with additive Gaussian noise, and (3) the score function is Lipschitz, and (4) for each mechanism $f_i$ in the SCM, the expected value of the second derivative of $f_i$ with respect to each parent of $X_i$ is lower-bounded by $C_m$ times $\\sigma_i$, the variance of the exogenous noise of $X_i$.\n\nThey show that, under these conditions, the estimated score function converges at a parametric rate (Theorem 1). Using this bound, they give a sample complexity bound for the event that the SCORE algorithm for causal order search returns a correct topological order (Theorem 2). Finally, they also use the bound from Theorem 1 to provide a sample complexity result for the score-matching objective used in those works.\n\nSTRENGTHS:\n### Significance\nThe paper appears to be a significant theoretical contribution addressing the sample complexity of score function estimation, which as an important topic for both causal structure learning and score-based generative modeling.\n\n### Clarity\nThe paper is written in a fairly clear style. Relevant notations are appropriately defined and summarized in the Appendix. For each theorem, there are accompanying remarks which clearly discuss the implications of the theorem. The motivation and the context for the work are easy to understand.\n\nWEAKNESSES:\n### Experimental Results\nThis is clearly a theory paper, and contributes enough theory such that there is no need for an extended section on experiments. In fact, given the space constraints, extensive experiments would most likely hurt the paper. However, a small set of experiments - e.g. corroborating Theorem 1 by plotting the loss function versus the number of samples - would take the paper from a good contribution to a great one (8 instead of 7). In my experience, experimental validation of theoretical results on sample complexity can be fairly difficult to obtain, and thus papers like this one would be a lot more useful in practice if they ran some carefully-chosen experiments.\n\nQUESTIONS:\n### Details\n1. The theorems (eg. Theorem 1 and Theorem 3) refer to a DNN trained by SGD. However, this is missing a condition on the learning rate. What is the condition on the learning rate?\n2. Presumably the usage of \"SGD\" in the theorems here implies that batch size = 1. Do the results extend to larger batch sizes? For clarity, I think it would be helpful to encapsulate the assumptions on the architecture and training into an Assumption environment.\n3. What is $\\tau$, the first argument to the covering number in line 203? Where is it defined?\n\n### Suggestions\n1. Since this paper sits at the intersection of score-based generative modeling (SGM) and causal structure learning (CSL), it is important to keep the background of both audiences in mind. In particular, coming from causal structure learning / statistics, I am surprised by Theorem 1, which seems to require that SGD finds a solution close to the global optimum. It looks like Theorem 1 uses some relatively recent results / techniques (e.g. Nguyen et al., 2021), which might be well-known in the SGM community but are not well-known in the CSL community. This paper provides a good opportunity to bring the advances of SGM to the CSL audience, and it would be valuable to provide more intuition about the techniques used here.\n2. Please reference all appendices in the paper. Appendices B, C, F, and G are not referenced in the paper.\n3. I think Lemma 2 would be more clear if you left out the node $i$ altogether. Since the associated term is zero anyways, it just gets in the way and serves no purpose in the left-hand side of the implication statement. It feels like the point that the variance term associated to $i$ is zero shouldn't be part of this theorem, but a separate observation.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study aims to investigate the sample complexity associated with score-matching and its applications in the field of causal discovery, providing valuable theoretical bounds. The authors present theoretical evidence that training a conventional deep ReLU neural network using stochastic gradient descent enables the accurate estimation of the score function. Additionally, they establish rigorous bounds on the error rate pertaining to the recovery of causal relationships, employing the score-matching-based methodology introduced by Rolland et al. [2022], under the assumption of an adequately precise estimation of the score function. Furthermore, an analysis is performed to determine the upper bound of score-matching estimation within the framework of score-based generative modeling, which not only bears relevance to causal discovery but also demonstrates independent significance within the wider domain of generative models.\n\nSTRENGTHS:\nThere are several main strengths in this paper. First, the authors extensively explored the relevant recent theories, including background research. Second, the error bounds are rigorously derived from well-known assumptions (1 and 3) and novel assumption (2) for causal discovery. Third, the use of this theory seems to have potential in multiple applications relating to causal discovery and SGM given that the architecture is general enough.\n\nWEAKNESSES:\nThere are some weaknesses in this paper. First, the background section is a bit too long compared to the main body (almost 5 pages with added related work in Section 5) resulting in only 2.5 pages new materials (excluding conclusion). Although it was advantageous to extensively explain the background (where there are multiple topics in this paper), it became a weakness due to its excessive length. \n\nThe second weakness is about deep ReLU neural networks, but there is no empirical experiments to corroborate theory. For example, one might show causal discovery errors and how the bounds in Theorem 2 are ‘good’ enough. If the bounds are too loose, the usefulness of the bounds will become low. (or how robust the result would be under the violation of the some of assumptions)\n\nGiven that there are some space that can be saved by removing (or moving) some of related work/preliminaries from main text, I recommend the authors to have some proof-of-concept experiments demonstrating the effectiveness of such theoretical bounds.\n\nQUESTIONS:\nNot a question but there are issues with \\begin{align} or other tex related comments. If you have an empty line in between text and mathematical formula, there will be a gap. You can see Eq 1 (and the one before and one next), 4, 5, 6, 7, (one next to 7), 8, next to 8, in Assumption 2, in Theorem 1, in Thm 2, in Thm 3. You can effectively save lots of space. \n\nThere are many remarks. But I guess they can be incorporated more smoothly just as a text with proper rephrasing (if needed). \n \n98th line contains a typo with missing parentheses in the equation.\n\nTheorem 3, Remark 1. Can you elaborate in the main text what enable your theorem applicable to high-dimensional data versus. Chen 2023a's results on low dimensional data?\n\nLIMITATIONS:\nAssumptions can be restrictive but the authors are well-understanding such restrictions and others also made similar assumptions. For the assumption 2, it is unclear how one can assume C_m properly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents error bounds (sample complexity and convergence rates) for the problems of score based generative modelling and score-order based causal discovery. It is primarily a theory paper, and one of the first few works to present error bounds for a causal discovery algorithm that is not conditional independence based.\n\nSTRENGTHS:\n(+) In score-function based causal discovery (which does not use conditional independence testing), there are no notable error bounds. So this work addresses the gap. Although it is tied to the order based method of Rolland et al, it is interesting as this might lead to more such work for other methods in causal discovery. Theoretical guarantees are important for causal discovery as real world validation is harder due to lack of ground truth.\n(+) The assumptions and technicalities are fairly well laid out. Although the score requires Lipschitz assumptions, thereby limiting the class of mechanisms, it is still relevant as it still might cover fairly large class of real world settings.\n(+) The theoretical analysis for score based generative modelling seems similar to what has been done for causal discovery, and is potentially interesting. However, I am unsure about its relevance wrt that field as I am not totally familiar with it.\n\nWEAKNESSES:\n(-) My main concern with this work is that it might be a bit too broad in scope. It is definitely okay and also welcome to have a broader scope (causal discovery plus score based generative modelling), but I don't think the current writing/presentation justifies it appropriately. I think a better way to present the results would have been to present the main result a bit more generally and clearly, and explain that result in the context of both causal discovery and score based generative modelling. I would have been happy to see just the causal discovery part alone as it merits quite a bit of discussion (since it is the first work doing so).\n(-) There are no experiments. Although it is primarily a theory paper, some simple empirical analysis of RELU deep networks for score function in causal discovery would have been very illustrative and relevant. It would have been good to see how the bounds when the assumptions are satisfied and how do they get worse if the assumptions are not satisfied.\n\nQUESTIONS:\nMy main question is : Is there a common theoretical part (proof techniques and statements) that can be generalised to both settings? If so, I think it might be good to highlight it. If not, I feel the paper might benefit from studying these two problems separately.\n\nLIMITATIONS:\nThe assumptions are fairly well laid out. A major limitation now is that there is no empirical analysis.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides detailed theoretical results for causal discovery and score-based generative modeling. For causal discovery, it first provides a sample complexity analysis for the estimation of the score function for nonlinear additive Gaussian noise models, then it proves that the error rate of a score-matching-based method, SCORE, converges linearly w.r.t the number of training data. For score-based generative modeling, it presents sample complexity bounds for the estimation of the score function in the ScoreSDE.\n\nSTRENGTHS:\n1. This paper provides sound and detailed theoretical results for score matching in the context of both causal discovery and score-based generative modeling. In both cases, the theoretical analyses hold with mild assumptions compared to previous work.\n\n2. This paper well organized overall. It provides a detailed discussion about preliminaries in Section 2, provides theoretical results for causal discovery and score-based generative modeling in Section 3 and Section 4 respectively. Although there are many symbols, I think this paper is relatively easy to follow.\n\n3. This paper points out limitations of the current theoretical results clearly in Section 6.\n\nWEAKNESSES:\n1. As demonstrated in Section 6 of this paper, the theoretical results are still not general enough due to some relatively strong assumptions.\n\n2. I suggest the authors change the title \"Error Bounds for Score Matching Causal Discovery\" because almost half of this paper is about score matching in score-based generative modeling, which has little to do with causal discovery.\n\n3. Although I have no doubt about the soundness of the provided theoretical results, the contribution of this paper is somewhat limited. This paper does not design more advanced algorithm, it only provides sample complexity bounds for existing algorithms under PAC framework. I'm not sure whether it should be regarded as a supplement for previous work or an independent research. However, considering that ScoreSDE and is widely used in respective field, I still recommond \"weak accept\" for this paper.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}