SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenyu Zhu†, Francesco Locatello‡*, Volkan Cevher†*
† École Polytechnique Fédérale de Lausanne ‡ Institute of Science and Technology Austria
†{zhenyu.zhu, volkan.cevher}@epfl.ch ‡francesco.locatello@ista.ac.at
This paper provides statistical sample complexity bounds for score-matching and its applications in causal discovery. We demonstrate that accurate estimation of the score function is achievable by training a standard deep ReLU neural network using stochastic gradient descent. We establish bounds on the error rate of recovering causal relationships using the score-matching-based causal discovery method of Rolland et al. [2022], assuming a sufficiently good estimation of the score function. Finally, we analyze the upper bound of score-matching estimation within the score-based generative modeling, which has been applied for causal discovery but is also of independent interest within the domain of generative models.
Score matching Hyvarinen [2005], an alternative to the maximum likelihood principle for unnormalized probability density models with intractable partition functions, has recently emerged as a new state-of-the-art approach that leverages machine learning for scalable and accurate causal discovery from observational data Rolland et al. [2022]. However, the theoretical analysis and guarantees in the finite sample regime are underexplored for causal discovery even beyond score-matching approaches.
Contributions: In this work, we give the first sample complexity error bounds for score-matching using deep ReLU neural networks. With this, we obtain the first upper bound on the error rate of the method proposed by Rolland et al. [2022] to learn the topological ordering of a causal model from observational data. Thanks to the wide applicability of score-matching in machine learning, we also discuss applications to the setting of score-based generative modeling. Our main contributions are:
1. We provide the analysis of sample complexity bound for the problem of score function estimation in causal discovery for non-linear additive Gaussian noise models which has a convergence rate of  $\log n / n$  with respect to the number of data. Importantly, our results require only mild additional assumptions, namely that the non-linear relationships among the causal variables are bounded and that the score function is Lipschitz. To the best of our knowledge, this is the first work to provide sampling complexity bounds for this problem.
2. We provide the first analysis of the state-of-the-art topological ordering-based causal discovery method SCORE [Rolland et al., 2022] and provide a correctness guarantee for the obtained topological order. Our results demonstrate that the algorithm's error rate converges linearly with respect to the number of training data. Additionally, we establish a connection between the algorithm's error rate and the average second derivative (curvature) of the non-linear relationships among the causal variables, discussing the impact of the causal model's inherent characteristics on the algorithm's error rate in identification.
*Share the senior authorship
3. We present sample complexity bounds for the score function estimation problem in the standard score-based generative modeling method, ScoreSDE [Song et al., 2021]. In contrast to previous results [Chen et al., 2023a], our bounds do not rely on the assumption of low-dimensional input data, and we extend the applicability of the model from a specific encoder-decoder network architecture to a general deep ReLU neural network.
High-level motivation and background: Causal discovery and causal inference refer to the process of inferring causation from data and reasoning about the effect of interventions. They are highly relevant in fields such as economics [Varian, 2016], biology [Sachs et al., 2005], and healthcare [Sanchez et al., 2022]. In particular, some causal discovery methods aim to recover the causal structure of a problem solely based on observational data.
The causal structure is typically represented as a directed acyclic graph (DAG), where each node is associated with a random variable, and each edge represents a causal mechanism between two variables. Learning such a model from data is known to be NP-hard [Chickering, 1996]. Traditional approaches involve testing for conditional independence between variables or optimizing goodness-of-fit measures to search the space of possible DAGs. However, these greedy combinatorial optimization methods are computationally expensive and difficult to extend to high-dimensional settings.
An alternative approach is to reframe the combinatorial search problem as a topological ordering task [Teyssier and Koller, 2012, Solus et al., 2021, Wang et al., 2021, Rolland et al., 2022, Montagna et al., 2023b,a, Sanchez et al., 2023], where nodes are ordered from leaf to root. This can significantly speed up the search process in the DAG space. Once a topological ordering is found, a feature selection algorithm can be used to prune potential causal relations between variables, resulting in a DAG.
Recently, Rolland et al. [2022] proposed the SCORE algorithm, which utilizes the Jacobian of the score function to perform topological ordering. By identifying which elements of the Jacobian matrix of the score function remain constant across all data points, leaf nodes can be iteratively identified and removed. This approach provides a systematic way to obtain the topological ordering and infer the causal relations within the entire model. This method has achieved state-of-the-art results on multiple tasks Rolland et al. [2022] and has been extended to improve scalability Montagna et al. [2023b] also using diffusion models Sanchez et al. [2023] and to non-Gaussian noise Montagna et al. [2023a]. Interestingly, these approaches separate the concerns of statistical estimation of the score function from the causal assumption used to infer the graph (e.g., non-linear mechanisms and additive Gaussian noise). This opens an opportunity to study the convergence properties of these algorithms in the finite data regime, which is generally under-explored in the causal discovery literature. In fact, if we had error bounds on the score estimate without additional complications from causal considerations, we could study their downstream effect when the score is used for causal discovery.
Unfortunately, this is far from trivial as the theoretical research on score matching lags behind its empirical success and progress would have far-reaching implications. Even beyond causal discovery, error bounds on the estimation of the score functions would be useful for score-based generative modeling (SGM) [Song and Ermon, 2019, Song et al., 2021]. These have achieved state-of-the-art performance in various tasks, including image generation [Dhariwal and Nichol, 2021] and audio synthesis [Kong et al., 2021]. There has been significant research investigating whether accurate score estimation implies that score-based generative modeling provably converges to the true data distribution in realistic settings [Chen et al., 2023b, Lee et al., 2022, 2023]. However, the error bound of score function estimation in the context of score-based generative modeling remains an unresolved issue due to the non-convex training dynamics of neural network optimization.
Notations: We use the shorthand  $[n] := \{1, 2, \ldots, n\}$  for a positive integer  $n$ . We denote by  $a(n) \lesssim b(n)$ : there exists a positive constant  $c$  independent of  $n$  such that  $a(n) \leqslant cb(n)$ . The Gaussian distribution is  $\mathcal{N}(\mu, \sigma^2)$  with the  $\mu$  mean and the  $\sigma^2$  variance. We follow the standard Bachmann-Landau notation in complexity theory e.g.,  $\mathcal{O}$ ,  $o$ ,  $\Omega$ , and  $\Theta$  for order notation. Due to space constraints, a detailed notation is deferred to Appendix A.
As this paper concerns topics in score matching estimation, diffusion models, neural network theory, and causal discovery, we first introduce the background and problem setting of our work.
2.1 Score matching
For a probability density function  $p(\pmb{x})$ , we call the score function the gradient of the log density with respect to the data  $\pmb{x}$ . To estimate the score function  $\nabla \log p(\pmb{x})$ , we can minimize the  $\ell_2$  loss over the function space  $S$ .
$$
\min  _ {\boldsymbol {s} \in \mathcal {S}} \mathbb {E} _ {p} [ \| \boldsymbol {s} (\boldsymbol {x}) - \nabla \log p (\boldsymbol {x}) \| ^ {2} ], \quad \hat {\boldsymbol {s}} = \arg \min  _ {\boldsymbol {s} \in \mathcal {S}} \mathbb {E} _ {p} [ \| \boldsymbol {s} (\boldsymbol {x}) - \nabla \log p (\boldsymbol {x}) \| ^ {2} ].
$$
The corresponding objective function to be minimized is the expected squared error between the true score function and the neural network:
$$
J _ {\mathrm {E S M}} (\boldsymbol {s}, p (\boldsymbol {x})) = \mathbb {E} _ {p (\boldsymbol {x})} \left[ \frac {1}{2} \left\| \boldsymbol {s} (\boldsymbol {x}) - \frac {\partial \log p (\boldsymbol {x})}{\partial \boldsymbol {x}} \right\| ^ {2} \right], \tag {1}
$$
We refer to this formulation as explicit score matching (ESM).
Denoising score matching (DSM) is proposed by Vincent [2011] to convert the inference of the score function in ESM into the inference of the random noise and avoid the computing of the second derivative. For the sampled data  $\pmb{x}$ ,  $\hat{\pmb{x}}$  is obtained by adding unit Gaussian noise to  $\pmb{x}$ . i.e.  $\hat{\pmb{x}} = \pmb{x} + \epsilon$ ,  $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$ . We can derive the conditional probability distribution and its score function:
$$
p (\hat {\boldsymbol {x}} | \boldsymbol {x}) = \frac {1}{(2 \pi) ^ {d / 2} \sigma^ {d}} \exp (- \frac {\| \boldsymbol {x} - \hat {\boldsymbol {x}} \| ^ {2}}{2 \sigma^ {2}}), \qquad \frac {\partial \log p (\hat {\boldsymbol {x}} | \boldsymbol {x})}{\partial \hat {\boldsymbol {x}}} = \frac {\boldsymbol {x} - \hat {\boldsymbol {x}}}{\sigma^ {2}}.
$$
Then the DSM is defined by:
$$
J _ {\mathrm {D S M}} (\boldsymbol {s}, p (\boldsymbol {x}, \hat {\boldsymbol {x}})) = \mathbb {E} _ {p (\boldsymbol {x}, \hat {\boldsymbol {x}})} \left[ \frac {1}{2} \left\| \boldsymbol {s} \left(\hat {\boldsymbol {x}} - \frac {\partial \log p (\hat {\boldsymbol {x}} | \boldsymbol {x})}{\partial \hat {\boldsymbol {x}}}\right) \right\| ^ {2} \right] = \mathbb {E} _ {p (\boldsymbol {x}, \hat {\boldsymbol {x}})} \left[ \frac {1}{2} \left\| \boldsymbol {s} (\hat {\boldsymbol {x}}) - \frac {\boldsymbol {x} - \hat {\boldsymbol {x}}}{\sigma^ {2}} \right\| ^ {2} \right]. \tag {2}
$$
Vincent [2011] have proven that minimizing DSM is equivalent to minimizing ESM and does not depend on the particular form of  $p(\hat{x} | \boldsymbol{x})$  or  $p(\boldsymbol{x})$ .
2.2 Neural network and function space
In this work, we consider a standard depth- $L$  width- $m$  fully connected ReLU neural network. Formally, we define a DNN with the output  $s_l(\pmb{x})$  in each layer
$$
\boldsymbol {s} _ {l} (\boldsymbol {x}) = \left\{ \begin{array}{c c} \boldsymbol {x} & l = 0, \\ \phi \left(\left\langle \boldsymbol {W} _ {l}, \boldsymbol {s} _ {l - 1} (\boldsymbol {x}) \right\rangle\right) & 1 \leq l \leq L - 1, \\ \left\langle \boldsymbol {W} _ {L}, \boldsymbol {s} _ {L - 1} (\boldsymbol {x}) \right\rangle & l = L, \end{array} \right. \tag {3}
$$
where the input is  $\pmb{x} \in \mathbb{R}^d$ , the output is  $\pmb{s}_L(\pmb{x}) \in \mathbb{R}^d$ , the weights of the neural networks are  $\pmb{W}_1 \in \mathbb{R}^{m \times d}$ ,  $\pmb{W}_l \in \mathbb{R}^{m \times m}$ ,  $l = 2, \dots, L-1$  and  $\pmb{W}_L \in \mathbb{R}^{d \times m}$ . The neural network parameters formulate the tuple of weight matrices  $\pmb{W} := \{\pmb{W}_i\}_{i=1}^L \in \{\mathbb{R}^{m \times d} \times (\mathbb{R}^{m \times m})^{L-2} \times \mathbb{R}^{d \times m}\}$ . The  $\mathcal{S}$  denotes the function space of Eq. (3).
The  $\phi = \max (0,x)$  is the ReLU activation function. According to the property  $\phi (x) = x\phi^{\prime}(x)$  of ReLU, we have  $s_l = D_lW_l s_{l - 1}$ , where  $D_{l}$  is a diagonal matrix defined as below.
Definition 1 (Diagonal sign matrix). For  $l \in [L - 1]$  and  $k \in [m]$ , the diagonal sign matrix  $D_{l}$  is defined as:  $(D_{l})_{k,k} = 1\{(\boldsymbol{W}_{l}\boldsymbol{s}_{l - 1})_{k} \geq 0\}$ .
Initialization: We make the standard random Gaussian initialization  $[W_{l}]_{i,j} \sim \mathcal{N}(0, \frac{2}{m})$  for  $l \in [L - 1]$  and  $[W_{L}]_{i,j} \sim \mathcal{N}(0, \frac{1}{d})$ .
2.3 Causal discovery
In this paper, we follow the setting in Rolland et al. [2022] and consider the following causal model, a random variable  $\pmb{x} \in \mathbb{R}^d$  is generated by:
$$
x ^ {(i)} = f _ {i} \left(\operatorname {P A} _ {i} (\boldsymbol {x})\right) + \epsilon_ {i}, \quad i \in [ d ], \tag {4}
$$
where  $f_{i}$  is a non-linear function,  $\epsilon_i\sim \mathcal{N}(0,\sigma_i^2)$  and  $\mathrm{PA}_i(\pmb {x})$  represent the set of parents of  $x^{(i)}$  in  $\pmb{x}$ . Then we can write the probability distribution function of  $\pmb{x}$  as:
$$
p (\boldsymbol {x}) = \prod_ {i = 1} ^ {d} p \left(x ^ {(i)} \mid \mathrm {P A} _ {i} (\boldsymbol {x})\right). \tag {5}
$$
For such non-linear additive Gaussian noise models Eq. (4), Rolland et al. [2022] provides Algorithm 1 to learn the topological order by score matching as follows:
Algorithm 1 SCORE matching causal order search (Adapted from Algorithm 1 in Rolland et al. [2022])
Input: training data  $\{(x_{(i)})_{i = 1}^{N}\}$    
Initialize:  $\pi = []$  nodes  $= \{1,\dots ,d\}$    
for  $k = 1,\ldots ,d$  do Estimate the score function  $s_{\mathrm{nodes}} = \nabla \log p_{\mathrm{nodes}}$  by deep ReLU network with SGD. Estimate  $V_{j} = \hat{\mathrm{Var}}_{\pmb{x}_{\mathrm{nodes}}}\left[\frac{\partial s_{j}(\pmb{x})}{\partial\pmb{x}^{(j)}}\right]$ $l\gets$  nodes[arg min  $j$ $V_{j}$ ]  $\pi \leftarrow [l,\pi ]$  nodes  $\leftarrow$  nodes-  $\{l\}$  Remove  $l$  -th element of  $\pmb{x}$    
end for   
Get the final DAG by pruning the full DAG associated with the topological order  $\pi$
2.4 Score-based generative modeling (SGM)
In this section, we give a brief overview of SGM following Song et al. [2021], Chen et al. [2023b].
2.4.1 Score-based generative modeling with SDEs
Forward process: The success of previous score-based generative modeling methods relies on perturbing data using multiple noise scales, and the proposal of the diffusion model is to expand upon this concept by incorporating an infinite number of noise scales. This will result in the evolution of perturbed data distributions as the noise intensity increases, which will be modeled through a stochastic differential equation (SDE).
$$
\mathrm {d} \boldsymbol {x} _ {t} = \boldsymbol {f} \left(\boldsymbol {x} _ {t}, t\right) \mathrm {d} t + g _ {t} \mathrm {d} \boldsymbol {w}, \quad \boldsymbol {x} _ {0} \sim p _ {0}. \tag {6}
$$
The expression describes  $\boldsymbol{x}_t$ , where the standard Wiener process (also known as Brownian motion) is denoted as  $\boldsymbol{w}$ , the drift coefficient of  $\boldsymbol{x}_t$  is represented by a vector-valued function called  $\boldsymbol{f}$ , and the diffusion coefficient of  $\boldsymbol{x}_t$  is denoted as  $g_t$ , a scalar function. In this context, we will refer to the probability density of  $\boldsymbol{x}_t$  as  $p_t$ , and the transition kernel from  $\boldsymbol{x}_s$  to  $\boldsymbol{x}_t$  as  $p_{st}(\boldsymbol{x}_t|\boldsymbol{x}_s)$ , where  $0\leq s < t\leq T$ . The Ornstein-Uhlenbeck (OU) process is a Gaussian process that is both time-homogeneous and a Markov process. It is distinct in that its stationary distribution is equivalent to the standard Gaussian distribution  $\gamma^d$  on  $\mathbb{R}^d$ .
Reverse process: We can obtain samples of  $x_0 \sim p_0^{\mathrm{SDE}}$  by reversing the process starting from samples of  $x_T \sim p_T^{\mathrm{SDE}}$ . An important finding is that the reversal of a diffusion process is a diffusion process as well. It operates in reverse time and is described by the reverse-time SDE:
$$
\mathrm {d} \boldsymbol {x} _ {t} = \left(\boldsymbol {f} \left(\boldsymbol {x} _ {t}, t\right) - g _ {t} ^ {2} \nabla_ {\boldsymbol {x}} \log p _ {t} \left(\boldsymbol {x} _ {t}\right)\right) \mathrm {d} t + g _ {t} \mathrm {d} \overline {{\boldsymbol {w}}}. \tag {7}
$$
When time is reversed from  $T$  to 0,  $\overline{w}$  is a standard Wiener process with an infinitesimal negative timestep of  $\mathrm{d}t$ . The reverse diffusion process can be derived from Eq. (7) once the score of each marginal distribution,  $\nabla \log p_t(\pmb{x}_t)$ , is known for all  $t$ . By simulating the reverse diffusion process, we can obtain samples from  $p_0^{\mathrm{SDE}}$ .
Some special settings: In order to simplify the writing of symbols and proofs, in this work we choose that  $\pmb{f}(\pmb{x}_t,t) = -\frac{1}{2}\pmb{x}_t$  and  $g(t) = 1$  which has been widely employed in prior research [Chen et al., 2023a,b, De Bortoli et al., 2021] for theoretical analysis in Ornstein-Uhlenbeck process in score-based generative modeling.
2.4.2 Score matching in diffusion model
We aim to minimize the equivalent objective for score matching:
$$
\min _ {\pmb {s} \in \mathcal {S}} \int_ {0} ^ {T} w (t) \mathbb {E} _ {\pmb {x} _ {0} \sim p _ {0}} \left[ \mathbb {E} _ {\pmb {x} _ {t} \sim p _ {0 t} (\pmb {x} _ {t} | \pmb {x} _ {0})} \left[ \| \nabla_ {\pmb {x} _ {t}} \log p _ {0 t} (\pmb {x} _ {t} | \pmb {x} _ {0}) - \pmb {s} (\pmb {x} _ {t}, t) \| _ {2} ^ {2} \right] \right] \mathrm {d} t.
$$
The transition kernel has an analytical form  $\nabla_{\pmb{x}_t}\log p_{0t}(\pmb{x}_t|\pmb{x}_0) = -\frac{\pmb{x}_t - \alpha(t)\pmb{x}_0}{h(t)}$ , where  $\alpha (t) = e^{-\frac{t}{2}}$  and  $h(t) = 1 - \alpha (t)^2 = 1 - e^{-t}$ .
The empirical score matching loss is:
$$
\min  _ {\boldsymbol {s} \in \mathcal {S}} \hat {\mathcal {L}} (\boldsymbol {s}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(\boldsymbol {x} _ {(i)}; \boldsymbol {s}\right), \tag {8}
$$
where the loss function  $\ell (\pmb{x}_{(i)};\pmb{s})$  is defined as:
$$
\ell (\pmb {x} _ {(i)}; \pmb {s}) = \frac {1}{T - t _ {0}} \int_ {t _ {0}} ^ {T} \mathbb {E} _ {\pmb {x} _ {t} \sim p _ {0 t} (\pmb {x} _ {t} | \pmb {x} _ {0} = \pmb {x} _ {(i)})} \big [ \left\| \nabla_ {\pmb {x} _ {t}} \log p _ {0 t} (\pmb {x} _ {t} | \pmb {x} _ {0} = \pmb {x} _ {(i)}) - \pmb {s} (\pmb {x} _ {t}, t) \right\| _ {2} ^ {2} \big ] \mathrm {d} t.
$$
Here we choose  $w(t) = \frac{1}{T - t_0}$ , and we observe that the expected loss  $\mathcal{L}(\cdot) = \mathbb{E}_{\boldsymbol{x} \sim p_0}[\hat{\mathcal{L}}(\cdot)]$ .
3 Theoretical results for causal discovery
In this section, we state the main theoretical results of this work. We present the assumptions on nonlinear additive Gaussian noise causal models in Section 3.1. Then, we present the sample complexity bound for score matching in causal discovery in Section 3.2. In Section 3.3 we provide the upper bound on the error rate for causal discovery using the Algorithm 1. The full proofs of Theorem 1 and 2 are deferred to Appendix E and F, respectively.
Assumption 1 (Lipschitz property of score function). The score function  $\nabla \log p(\cdot)$  is 1-Lipschitz.
Remark: The Lipschitz property of the score function is a standard assumption commonly used in the existing literature [Block et al., 2020, Lee et al., 2022, Chen et al., 2023b,a]. However, for causal discovery, this assumption limits the family of mechanisms that we can cover.
Assumption 2 (Structural assumptions of causal model). Let  $p$  be the probability density function of a random variable  $x$  defined via a non-linear additive Gaussian noise model Eq. (4). Then,  $\forall i \in [d]$  the non-linear function is bounded,  $|f_i| \leq C_i$ . And  $\forall i, j \in [d]$ , if  $j$  is one of the parents of  $i$ , i.e.  $x^{(j)} \Rightarrow x^{(i)}$ , then there exist a constant  $C_m$  that satisfy:
$$
\mathbb {E} _ {p (\boldsymbol {x})} \left(\frac {\partial^ {2} f _ {i} (P A _ {i} (\boldsymbol {x}))}{\partial x ^ {(j) 2}}\right) \geq C _ {m} \sigma_ {i} ^ {2}.
$$
Remark: This is a novel assumption that we introduce, relating the average second derivative of a mechanism (related to its curvature) to the noise variance of the child variable. This will play a crucial yet intuitive role in our error bound: identifiability is easier when there is sufficient non-linearity of a mechanism with respect to the noise of the child variable. Consider the example of a quadratic mechanism, where the second derivative is the leading constant of the polynomial. If this constant is small (e.g., close to zero), the mechanism is almost linear and we may expect that the causal model should be harder to identify. Similarly, if the child variable has a very large variance, one may expect it to be more difficult to distinguish cause from effect, as the causal effect of the parent is small compared to the noise of the child. According to Assumption 2, we can derive the identified ability margin for leaf nodes and parent nodes.
Lemma 1. If a non-linear additive Gaussian noise model Eq. (4) satisfies Assumption 2. Then,  $\forall i,j\in [d]$ , we have:
$$
i \text {i s a l e a f} \Rightarrow \operatorname {V a r} \left(\frac {\partial s _ {i} (\boldsymbol {x})}{\partial x ^ {(i)}}\right) = 0, j \text {i s n o t a l e a f} \Rightarrow \operatorname {V a r} \left(\frac {\partial s _ {j} (\boldsymbol {x})}{\partial x ^ {(j)}}\right) \geq C _ {m}.
$$
This lemma intuitively relates our identifiability margin with the decision rule of SCORE Rolland et al. [2022] to identify leaves. Non-leaf nodes should have the variance of their score Jacobian sufficiently far from zero. As one may expect, we will see in Theorem 2 that the closer  $C_m$  is to zero, the more likely it is that the result of the algorithm will be incorrect given finite samples.
3.2 Error bound for score matching in causal discovery
We are now ready to state the main result of the score matching in causal discovery. We provide the sample complexity bounds of the explicit score matching Eq. (1) that using denoising score matching Eq. (2) in Algorithm 1 for non-linear additive Gaussian noise models Eq. (4).
Theorem 1. Given a DNN defined by Eq. (3) trained by SGD for minimizing empirical denoising score matching objective. Suppose Assumption 1 and 2 are satisfied. For any  $\varepsilon \in (0,1)$  and  $\delta \in (0,1)$ , if  $\sigma_{i} \approx \sigma$  and  $\frac{C_i}{\sigma_i} \approx 1$ ,  $\forall i \in [d]$ . Then with probability at least  $1 - 2\delta - 4\exp(-\frac{d}{32}) - 2L\exp(-\Omega(m)) - \frac{1}{nd}$  over the randomness of initialization  $\mathbf{W}$ , noise  $\epsilon$  and  $\epsilon_{i}$ , it holds that:
$$
J _ {E S M} (\hat {\boldsymbol {s}}, p (\boldsymbol {x})) \lesssim \frac {\sigma^ {2} d \log n d}{n \varepsilon^ {2}} \log \frac {\mathcal {N} _ {c} (\frac {1}{n} , \mathcal {S})}{\delta} + \frac {1}{n} + d \varepsilon^ {2},
$$
where the  $\mathcal{N}_c\left(\frac{1}{n}, S\right)$  is the covering number of the function space  $S$  for deep ReLU neural network.
Remark: