Title: Breaking the curse of dimensionality in structured density estimation
Venue: NeurIPS 2024
Accepted: yes
Keywords: nonparametric statistics

density estimation

graphical models

sample complexity

curse of dimensionality
Abstract: We consider the problem of estimating a structured multivariate density, subject to Markov conditions implied by an undirected graph. In the worst case, without Markovian assumptions, this problem suffers from the curse of dimensionality. Our main result shows how the curse of dimensionality can be avoided or greatly alleviated under the Markov property, and applies to arbitrary graphs. While existing results along these lines focus on sparsity or manifold assumptions, we introduce a new graphical quantity called ``graph resilience'' and show that it dictates the optimal sample complexity. Surprisingly, although one might expect the sample complexity of this problem to scale with local graph parameters such as the degree, this turns out not to be the case. Through explicit examples, we compute uniform deviation bounds and illustrate how the curse of dimensionality in density estimation can thus be circumvented. Notable examples where the rate improves substantially include sequential, hierarchical, and spatial data.

Breaking the curse of dimensionality in structured density estimation

Abstract

We consider the problem of estimating a structured multivariate density, subject to Markov conditions implied by an undirected graph. In the worst case, without Markovian assumptions, this problem suffers from the curse of dimensionality. Our main result shows how the curse of dimensionality can be avoided or greatly alleviated under the Markov property, and applies to arbitrary graphs. While existing results along these lines focus on sparsity or manifold assumptions, we introduce a new graphical quantity called 'graph resilience' and show how it controls the sample complexity. Surprisingly, although one might expect the sample complexity of this problem to scale with local graph parameters such as the degree, this turns out not to be the case. Through explicit examples, we compute uniform deviation bounds and illustrate how the curse of dimensionality in density estimation can thus be circumvented. Notable examples where the rate improves substantially include sequential, hierarchical, and spatial data.

1 Introduction

Density estimation is a classical problem in statistical machine learning, and provides the backbone of modern generative models such as diffusion models, which are now state-of-the-art density estimators for a variety of applications, as well as normalizing flows, energy-based models, and implicit generative models. At the same time, density estimation is a notoriously difficult problem in high-dimensions, known to suffer from the so-called curse of dimensionality. When the density depends on only a few variables or, more generally, is supported on a low-dimensional manifold, it is known that the curse of dimensionality can be circumvented by substituting the ambient dimension d with the effective dimension s (e.g. Lafferty and Wasserman, 2008; Yang and Tokdar, 2015). But what happens when the distribution is spread over the entire space in a structured manner-is it still possible to circumvent the curse of dimensionality?

Three representative examples are given in Figure 1, corresponding to sequential, hierarchical, and spatial (or convolutional) data. In these examples, although both manifold and sparsity assumptions are violated, there are structured dependencies that one might hope to leverage when estimating the underlying density. These kinds of structures are pervasive in machine learning applications. For one example, consider computer vision and imaging. Images have long been modeled as a grid graph where adjacent pixels correspond to adjacent vertices (see Keener (2010), for example). Such an assumption is very natural: pixels tend to be strongly dependent on nearby pixels and independent of far away pixels. See Figure 2 for an example of this.

These examples are particularly appealing in applications, however, we emphasize that our problem setting is considerably more general, and applies to general dependence structures given by any

Path (sequential data) Graph resilience: r = O (log d ) Samples: n ≳ (1 /ε ) O (log d )

Tree

(hierarchical data)

Graph resilience:

r

=

O

(1)

Samples:

n

≳

(1

/ε

)

O

(1)

Grid

(spatial data)

Graph resilience:

r

=

o

(

d

)

Samples:

n

≳

(1

/ε

)

o

(

d

)

Markov random field. Since real data is expected to (approximately) exhibit these types of structures, a reasonable question to ask is whether or not the curse of dimensionality persists in such structured settings.

Structured density estimation To formalize the notion of structured dependencies in a highdimensional, multivariate distribution, we adopt the framework of undirected graphical models, also known as Markov random fields (MRFs). In this setting, we are given samples from an unknown distribution P , with density p , over the random vector X = ( X 1 , . . . , X d ) , and it is assumed that P is Markov to some undirected graph G (see Section 3.1 for definitions). We treat both cases where G is a priori known, and where it is unknown but is in some known subset of all graphs. The graph G encodes the underlying dependence structure between the variables, which we hope simplifies the estimation problem. For example, when P is a Gaussian, this gives rise to the well-known Gaussian graphical model (Speed and Kiiveri, 1986), and various extensions of this idea to nonparametric settings are known, including trees (Liu et al., 2011; Györfi et al., 2022) and nonparanormal models (Liu et al., 2009). While this line of work also discusses the problem of structure learning , our focus is on the problem of nonparametric density estimation , which is comparatively understudied in graphical models. This may come as a surprise given the outsized literature on the general density estimation problem; see Section 2 for an overview of related work. In contrast to most existing work on density estimation, in lieu of imposing parametric or functional restrictions on P , the only assumption we make in addition to the Markov assumption is Lipschitz continuity.

Overview Our main result establishes the sample complexity of estimating such a density-for arbitrary graphs G -in total variation (TV) distance: It is approximately (1 /ε ) r +2 , where r ≪ d is a novel graphical parameter we call graph resilience that depends only on G . Roughly, r is a measure of how connected the graph G is; the easier it is to disconnect G , the smaller r will be. The examples in Figure 1 illustrate a range of values from constant r = O (1) to sublinear r = O ( √ d ) = o ( d ) . In the former case, this leads to an exponential improvement in the sample complexity over the usual nonparametric sample complexity of (1 /ε ) d .

While it is not surprising that graphical structure (i.e. sparsity in the form of conditional independence) can make estimation easier, what is surprising is the quantity involved: It is not, as one might guess, one of the 'usual' suspects such as sparsity, degree, or width. To capture the effective dimension of the problem and its resulting sample complexity, we introduce the concept of graph resilience . In particular, the usual suspects are insufficient to break the curse of dimensionality, whereas graph resilience does. Moreover, it is easy to construct examples where these are not only insufficient, but wholly misleading: Graph resilience can be controlled in graphs with unbounded degree, sparsity, and/or diameter.

Our work also marks a substantial departure from the existing literature that focuses primarily on functional restrictions (e.g. compositional structure), sparsity (e.g. density regression), or lowdimensional embeddings (e.g. manifold hypothesis). Instead, we impose no explicit restrictions on the functional form (although certain restrictions are implicit through the Markov assumption). In practice, empirical evidence points to a combination of these properties prevailing in real-world data,

Correlation

Cond.Correlation

30

30

1.0

25

25

0.8

20

20

0.6

15

15

0.4

10

10

0.2

5

5

0.0

0

0

5

10

15

20

25

30

0

0

5

10

15

20

25

30

and thus our approach hopefully serves to provide another practical assumption under which density estimation is feasible, and in particular, the curse of dimensionality can be avoided.

Contributions More precisely, we make the following contributions.

(Section 3.2) We introduce the graphical property of resilience (Definition 3.2), which approximately quantifies the connectivity of an undirected graph. Resilience is defined through the process of disintegration (Definition 3.1), which is described in detail.

(Section 3) We prove that the sample complexity of estimating a density p that is Markov to any undirected graph G scales with the resilience r = r ( G ) , as opposed to the dimension d (Theorem 3.1). We also provide examples to show that other metrics such as degree, diameter, and sparsity cannot properly capture the sample complexity (Section 4.3). We also show that efficient estimation is still possible even if G is unknown (Theorem 3.2).

(Section 4) We demonstrate numerous concrete examples where the resilience (and hence the sample complexity) can either be exactly calculated or bounded. These examples include familiar graphs such as trees (including paths for sequential data), cliques, and grids (also known as lattice graphs), and represent a broad continuum of possible complexities (Section 4.2).

All told, the potential savings implied by our results can be substantial, and are not isolated or pathological in any way: If there is a single independence relation satisfied by P , the effective dimension will be strictly less than the ambient dimension d , and in practical applications such as spatial or imaging data, there is an exponential savings in the sample complexity (see Figure 1). As we show, the graph resilience reveals a continuum of complexities ranging from dimensionindependent (i.e. r = O (1) ), in which case the curse of dimensionality is circumvented completely, to dimension-dependent with nontrivial savings (e.g. r = o ( d ) ).

2 Related work

We begin by recalling classical rates and results on density estimation. The standard nonparametric rate for estimating an L -Lipschitz density p in d dimensions in TV distance is n -1 / ( d +2) ; see Devroye and Gyorfi (1985); Tsybakov (2009); Giné and Nickl (2015) for more details. This rate ignores dimension-dependent constants that affect finite-sample rates, and a more refined bound on the sample

complexity (ignoring log-factors) is given in McDonald (2017):

See also Ghorbani et al. (2020); Jiao et al. (2023). For comparison, our main result is that only

samples are needed (again up to log-factors) when p is Markov to an undirected graph G with resilience r = r ( G ) , and this rate cannot be improved among graphs whose resilience is at most r . The improvement over (1) is clear: Not only the exponent, which dominates the rate, but also the constant is improved by replacing d with r , which can be much smaller than d (see examples in Section 4.2).

Ignoring the dimension-dependent constant factor (as most papers do), the basic idea behind most results on circumventing the curse of dimensionality is to replace the d -dependence in the exponent of (1) with some s < d , where s is the effective dimension of the problem. Examples include sparsity, low-dimensional embeddings (e.g. manifold assumptions), and hierarchical and/or compositional structure. Viewed from this perspective, our main contribution is to propose a new measure of effective dimension in structured data, where s = r is the graph resilience.

Recently, there has been a renewed interest in this problem along two broad axes: 1) Generative models as density estimators, and 2) Breaking the curse of dimensionality. In the remainder of this section, we review this related work.

2.1 Density estimation

As noted in the introduction, density estimation is a classical problem with a literature dating back more than 50 years. Notably, Stone (1980, 1982) established minimax rates for nonparametric estimation problems including density estimation and regression. These papers derived the now-classical nonparametric rate n -β/ (2 β + d ) for β -smooth densities, which implies the curse of dimensionality, i.e. unless p is very smooth, then the sample complexity of estimating p is exponential in the dimension. Yet, at the same time, the stark practical success of generative models suggest that high-dimensional density estimation may not be quite as intractable as this slow rate suggests. Motivated by these empirical observations, a growing line of work has established minimax optimality for a range of generative models, including GANs (Liang, 2017; Singh et al., 2018; Singh and Póczos, 2018; Uppal et al., 2019, 2020), diffusion models (Oko et al., 2023; Zhang et al., 2024; Cole and Lu, 2024; Tang and Yang, 2024), and variational autoencoders (Tang and Yang, 2021; Kwon and Chae, 2024).

Other theoretical developments have focused on efficient algorithms (Acharya et al., 2021, 2017; Chan et al., 2014) in the univariate case.

There has also been recent interest in developing density estimators that exploit graphical structure (Germain et al., 2015; Johnson et al., 2016; Khemakhem et al., 2021; Wehenkel and Louppe, 2021; Chen et al., 2024). Of course, there is an enormous literature on algorithms and methods for general density estimation that we cannot cover here.

Most closely related to our work are the papers Liu et al. (2007, 2011); Györfi et al. (2022). Liu et al. (2007) use the RODEO estimator on a model that satisfies a sparsity assumption, i.e. only s ≪ d variables are involved in the nonparametric component. Liu et al. (2011) use forests to approximate the underlying density under certain regularity conditions; Györfi et al. (2022) relax these conditions and replace forests with trees. Their main result is a pointwise O ( n -1 / 4 ) rate of convergence for estimating a tree-structured density, which is notably dimension-independent. The main difference between our results and these previous results is that our results apply to general graphs G that may not be trees or forests, in addition to being uniform in p and unimprovable (cf. Remark 1 in Györfi et al., 2022, in particular). Most importantly, moving beyond tree-based models requires new ideas, and motivates our introduction of the graph resilience to measure the effective dimension of the estimation problem.

After the initial posting of our paper, we were made aware of the related work by Bos and SchmidtHieber (2023) which proposes a supervised approach to density estimation that also leverages the Markov assumption to break the curse of dimensionality in density estimation.

2.2 Curse of dimensionality

There is a long line of literature on understanding how and when the curse of dimensionality can be avoided. Common assumptions include the manifold assumption (Pelletier, 2005; Ozakin and Gray, 2009; Jiang, 2017; Schmidt-Hieber, 2019; Nakada and Imaizumi, 2020; Berenfeld et al., 2022; Jiao et al., 2023), additive structure (Stone, 1985; Raskutti et al., 2012), compositional structure (Horowitz and Mammen, 2007; Juditsky et al., 2009; Kohler and Krzy˙ zak, 2017; Schmidt-Hieber, 2017; Bauer and Kohler, 2019; Kohler and Langer, 2021; Shen et al., 2021), low-rank structure (Hall and Zhou, 2003; Hall et al., 2005; Song and Dai, 2013; Amiridi et al., 2022a,b; Vandermeulen and Ledent, 2021; Vandermeulen, 2023) and sparsity (Liu et al., 2007; Lafferty and Wasserman, 2008; Yang and Tokdar, 2015). Bach (2017) showed that neural networks are adaptive to many of these underlying structures.

This line of work is particularly relevant as it pertains to breaking the curse of dimensionality via structural assumptions on the unknown parameter. Notably, it seems that the advantages of (in)dependence via the Markov property have not been thoroughly investigated. Our work aims to fill this gap for a wide range of structured models that do not fit into any of the classes above. Indeed, it is easy to construct densities that are non-sparse (i.e. every variable is active), non-additive and non-compositional (we consider arbitrary continuous densities), and are not supported on any lower-dimensional manifold, but that are Markov to a given graph G .

3 Main Results

Before presenting the main results of this paper we must present some basic background.

3.1 Background Definitions and Notation

Throughout the paper, we use undirected graphs to model the dependencies in P . We adopt the usual terminology and conventions from graphical models: G = ( V, E ) is an undirected graph with V = X = [ d ] and d = dim( X ) . To avoid technical complications, we assume compact support with X = ( X 1 , . . . , X d ) ∈ [0 , 1] d . Two disjoint subsets A,B ⊂ V are said to be separated by C if all paths connecting A to B intersect C ; equivalently, the subgraph over ( A ∪ B ) -C is disconnected. The distribution P is called Markov with respect to G if

where ⊥ ⊥ P denotes conditional independence in P . In other words, graph separation implies conditional independence, but not necessarily vice versa. See Lauritzen (1996) for a review of graphical modeling terminology.

We term a subgraph of G to be a component of G (sometimes called a connected component ) if it is a maximal connected subgraph of G . A path in G is a sequence of vertices ( v 0 , v 1 , . . . , v k ) such that v i -v i +1 in G (i.e. ( i, i +1) ∈ E ). A simple path is a path without repeated vertices. The length of a path is the number of edges in the path; e.g. the length of ( v 0 , v 1 , . . . , v k ) is k . A geodesic path between two vertices is any path of shortest length, and the distance between two vertices is the length of any geodesic between them. For graphs G,G ′ the notation G ⊕ G ′ denotes a disjoint union of graphs, i.e., the graph whose vertex set is the disjoint union of the vertices in G and G ′ and inherits edges from the edges in G and G ′ . For a graph G = ( V, E ) , for V ′ ⊂ V , the graph G \ V ′ denotes the graph with vertices V \ V ′ and the edges { i, j } ∈ E where { i, j } ⊆ V \ V ′ .

For any d ∈ N and L ≥ 0 , let D d be the set of densities on [0 , 1] d and D d,L ⊂ D d be those densities which are L -Lipschitz continuous. For any graph G with d vertices, we define D ( G ) ⊆ D d , such that, for p ∈ D ( G ) and ( X 1 , . . . , X d ) ∼ p , the entries of the random variable, X 1 , . . . , X d , satisfy the global Markov property with respect to the graph G . Finally let D L ( G ) ≜ D ( G ) ∩ D d,L . When estimating these densities we will sometimes use the term sample complexity . This refers to the number of samples necessary to estimate a density to within ε error in total variation distance. For functions, f on [0 , 1] d we define ∥ f ∥ 1 = ∫ | f ( x ) | dx ; this is the total variation distance.

3.2 Graph Resilience

The key concept in this work, which characterizes the difficulty of estimating densities in D ( G ) , is what we term the graph resilience of G . Graph resilience is based on a process we call a disintegration .

1

2

3

4

5

6

7

1

2

3

4

5

7

2

2

4

3

null graph

Definition 3.1. For a graph G = ( V, E ) , an r -tuple ( V 1 , . . . , V r ) with V i ⊆ V is called a disintegration of G if:

{ V i } i is a partition of V ;

The elements of V 1 all lie in different components of G ;

| V 1 | is equal to the number of components in G ;

For all i ∈ [ r -1] the elements of V i +1 lie in different components of G \ ⋃ i j =1 V j ;

| V i +1 | is equal to the number of components in G \ ⋃ i j =1 V j .

The length of a disintegration is the value r in the above definition. A disintegration of length r will be called an r -disintegration.

The first disintegration step, V 1 above, can be thought of as the process of picking one vertex from each component in G and removing it. The next step of a disintegration, V 2 above, then selects and removes one vertex from each component of the resultant graph. An r -disintegration is a sequence of r such steps until one is left with the null graph. It is possible for a graph to admit many different disintegrations of different lengths. A visual representation of the steps of a graph disintegration can be found in Figure 3.

The resilience of a graph G describes the length of the shortest possible disintegration of G .

Definition 3.2. For a graph G , the resilience of G , denoted r ( G ) , is the smallest r such that there exists a r -disintegration of G .

We elaborate more on some properties of graph resilience in Section 4, including upper bounds for the graph resilience of several graphs corresponding to image or sequence data. First, we need to establish that graph resilience is the right metric for quantifying the sample complexity of density estimation.

3.3 Sample complexity

The following theorem characterizes the difficulty of estimating a density which is known to satisfy the Markov property in terms of its graph. All proofs are deferred the appendix.

Theorem 3.1. Let G be a (known) graph whose number of vertices is d and resilience is r . Let L ≥ 0 . For any 0 < ε < 1 , there exists an algorithm that takes n = Ω ( rd r/ 2+1 L r ε r +2 log( dL ε ) + log(1 /δ ) ε 2 ) i.i.d. samples drawn from any p ∈ D L ( G ) and returns a distribution q such that

This corresponds to a convergence rate of ˜ O ( n -1 / ( r +2) ) and is uniform over D L ( G ) . For comparison, the rate O ( n -1 / ( d +2) ) is known to be optimal for estimating densities in D d,L in the total variation norm (e.g. Beirlant and Gyorfi, 1998). Consequently the rate ˜ O ( n -1 / ( r +2) ) indicates that the rate of convergence when estimating densities in D L ( G ) is akin to estimating densities in D r,L . We will see in Section 4.2 that, for graphs typically used to represent audio or image data, the effective dimension for estimating densities can be drastically, even exponentially, smaller than the ambient dimension.

While it may be reasonable to assume a known graph in some situations, one often encounters the situation where the graph is unknown. If one assumes that the graph G is unknown, but that G lies in some subset of graphs whose maximum graph resilience is bounded above by some value r then it is still possible to achieve a rate of ˜ O ( n -1 / ( r +2) ) .

Theorem 3.2. Let G be the set of all graphs whose number of vertices is d and resilience is r . Let L ≥ 0 . For any 0 < ε < 1 , there exists an algorithm that takes n = Ω ( rd r/ 2+1 L r ε r +2 log( dL ε ) + log(1 /δ ) ε 2 )

i.i.d. samples drawn from any p ∈ ⋃ G ∈G D L ( G ) and returns a distribution q such that

We show in Appendix D that these rates are optimal (up to a polylogarithmic factor) for all dimensions d and resiliences r . However, it's worth noting that these rates are not optimal for all graphs G . For instance, for the special case of trees see Györfi et al. (2022), where better rates can be achieved. The case of general graphs G is an open problem.

3.4 Proof Outline and Practical Consequences

We will outline our proof techniques and demonstrate their relation to practically implementable estimators. Our theorem proofs employ disintegrations to construct a class of densities that closely approximate those with the given MRF. This class contains densities that take the form of histograms, i.e., piecewise constant densities on a grid. The remainder of the argument is relatively standard: A finite collection of the aforementioned class is found to cover the space, from which our estimate is chosen using a method akin to Scheffé tournaments (Scheffe, 1947; Yatracos, 1985; Devroye and Lugosi, 2001; Ashtiani et al., 2018). Disintegrations are the core novel aspect of this work, characterizing a class of histograms much smaller than the full set, thereby reducing its metric entropy and enhancing the sample efficiency of our selection.

At a high level, a disintegration outlines a method to estimate a density by inductively conditioning out the entries of a random vector. For instance, consider a random vector [ X 1 , X 2 ] . The disintegration in Figure 3.4 corresponds to first estimating a histogram for X 2 , and then, for each bin b of the X 2 histogram, estimating a histogram for

X 1 | X 2 ∈ b . The resilience of a graph simply characterizes the shortest disintegration, i.e. the most efficient decomposition of a distribution into factors. Removing one vertex from each component of a graph captures the idea that, after sufficient conditioning, these components become independent. This allows us to avoid estimating the high-dimensional joint density of all vertices in the graph. Instead, we can estimate the low-dimensional components individually and take their product, which is effective due to the following inequality

Thus a disintegration can be thought of as a 'meta-algorithm' providing an ordering for estimating conditional densities. In practice, it would be up to the implementation to determine how to handle the one-dimensional and conditional density estimation.

4 Graph Resilience Examples

Theorems 3.1 and 3.2 demonstrate that the graph resilience acts as the effective dimension of a nonparametric estimation problem, however, graph resilience is still somewhat of an opaque property. In this section we will go over basic properties of graph resilience and describe graph resiliences for some graphs that reflect real-world settings.

4.1 Graph Resilience Properties

Here we present some basic results regarding graph resilience. We first introduce two very basic lemmas outlining the most fundamental properties of graph resilience. The first lemma shows how graph resilience behaves with disjoint graph union.

This lemma encapsulates the notion that estimating joint density for a collection of independent random vectors, e.g., ( Y 1 , . . . , Y m ) ∼ ∏ m i =1 µ i , from a collection of samples is no more difficult than estimating each independent vector individually, ̂ µ i ≈ µ i , and taking the product measure, ∏ m i =1 ̂ µ i ≈ ∏ m i =1 µ i .

The second lemma demonstrates the behavior of graph resilience upon vertex removal.

This corresponds to conditioning the random variables in V \ V ′ on the | V ′ | -dimensional random vector corresponding to V ′ .

Simpler graphs, in terms of number of edges and vertices, tend to have smaller graph resilience. The following lemma shows that adding edges to a graph will, at most, increase its resilience by the number of edges added.

Lemma 4.3. Let G = ( V, E ) be a graph, let E ′ be a set of edges for vertices V , and let G ′ = ( V, E ∪ E ′ ) , then r ( G ′ ) ≤ r ( G ) + | E ′ | .

For a pair of graphs G,G ′ , let the relation G ′ ≤ G denote that G ′ is isomorphic to some subgraph of G . The following lemma demonstrates that removing random variables and dependencies between random variables can only reduce graph resilience.

Lemma 4.4. For a pair of graphs G and G ′ , if G ′ ≤ G , then r ( G ′ ) ≤ r ( G ) .

The following corollary follows directly from the previous Lemma 4.4 and Lemma 4.6, which we present later.

Corollary 4.5. Let G be a graph whose largest clique contains k vertices, then r ( G ) ≥ k .

4.2 Example Graph Resiliences

From these properties of graph resilience we can derive the graph resilience of some example graphs. Some results from this section are summarized in Table 1. We begin with a couple of simple cases.

The following lemma demonstrates the surprising fact that, if even a single edge is missing from the graph, then the effective dimension of the estimation problem is strictly smaller than the ambient dimension d .

Lemma 4.6. For a graph G = ( V, E ) , r ( G ) = | V | if and only if G is a complete graph.

Analogously, a graph can only have resilience 1 if it is the empty graph.

Lemma 4.7. For a graph G = ( V, E ) , r ( G ) = 1 if and only if E = ∅ .

The star graph is an example of a connected graph whose resilience is very small. Recall that a star graph S d is a graph with a single central vertex connected to every other node, i.e. E = { ( i, j ) : j = i } . Equivalently, it is (a) a complete bipartite graph with a single vertex in one partition or (b) a tree of depth one with a single root.

̸

In the remainder of this section, we describe additional examples of graphs whose resilience can be bounded. We focus on four broad classes of graphs, categorized by the application of interest: Hierarchical data (e.g. language, biology, phylogeny), sequential data (e.g. audio, video, language), spatial data (e.g. images, vision, signal processing), and clustered data (e.g. genetics, ecology, medicine). See Figure 1 for a guide to these classes of graphs.

The following definition will be helpful for analyzing graphs that have a linear or grid shape.

Definition 4.1. For a graph G and n ∈ N , the power graph G n , is the graph which inherits its vertices from G and a pair of vertices in G n are adjacent when their distance in G is at most n .

4.2.1 Hierarchical Data

Hierarchical data arises from distributions that have a tree-like structure, i.e. G is tree. In the worst-case, the resilience of a tree can scale at most logarithmically with d , however, in practical applications where the tree has bounded depth the resilience is also bounded.

Lemma 4.9. Let G be a k -ary tree with depth m , then r ( G ) ≤ m .

Lemma 4.10. Let G be a tree with d vertices, then r ( G ) ≤ log 2 ( d ) + 1 .

4.2.2 Sequential Data

A path graph naturally represents sequential data. For a more realistic model one may assume that a vertex in a path graph is connected to all vertices within a given distance, so that nearby vertices are highly dependent, but far away indices are less dependent. Even with this assumption we see that, like tree graphs, the path graph's resilience only grows at rate O (log d ) .

Definition 4.2. The path graph of length d , denoted L d , is the graph ( V, E ) with V = [ d ] and E = {{ i, j } | | i -j | = 1 } .

Proposition 4.11. Recall that L d is the path graph of length d as defined in Definition 4.2. For any s, t ∈ N , we have

Here, L t t (2 s -1) is the power graph of L t (2 s -1) as defined in Definition 4.1.

This, along with Lemma 4.4, yields the following rate on graph resilience for path graphs.

Corollary 4.12. For any d ∈ N and any constant t ∈ N , we have

4.2.3 Spatial Data

Image data is naturally represented via a grid graph. The following graph describes a k × k ′ grid of vertices with every vertex connected connected to its vertical, horizontal, and diagonal, neighbors.

̸

Definition 4.3. The grid graph of shape k × k ′ , denoted L k × k ′ , is the graph ( V, E ) with V = [ k ] × [ k ′ ] and E = {{ ( i, j ) , ( i ′ , j ′ ) } | ( i, j ) = ( i ′ , j ′ ) , | i -i ′ | ≤ 1 , | j -j ′ | ≤ 1 } .

Proposition 4.13. Recall that L k × k be the grid graph of shape k × k as defined in Definition 4.3. For any s, t ∈ N , we have

Here, L t t (2 s -1) × t (2 s -1) is the power graph of L t (2 s -1) × t (2 s -1) as defined in Definition 4.1.

Examples of a grid graph and its power can be found in Figure 5 in the appendix. The previous proposition gives us a general rate of √ d for grid graphs.

Corollary 4.14. For any k ∈ N and any constant t ∈ N , we have

Note that the graph L t k × k has d vertices.

4.2.4 Clustered Data

Another common type of structured data exhibits clusters: Variables in the same cluster are dependent, while variables in different clusters are independent. This type of structure can be modeled with disjoint cliques, where each clique represents a cluster or community. Let V i ⊂ V denote each cluster, with V i ∩ V j = ∅ and V = V 1 ∪··· ∪ V m (i.e. { V i } is a partition of V ). Let d i ≜ | V i | , G i ≜ K d i be a clique (complete graph) on V i , and G = ⊕ i G i . Then it follows from Lemmas 4.6 and 4.1 that r ( G ) = max i d i . In particular, if d i = O (1) , then r ( G ) = O (1) . Of course, this simply recovers the well-known result that a density that factorizes into a product of densities can be estimated at a rate that depends only on the largest factor. Using our graph resilience analysis we have that, for a stochastic block model, the graph resilience is bounded by the size of the blocks times the graph that encapsulates dependencies between the blocks.

Lemma 4.15. Let G ′ = ([ k ] , E ′ ) be a graph. Let G = ( V, E ) be a graph, such that there is a partition of V = ⋃ k i =1 V i , where, if { v, v ′ } ∈ E it follows that v ∈ V i and v ′ ∈ V j where i = j or i and j are adjacent in G ′ . Then r ( G ) ≤ r ( G ′ ) max i ∈ [ k ] | V i | .

This has the useful interpretation that the effect of additional dependencies between clusters does not interact with the clusters themselves. For example, if we allow for noisy interactions between clusters (e.g. as in a stochastic block model), the complexity scales separately with the noise and the size of the largest cluster.

4.3 Comparison to other graphical properties

The examples in the previous section can be used to show that the classical graphical properties that one might expect to govern the sample complexity surprisingly fail to capture the sample complexity.

Degree The first and most obvious is the maximum degree of G . The star graph S d is an example where the resilience is O (1) , and hence the sample complexity is (1 /ε ) O (1) , but the maximum degree is Ω( d ) . Thus, the degree cannot properly capture the sample complexity. Moreover, the path graph shows the reverse: The maximum degree can be O (1) while the resilience is Ω(log d ) .

Diameter The diameter diam( G ) of a graph G is the length of its longest geodesic path. A clique K d thus has diam( K d ) = 1 (since every node is connected to every other node), but r ( K d ) = d . It is clear even from classical results that a clique cannot be estimated any faster than (1 /ε ) d . Thus, the diameter also cannot capture the sample complexity.

Sparsity Call a density p on d variables s -sparse if p ( x ) = q ( x S ) , where S ⊂ [ d ] with | S | = s . Now suppose p is s -sparse and q is Markov to a star graph S s on s vertices. Then Theorem 3.1 implies that p can be estimated with (1 /ε ) O (1) samples (since r ( G ) = O (1) ). Thus, taking s →∞ , it follows that the sparsity s cannot properly capture the sample complexity.

In other words, not only does the graph resilience properly capture the sample complexity of structured density estimation, it is also not simply a proxy for commonly used graphical properties.

5 Conclusion

This work has introduced a new concept, graph resilience , and demonstrated that it controls the complexity of estimating densities satisfying the Markov property. This characterization has shown that estimating such densities can be significantly easier than previous works have indicated. This finding contributes to the broader understanding of graph theoretical properties in statistical estimation and, more generally, to nonparametric estimation. Although our approach sheds light on the intrinsic possibilities and barriers to breaking the curse of dimensionality, the development of practical methods remains an important open problem. For example, can neural networks achieve these rates? Recent concurrent work has provided insight into the use of neural density estimators, see e.g. Bos and Schmidt-Hieber (2023); Cole and Lu (2024); Vandermeulen et al. (2024). Finally, the concept of resilience is a new and unexplored property with significant potential for discovering graphs that yield good estimation properties.

Acknowledgments and Disclosure of Funding

Robert A. Vandermeulen was supported by German Federal Ministry of Education and Research (BMBF) grants BIFOLD23B and BIFOLD24B. Wai Ming Tai was supported by Singapore AcRF Tier 2 grant MOE-T2EP20122-0001. B.A. was supported by NSF IIS-1956330, NIH R01GM140467, and the Robert H. Topel Faculty Research Fund at the University of Chicago Booth School of Business.

References

Jayadev Acharya, Ilias Diakonikolas, Jerry Li, and Ludwig Schmidt. Sample-optimal density estimation in nearly-linear time. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1278-1289. SIAM, 2017.

Jayadev Acharya, Clement Canonne, Aditya Vikram Singh, and Himanshu Tyagi. Optimal rates for nonparametric density estimation under communication constraints. Advances in Neural Information Processing Systems , 34:26754-26766, 2021.

Magda Amiridi, Nikos Kargas, and Nicholas D. Sidiropoulos. Low-rank characteristic tensor density estimation part I: Foundations. IEEE Transactions on Signal Processing , 70:2654-2668, 2022a. doi: 10.1109/TSP.2022.3175608.

Magda Amiridi, Nikos Kargas, and Nicholas D. Sidiropoulos. Low-rank characteristic tensor density estimation part II: Compression and latent density estimation. Trans. Sig. Proc. , 70: 2669-2680, January 2022b. ISSN 1053-587X. doi: 10.1109/TSP.2022.3158422. URL https: //doi.org/10.1109/TSP.2022.3158422 .

Hassan Ashtiani, Shai Ben-David, Nicholas Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Nearly tight sample complexity bounds for learning mixtures of gaussians via sample compression schemes. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 3412-3421. Curran Associates, Inc., 2018.

Francis Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research , 18(19):1-53, 2017.

Benedikt Bauer and Michael Kohler. On deep learning as a remedy for the curse of dimensionality in nonparametric regression. Annals of Statistics , 47(4):2261-2285, 2019.

Beirlant and L. Gyorfi. On the L 1 -error in histogram density estimation: The multidimensional case. Journal of Nonparametric Statistics , 9(2):197-216, 1998. doi: 10.1080/10485259808832742. URL https://doi.org/10.1080/10485259808832742 .

Clément Berenfeld, Paul Rosa, and Judith Rousseau. Estimating a density near an unknown manifold: a bayesian nonparametric approach. arXiv preprint arXiv:2205.15717 , 2022.

Thijs Bos and Johannes Schmidt-Hieber. A supervised deep learning method for nonparametric density estimation. arXiv e-prints , art. arXiv:2306.10471, June 2023. doi: 10.48550/arXiv.2306. 10471.

Siu On Chan, Ilias Diakonikolas, Rocco A Servedio, and Xiaorui Sun. Near-optimal density estimation in near-linear time using variable-width histograms. Advances in neural information processing systems , 27, 2014.

Asic Chen, Ruian Ian Shi, Xiang Gao, Ricardo Baptista, and Rahul G Krishnan. Structured neural networks for density estimation and causal inference. Advances in Neural Information Processing Systems , 36, 2024.

Frank Cole and Yulong Lu. Score-based generative models break the curse of dimensionality in learning a family of sub-gaussian probability distributions. arXiv preprint arXiv:2402.08082 , 2024.

Devroye and L. Gyorfi. Nonparametric Density Estimation: The L 1 View . Wiley Interscience Series in Discrete Mathematics. Wiley, 1985. ISBN 9780471816461.

Devroye and G. Lugosi. Combinatorial Methods in Density Estimation . Springer, New York, 2001.

Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning , pages 881-889. PMLR, 2015.

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Discussion of: 'nonparametric regression using deep neural networks with ReLU activation function'. The Annals of Statistics , 48(4):1898 - 1901, 2020. doi: 10.1214/19-AOS1910. URL https: //doi.org/10.1214/19-AOS1910 .

Evarist Giné and Richard Nickl. Mathematical Foundations of Infinite-Dimensional Statistical Models . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2015.

László Györfi, Aryeh Kontorovich, and Roi Weiss. Tree density estimation. IEEE Transactions on Information Theory , 69(2):1168-1176, 2022.

Peter Hall and Xiao-Hua Zhou. Nonparametric estimation of component distributions in a multivariate mixture. Annals of Statistics , pages 201-224, 2003.

Peter Hall, JS Marron, and Amnon Neeman. Geometric representation of high dimension, low sample size data. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 67(3): 427-444, 2005.

Joel L Horowitz and Enno Mammen. Rate-optimal estimation for a general class of nonparametric regression models with unknown link functions. Annals of Statistics , 35(6):2589-2619, 2007.

Heinrich Jiang. Uniform convergence rates for kernel density estimation. In International Conference on Machine Learning , pages 1694-1703, 2017.

Yuling Jiao, Guohao Shen, Yuanyuan Lin, and Jian Huang. Deep nonparametric regression on approximate manifolds: Nonasymptotic error bounds with polynomial prefactors. The Annals of Statistics , 51(2):691-716, 2023.

Matthew J Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast inference. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29 , pages 2946-2954. Curran Associates, Inc., 2016.

Anatoli B. Juditsky, Oleg V. Lepski, and Alexandre B. Tsybakov. Nonparametric estimation of composite functions. The Annals of Statistics , 37(3):1360 - 1404, 2009. doi: 10.1214/08-AOS611. URL https://doi.org/10.1214/08-AOS611 .

Robert W. Keener. Bayesian Inference: Modeling and Computation , pages 301-318. Springer New York, New York, NY, 2010. ISBN 978-0-387-93839-4. doi: 10.1007/978-0-387-93839-4_15. URL https://doi.org/10.1007/978-0-387-93839-4_15 .

Ilyes Khemakhem, Ricardo Monti, Robert Leech, and Aapo Hyvarinen. Causal autoregressive flows. In International Conference on Artificial Intelligence and statistics , pages 3520-3528. PMLR, 2021.

Michael Kohler and Adam Krzy˙ zak. Nonparametric regression based on hierarchical interaction models. IEEE Transactions on Information Theory , 63(3):1620-1630, 2017.

Michael Kohler and Sophie Langer. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics , 49(4):2231-2249, 2021.

Hyeok Kyu Kwon and Minwoo Chae. Minimax optimal density estimation using a shallow generative model with a one-dimensional latent variable. In International Conference on Artificial Intelligence and Statistics , pages 469-477. PMLR, 2024.

John Lafferty and Larry Wasserman. Rodeo: sparse, greedy nonparametric regression. Annals of Statistics , 36(1):28-63, 2008.

Steffen L Lauritzen. Graphical models . Oxford University Press, 1996.

Tengyuan Liang. How well can generative adversarial networks (GAN) learn densities: A nonparametric view. arXiv preprint arXiv:1712.08244 , 2017.

Han Liu, John Lafferty, and Larry Wasserman. Sparse nonparametric density estimation in high dimensions using the rodeo. In Artificial Intelligence and Statistics , pages 283-290. PMLR, 2007.

Han Liu, John Lafferty, and Larry Wasserman. The nonparanormal: Semiparametric estimation of high dimensional undirected graphs. Journal of Machine Learning Research , 10(Oct):2295-2328, 2009.

Han Liu, Min Xu, Haijie Gu, Anupam Gupta, John Lafferty, and Larry Wasserman. Forest density estimation. The Journal of Machine Learning Research , 12:907-951, 2011.

Daniel McDonald. Minimax density estimation for growing dimension. In Artificial Intelligence and Statistics , pages 194-203. PMLR, 2017.

Ryumei Nakada and Masaaki Imaizumi. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. Journal of Machine Learning Research , 21(174):1-38, 2020.

Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 26517-26582. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/oko23a.html .

Arkadas Ozakin and Alexander Gray. Submanifold density estimation. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems , volume 22. Curran Associates, Inc., 2009. URL https://proceedings.neurips.cc/ paper_files/paper/2009/file/2ac2406e835bd49c70469acae337d292-Paper.pdf .

Bruno Pelletier. Kernel density estimation on riemannian manifolds. Statistics & probability letters , 73(3):297-304, 2005.

Garvesh Raskutti, Martin J Wainwright, and Bin Yu. Minimax-optimal rates for sparse additive models over kernel classes via convex programming. Journal of machine learning research , 13(2), 2012.

R.D. Reiss. Approximate distributions of order statistics: with applications to nonparametric statistics . Springer series in statistics. Springer, 1989. ISBN 9783540968511. URL https: //books.google.de/books?id=DxzvAAAAMAAJ .

Henry Scheffe. A Useful Convergence Theorem for Probability Distributions. The Annals of Mathematical Statistics , 18(3):434 - 438, 1947. doi: 10.1214/aoms/1177730390. URL https: //doi.org/10.1214/aoms/1177730390 .

Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. arXiv preprint arXiv:1708.06633 , 2017.

Johannes Schmidt-Hieber. Deep ReLU network approximation of functions on a manifold. arXiv preprint arXiv:1908.00695 , 2019.

Guohao Shen, Yuling Jiao, Yuanyuan Lin, Joel L Horowitz, and Jian Huang. Deep quantile regression: Mitigating the curse of dimensionality through composition. arXiv preprint arXiv:2107.04907 , 2021.

Shashank Singh and Barnabás Póczos. Minimax distribution estimation in Wasserstein distance. arXiv e-prints , art. arXiv:1802.08855, February 2018. doi: 10.48550/arXiv.1802.08855.

Shashank Singh, Ananya Uppal, Boyue Li, Chun-Liang Li, Manzil Zaheer, and Barnabás Póczos. Nonparametric density estimation under adversarial losses. Advances in Neural Information Processing Systems , 31, 2018.

Le Song and Bo Dai. Robust low rank kernel embeddings of multivariate distributions. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/2013/file/ 49b8b4f95f02e055801da3b4f58e28b7-Paper.pdf .

Terence P Speed and Harri T Kiiveri. Gaussian Markov distributions over finite graphs. The Annals of Statistics , pages 138-150, 1986.

Charles J Stone. Optimal rates of convergence for nonparametric estimators. The Annals of Statistics , 8(6):1348-1360, 1980.

Charles J Stone. Optimal global rates of convergence for nonparametric regression. The annals of statistics , pages 1040-1053, 1982.

Charles J Stone. Additive regression and other nonparametric models. Ann. Statist. , 13(1):689-705, 1985.

Rong Tang and Yun Yang. On empirical bayes variational autoencoder: An excess risk bound. In Conference on Learning Theory , pages 4068-4125. PMLR, 2021.

Rong Tang and Yun Yang. Adaptivity of diffusion models to manifold structures. In International Conference on Artificial Intelligence and Statistics , pages 1648-1656. PMLR, 2024.

A.B. Tsybakov. Introduction to nonparametric estimation. Springer Series in Statistics, New York , page 214, 2009. cited By 1.

Ananya Uppal, Shashank Singh, and Barnabás Póczos. Nonparametric density estimation & convergence rates for GANs under Besov IPM losses. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 9086-9097, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ fb09f481d40c4d3c0861a46bd2dc52c0-Abstract.html .

Ananya Uppal, Shashank Singh, and Barnabas Poczos. Robust density estimation under Besov IPM losses. Advances in Neural Information Processing Systems , 33:5345-5355, 2020.

Robert A Vandermeulen. Sample complexity using infinite multiview models. arXiv preprint arXiv:2302.04292 , 2023.

Robert A Vandermeulen and Antoine Ledent. Beyond smoothness: Incorporating low-rank analysis into nonparametric density estimation. Advances in Neural Information Processing Systems , 34: 12180-12193, 2021.

Robert A Vandermeulen, Wai Ming Tai, and Bryon Aragam. Dimension-independent rates for structured neural density estimation. arXiv preprint arXiv:2411.15095 , 2024.

Antoine Wehenkel and Gilles Louppe. Graphical normalizing flows. In International Conference on Artificial Intelligence and Statistics , pages 37-45. PMLR, 2021.

Yun Yang and Surya T Tokdar. Minimax-optimal nonparametric regression in high dimensions. Annals of Statistics , 43(2):652-674, 2015.

Yannis G. Yatracos. Rates of Convergence of Minimum Distance Estimators and Kolmogorov's Entropy. The Annals of Statistics , 13(2):768 - 774, 1985. doi: 10.1214/aos/1176349553. URL https://doi.org/10.1214/aos/1176349553 .

Kaihong Zhang, Heqi Yin, Feng Liang, and Jingbo Liu. Minimax optimality of score-based diffusion models: Beyond the density lower bound assumptions. arXiv preprint arXiv:2402.15602 , 2024.

A Grid Figure

B Proofs on Density Estimation

We begin by introducing some objects and notation that will be essential for proving the main theorems.

Let N = { 1 , 2 , . . . } . For any m ∈ N , let [ m ] ≜ { 1 , 2 , . . . , m } . For any b ∈ N , let ∆ b be the simplex in R b , i.e.

For any d, b ∈ N , let T d,b be the set of probability tensors in R b × d . This can be thought of as the joint probability tables over d variables with states in [ b ] . For any d, b ∈ N and T ∈ T d,b , we say a random variable X ∼ T if

For any graph G with d vertices, let T b ( G ) ⊆ T d,b , such that, for T ∈ T b ( G ) and ( X 1 , . . . , X d ) ∼ T , then X 1 , . . . , X d satisfies the Markov property with respect to the graph G . We consider the domain [0 , 1) d and split it into b bins per dimension, i.e. there are b d bins in total. For any d, b ∈ N and multi-index A ∈ [ b ] d , let Λ d,b,A be the bin indexed at A and Λ d,b be the set of all bins, i.e.

For any d, b ∈ N , let H d,b be the set of all histograms with the bins from Λ d,b , i.e.

Let U d,b : T d,b →H d,b be the natural ℓ 1 → L 1 linear isometry between the spaces.

For any d ∈ N and L > 0 , let D d be the set of densities on [0 , 1) d and D d,L ⊂ D d be those densities which are L -Lipschitz continuous.

Definition B.1. For A , some subset of a ℓ 1 Euclidean tensor space or L 1 function space, we define

In this work we will use the following version of the Markov property: for I, J ⊂ V , with I and J not intersecting or adjacent, X I ⊥ ⊥ X J | X V \ I ∪ J . Note that this property is satisfied when the 'global' Markov property is satisfied.

Lemma B.1 (Lemma 3.3.7 from Reiss (1989)) . For probability measures ∥ ∥ ∥ ∏ t i =1 µ i -∏ t i =1 ν i ∥ ∥ ∥ 1 ≤ ∑ t i =1 ∥ µ i -ν i ∥ 1 .

Lemma B.2 (Lemma A.1 from Vandermeulen and Ledent (2021)) . For all b and 0 < ε ≤ 1 , N ( ∆ b , ε ) ≤ (2 b/ε ) b .

Lemma B.3 (Theorem 3.4 page 7 of Ashtiani et al. (2018), Theorem 3.6 page 54 of Devroye and Lugosi (2001)) . There exists a deterministic algorithm that, given a collection of distributions p 1 , . . . , p M , a parameter ε > 0 and at least log ( 3 M 2 /δ ) 2 ε 2 iid samples from an unknown distribution p , outputs an index j ∈ [ M ] such that

with probability at least 1 -δ 3 .

B.1 Preliminary Technical Results

This section contains intermediate technical results which will aid in proving more core results.

Lemma B.4. For any 0 < ε < 1 , any b, t ∈ N and any graph G i for i ∈ [ t ] , we have

Proof of Lemma B.4. Let C i be any ( ε/t ) -cover for T b ( G i ) such that N ( T b ( G i ) , ε/t ) = | C i | for all i ∈ [ t ] . If we manage to argue that the set

is an ε -cover of T b ( ⊕ t i =1 G i ) , then we immediately show the statement since

Now, to show that C is an ε -cover, we observe that any T ∈ T b ( ⊕ t i =1 G i ) has the form of T 1 ×···× T t where T i ∈ T b ( G i ) for all i ∈ [ t ] since we recall that ⊕ t i =1 G i is simply a union of the graphs with no edges across different connected components which implies the vertices in different connected components are independent. Since C i is an ( ε/t ) -cover of T b ( G i ) , there is a ˜ T i ∈ C i such that ∥ ˜ T i -T i ∥ 1 ≤ ε/t . By taking ˜ T = ˜ T 1 ×··· × ˜ T t ∈ C , there exists a ˜ T ∈ C such that, by Lemma B.1, we have

Lemma B.5. For any 0 < α < 1 , any 0 < ε < 1 , any b ∈ N , any graph G and any vertex v in G , we have

Proof of Lemma B.5. Without loss of generality, we assume v = 1 . Suppose the graph G has d vertices. Let C 0 be any (1 -α ) ε -cover of T b ( G \ { 1 } ) such that N ( T b ( G \ { 1 } ) , (1 -α ) ε ) = | C 0 |

and C 1 be any αε -cover of T 1 ,b such that N ( T 1 ,b , αε ) = | C 1 | . We now construct a set C ∈ T b ( G ) as follows.

If we manage to argue that the set C is an ε -cover of T b ( G ) , then we immediately show the statement since

Now, to show that C is an ε -cover, we observe that, for any T ∈ T b ( G ) and A ∈ [ b ] d , we can express

Since C 0 (resp. C 1 ) is an (1 -α ) ε -cover of T b ( G \ { 1 } ) (resp. an αε -cover of T 1 ,b ), there exist ˜ T (1) , . . . , ˜ T ( b ) ∈ C 0 and ˜ δ ∈ C 1 such that

Then, we take ˜ T ∈ C where and we have

By using (4), we further bound each term

Combining these two inequalities, we conclude that, for any T ∈ T b ( G ) , there exists a ˜ T ∈ C such that ∥ ˜ T -T ∥ 1 ≤ αε +(1 -α ) ε = ε.

B.2 Controlling Bias and Variance

Like many the analysis of many estimators this proof has two pain parts, analysis of the bias, and the analysis of the variance.

B.2.1 Variance

To control the variance we will use the follow bound on covering numbers

Proposition B.6. For any b ∈ N , any 0 < ε < 1 and any graph G whose number of vertices is d and resilience is r , we have

Proof of Proposition B.6. We first assume that G is a connected graph and prove that

Note that the exponent in the factor d r is r as opposed to r +1 in the original statment. We will prove the statement by induction on r .

Base case r = 1 : Note that the graph G is a graph with only one vertex. By Lemma B.2, we have

thereby satisfying (5).

Inductive step r > 1 : Let v be the vertex selected in the first disintegration step of the r -disintegration and G 1 , . . . , G t be the connected components after removing v from G . Also, for all i ∈ [ t ] , let r i = r ( G i ) and d i be the number of vertices in G i . By Lemma B.5, we first have

Note that the resilience of each G i is at most r -1 (i.e. r i ≤ r -1 ) and hence we invoke the inductive assumption. We have

By plugging (8) into (7) and (6), we have

Now, we remove the assumption of G being connected. Suppose G has t connected components, G 1 , . . . , G t whose number of vertices is d i and resilience is r i . We have

B.2.2 Analysis of Bias

Theorem B.7. For any d, b ∈ N , any L > 0 , any graph G and any p ∈ D L ( G ) , we have

Proof of Theorem B.7. For any d, b ∈ N and A ∈ [ b ] d , recall that Λ d,b,A = ∏ d i =1 [ A i -1 b , A i b ) . Let λ A be the centroid of Λ d,b,A . We define

̸

where χ Λ d,b,A is the indicator, i.e. χ Λ d,b,A ( x ) = { 1 if x ∈ Λ d,b,A 0 if x / ∈ Λ d,b,A and Z is the normalizing factor, i.e. Z = ∫ x ∈ [0 , 1) d ˜ p ′ ( x ) dx .

̸

Since the construction requires Z = 0 , we now first show that when Z = 0 the statement holds trivially. When Z = 0 , we have

which implies

Hence, we further have

̸

Namely, L √ d/b ≥ 2 and therefore the statement holds for any p ′ ∈ U d,b ( T b ( G )) . Now, we assume that Z = 0 . We will prove the statement by arguing 1) ˜ p ∈ U d,b ( T b ( G )) and 2) ∥ p -p ∥ ≤ √ dL/b .

˜ 1

To prove 1) ˜ p ∈ U d,b ( T b ( G )) : For any partition V 0 ∪ V 1 ∪ V 2 of the vertex set of G such that V 0 separates V 1 and V 2 , we would like to show, for any x ∗ ∈ [0 , 1) d (without loss of generality, we reorder the indices such that x = ( x V 1 , x V 2 , x V 0 ) )

̸

where we use : to indicate integrating with respect to the corresponding variables, i.e.

and d 1 (resp. d 2 ) is the size of V 1 (resp. V 2 ). It is equivalent to show

We now analyze each of ˜ p ( x ∗ V 1 , x ∗ V 2 , x ∗ V 0 ) , ˜ p (: , : , x ∗ V 0 ) , ˜ p ( x ∗ V 1 , : , x ∗ V 0 ) , ˜ p (: , x ∗ V 2 , x ∗ V 0 ) . Let λ (resp. λ ∗ ) be the closest centroid to x for any x ∈ [0 , 1) d (resp. x ∗ ). For ˜ p ′ ( x ∗ V 1 , x ∗ V 2 , x ∗ V 0 ) , we have

Here, recall that we use : to indicate integrating with respect to the corresponding variables, i.e., for any centroid λ including λ ∗ ,

For ˜ p ′ (: , : , x ∗ V 0 ) , we have

Since, for any A ∈ [ b ] d , ˜ p ′ ( x V 1 , x V 2 , x ∗ V 0 ) = p ( λ V 1 , λ V 2 , λ ∗ V 0 ) for all ( x V 1 , x V 2 , x ∗ V 0 ) ∈ Λ d,b,A by (9), we have

By plugging it into the equation for ˜ p (: , : , x ∗ V 0 ) , we have

For ˜ p ′ ( x ∗ V 1 , : , x ∗ V 0 ) , we have

Similarly, for ˜ p ′ (: , x ∗ V 2 , x ∗ V 0 ) , we have

By using (11), (12), (13) and (14), we can conclude (10).

To prove 2) ∥ p -p ∥ 1 ≤ dL/b : We first express

Note that ˜ p ′ may not be a probability density.

For the first term ∥ p -˜ p ′ ∥ 1 , we have

For any x ∈ Λ d,b,A , we have ˜ p ′ ( x ) = p ( λ A ) by the definition of ˜ p ′ . Also, since p ∈ D L ( G ) , p satisfies the L -Lipschitz condition which implies

By plugging (17) into (16), we have

For the second term ∥ ˜ p ′ -˜ p ∥ 1 , we observe that ˜ p is simply a scaled version of ˜ p ′ by a factor of 1 /Z . Namely, we have

By (18) and the fact of ∥ p ∥ 1 = 1 , we further have √

By plugging (18) and (19) into (15), we have √

B.3 Proof of Main Theorems

With the previous results established we can now prove the core sample complexity results of this work.

Proof of Theorem 3.1. Recall that Lemma B.3 states the following. There is a deterministic algorithm that, given a collection of M distributions C = { p 1 , . . . , p M } , any 0 < ε < 1 and at least log(3 M 2 /δ ) 2 ε 2 i.i.d. samples drawn from an unknown distribution p , outputs an index j ∈ [ M ] such that

We will use the algorithm from Lemma B.3 by taking the collection C to be the ε -cover for T b ( G ) . Here, we determine b ∈ N later. By Proposition B.6, we have

and, by Theorem B.7, there exists a p ′ ∈ U d,b ( T b ( G )) such that

which implies there exists a p ′′ ∈ C such that

Namely, whenever n ≥ db r log( 2 d r +1 b ε ) ε 2 + log(3 /δ ) 2 ε 2 i.i.d. samples are drawn from p , the distribution q returned by the algorithm in Lemma B.3 satisfies

Now, by picking b = Θ( dL ε ) , the desired result follows.

Proof of Theorem 3.2. The proof is the same as in Theorem 3.1 except that we need to another ε -cover for ⋃ G ∈G T b ( G ) . For any G ∈ G , let C G be the ε -cover for T b ( G ) . By taking ⋃ G ∈G C G to be the ε -cover for ⋃ G ∈G T b ( G ) , we have

By Proposition B.6, each of | C G | is bounded above by ( 2 d r +1 b ε ) db r and, by considering all possible graphs, |G| is bounded above by 2 d 2 . In particular, when r = 1 , we have |G| = 1 . Hence, we have

and we can conclude the desired result by following the rest of the proof in Theorem 3.1.

C Proofs on Graph Resilience

For proofs in this section it will be implicit that graphs with a subscript, G i , are equal to ( V i , E i ) .

For proofs in this section we will introduce a useful concept we term a pre-disintegration of a graph .

Definition C.1. For any graph G = ( V, E ) , we define a pre-disintegration of G to be a tuple of subsets of V which satisfies the properties 1, 2, and, 4 in Definition 3.1 of disintegration, noting that some entries of a pre-disintegration may be the empty set, including the first entry.

In other words, the only difference between a pre-disintegration and a disintegration is that, in each step, we do not have to select a vertex in every connected component. The length of a pre-disintegration , D , is defined to be the largest r such that D r is nonempty and we call such pre-disintegration a r -pre-disintegration. We now have the following lemma on pre-disintegrations.

Lemma C.1. For any graph G , if there is a ℓ -pre-disintegration of G , then r ( G ) ≤ ℓ .

Proof of Lemma C.1. Let D be the set of all pre-disintegration of G . For any D ∈ D and any vertex v ∈ V where V is the vertex set of G , we define D -1 to be a function from V to N such that D -1 ( v ) is the index i where v ∈ D i . We will now define a partial order on D . For any D and D ′ in D , we say D ≤ D ′ if D -1 ( v ) ≤ D ′ -1 ( v ) for all v ∈ V . Namely, every vertex v appears no later in D than in D ′ . From the assumption, there exists a ℓ -pre-disintegration of G , ̂ D . Similarly to the definition of the length of a pre-disintegration, trailing empty sets are simply ignored to satisfy the antisymmetry property for a partial order. Hence, the set { D ∈ D | D ≤ ̂ D } is finite and there must exist a minimal element, D ∗ . By removing trailing empty sets from D ∗ , the length of D ∗ is no larger than ℓ .

̸

Now, we need to show D ∗ is a disintegration. Since D ∗ is a pre-disintegration, it satisfies the properties 1, 2, and, 4 in Definition 3.1. Namely, we need to check the properties 3 and 5. For 3, we prove it by contradiction. When D ∗ violates 3, there is a connected component G ′ of G such that no v ∈ V ′ , where V ′ is the vertex set of G ′ , is in D ∗ 1 . Let v ′ be arg min v ∈ V ′ D ∗-1 ( v ) and i ′ be D ∗-1 ( v ′ ) , i.e. the earliest vertex in V ′ appears in D ∗ . We construct a new pre-disintegration ̂ D ∗ by setting ̂ D ∗ = D ∗ except that ̂ D ∗ 1 = D ∗ 1 ∪{ v ′ } and ̂ D ∗ i ′ = D ∗ i ′ \ { v ′ } , i.e. we move v ′ to the first step. Hence, we have ̂ D ∗ ≤ D ∗ and ̂ D ∗ = D ∗ which contradicts the minimality of D ∗ . We can perform the identical argument to show 5.

Equipped with Lemma C.1, we can now commence with the proofs from the main text.

Proof of Lemma 4.1. We will assume V 1 , . . . , V m contain distinct elements for convenience. We now prove the statement by showing the following two inequalities: 1) max i ∈ [ m ] r ( G i ) ≥ r ( ⊕ i ∈ [ m ] G i ) and 2) max i ∈ [ m ] r ( G i ) ≤ r ( ⊕ i ∈ [ m ] G i ) .

For 1) max i ∈ [ m ] r ( G i ) ≥ r ( ⊕ i ∈ [ m ] G i ) : Let r be the maximum of the graph resilience of G i , i.e. r = max i ∈ [ m ] r ( G i ) , and D ( i ) be a r ( G i ) -disintegration for i ∈ [ m ] . We construct a r -tuple D by setting the j -th step to be D j = ⋃ i ∈ [ m ] D ( i ) j . It is easy to check that D satisfies Definition 3.1 and hence D is a r -disintegration of ⊕ i ∈ [ m ] G i . By Definition 3.2, the inequality r ≥ r ( ⊕ i ∈ [ m ] G i ) follows.

For 2) max i ∈ [ m ] r ( G i ) ≤ r ( ⊕ i ∈ [ m ] G i ) : Let r be the graph resilience of ⊕ i ∈ [ m ] G i , i.e. r = r ( ⊕ i ∈ [ m ] G i ) , and D be a r -disintegration of ⊕ i ∈ [ m ] G i . We construct m tuples D (1) , . . . , D ( m ) by setting

It is easy to check that, for each j ∈ [ m ] , D ( j ) satisfies Definition C.1 and hence D ( j ) is a r -predisintegration of G j . By Definition 3.2 and Lemma C.1, we have r ( G i ) ≤ r for all i ∈ [ m ] and hence the inequality max i ∈ [ m ] r ( G i ) ≤ r follows.

Proof of Lemma 4.2. We will prove for the case where | V ′ | = 1 . Note that the lemma statement follows from repeated application of this case.

We will proceed by contradiction. Suppose there exists G and v such r ( G ) ≥ r ( G \ { v } ) + 2 . Let ˜ D be a r ( G \ { v } ) -disintegration of G \ { v } . If we define a tuple D with D 1 = v and D i +1 = ˜ D i for all i , it is clear that D is a ( r ( G \ { v } ) + 1) -pre-disintegration of G . The contradiction follows from the application of Lemma C.1.

Proof of Lemma 4.3. We will prove for the case where | E ′ | = 1 , the lemma follows from repeated application of this case. Without loss of generality let E ′ = (1 , 2) . Let D be an r -disintegration of G . Let ˜ D be a ( r +1) -tuple with ˜ D 1 = { 1 } and ˜ D i = D i -1 \ { 1 } for i ∈ { 2 , . . . , r +1 } . Then ˜ D is an pre-disintegration of G ′ with length r +1 . This case follows from Lemma C. 1 .

Proof of Lemma 4.4. Let D be a r ( G ) -disintegration. Let ˜ D be the sequence of subsets of V ′ when one removes the elements of in V \ V ′ from D . Clearly, ˜ D is a pre-disintegration of G ′ , with length at most r ( G ) . The lemma follows from application of Lemma C.1.

Proof of Lemma 4.6. It is clear that when G with d vertices is a complete graph that r ( G ) = d . For the reverse direction, suppose that G is a graph with V = [ d ] with no edge between d -1 and d . We can define a pre-disintegration where D i = { i } for i ∈ [ d -2] and D d -1 = { d -1 , d } . The lemma follows from application of Lemma C.1.

Proof of Lemma 4.7. When E = ∅ clearly D 1 = V is a disintegration.

̸

If E = ∅ then there must exist two vertices in some component so D 1 = V does not satisfy the component property of a disintegration, thus D 2 = ∅ for any disintegration of G .

̸

Proof of Lemma 4.8. One can simply choose the center vertex as the first entry of a disintegration and the remaining vertices as the rest.

Proof of Lemma 4.9. We construct a r -tuple D as follows. For all i ∈ [ r ] , set D i to be the set of all tree vertices at the i -th level in G . Here, we define the root node to be at the first level. It is easy to check that D a r -disintegration. By Definition 3.2, we have r ( G ) ≤ r .

Proof of Lemma 4.10. Let V be the vertex set. For any vertex v ∈ V , let α v be the maximum number of vertices in the connected components after removing v . We would like to show that there exists a vertex v ∗ such that α v ∗ ≤ ⌊ d 2 ⌋ . We prove it by contradiction. Let v ′ be arg min v ∈ V α v and V 1 , . . . , V t be the vertex sets of the connected components after removing v ′ . We have α v ′ > ⌊ d 2 ⌋ which means that there is a connected component whose number of vertices is strictly larger than ⌊ d 2 ⌋ . Note that there is only one such component since the sum of the number of vertices of other components is strictly less than d -1 -⌊ d 2 ⌋ . WLOG, let V 1 be the vertex set of that component. Let v ′′ be the vertex in V 1 sharing an edge with v ′ . We want to argue that α v ′′ < α v ′ which contradicts the definition of v ′ . To see this, we examine the number of vertices in the connected components after removing v ′′ . We use the fact that G is a tree. For the component whose vertex set is ∪ t i =2 V i ∪ { v ′ } , the size is strictly less than ( d -1 -⌊ d 2 ⌋ ) + 1 = d -⌊ d 2 ⌋ ≤ α v ′ . For any component inside V 1 , the size is less than ⌊ d 2 ⌋ -1 < α v ′ . Namely, we have α v ′′ < α v ′ . Therefore, there exists a vertex v ∗ such that α v ∗ ≤ ⌊ d 2 ⌋ .

Now, we remove v ∗ in the first step and all the connected components, G 1 , . . . , G t , has at most ⌊ d 2 ⌋ vertices. By induction, we have

Proof of Proposition 4.11. We will prove the statement by induction on s .

Base case s = 1 : When s = 1 , we observe that L t t (2 1 -1) = L t t is a complete graph with t vertices. By Lemma 4.6, we have r ( L t t ) = t ≤ 1 · t .

Inductive step s > 1 : Let V ′ ⊂ V be { t (2 s -1 -1) + i | i ∈ [ t ] } . Note that | V ′ | = t . By Lemma 4.2, we have

In other words, we first remove the "middle" t vertices from the graph. By the definition of path graphs (Definition 4.2), it is easy to check that there is no edge between i and j for i = 1 , . . . , t (2 s -1 -1) and j = t (2 s -1 -1) + t +1 , . . . , t (2 s -1) . Therefore, there are two connected components and clearly each of them is isomorphic to L t t (2 s -1 -1) . Let G 1 and G 2 be the two connected components, i.e. L t (2 s -1) \ V ′ = G 1 ⊕ G 2 . Then, we have

Proof of Corollary 4.12. Let s be ⌈ log 2 ( d t +1) ⌉ . By the definition of path graphs (Definition 4.2), we have L t d ≤ L t t (2 s -1) (recall that ≤ means being isomorphic to a subgraph). Hence, we have

Proof of Proposition 4.13 . We will prove the statement by induction on s .

Base case s = 1 : When s = 1 , we observe that L t t (2 1 -1) × t (2 1 -1) = L t t × t is a complete graph with t 2 vertices. By Lemma 4.6, we have r ( L t t × t ) = t 2 ≤ 4 t 2 2 1 .

Inductive step s > 1 : Let d ′ be t (2 s -1 -1) . Let V ′ ⊂ V be

Note that | V ′ | = t 2 +4 t 2 d ′ ≤ 4 t 2 2 s -1 . By Lemma 4.2, we have

In other words, we first remove the vertical and horizontal "middle" stripes from the graph. By the definition of grid graphs (Definition 4.3), it is easy to check that there is no edge between ( i 1 , j 1 ) and ( i 2 , j 2 ) if one of the followings happens:

d

′

=

t

(2

s

-

1

-

1)

d

′

=

t

(2

s

-

1

-

1)

t

d

′

=

t

(2

s

-

1

-

1)

d

′

=

t

(2

s

-

1

-

1)

t

G

3

G

4

G

1

G

2

V

′

V

′

V

′

V

′

V

′

i 1 = 1 , . . . , t (2 s -1 -1) and i 2 = t (2 s -1 -1) + t +1 , . . . , t (2 s -1)

Therefore, there are four connected components and clearly each of them is isomorphic to L t t (2 s -1 -1) × t (2 s -1 -1) . Let G 1 , G 2 , G 3 , G 4 be the four connected components, i.e. L t t (2 s -1) × t (2 s -1) \ V ′ = G 1 ⊕ G 2 ⊕ G 3 ⊕ G 4 . See figure 6 for the graphical illustration. Then, we have

Proof of Corollary 4.14. Let s be ⌈ log 2 ( k t +1) ⌉ . By the definition of grid graphs (Definition 4.3), we have L t k × k ≤ L t t (2 s -1) × t (2 s -1) (recall that ≤ means being isomorphic to a subgraph). Hence, we have

Proof of Lemma 4.15. Wewill assume V 1 , . . . , V k all contain m vertices and for all i for all v, v ′ ∈ V i , v is adjacent to v ′ (all of the blocks are fully connected). We will prove the lemma for this case, the lemma statement then follows from application of Lemma 4.4.

We will denote the vertices of V i as v i, 1 , . . . , v i,m . Let D ′ be a t -disintegration of G ′ . Let D be a t × m -tuple, where D i,j = { v i,j | i ∈ D i , j = j } . If one considers D in lexicographical order, then it is clearly a pre-disintegration of G with length mt . This case then follows from Lemma C.1.

D Optimality of Rates with Respect to d and r

Consider a fixed dimension d and resilience r . Let G be a graph where vertices 1 , . . . , r form a complete subgraph, and vertices r +1 , . . . , d are isolated (not adjacent to any other vertex). Let L > 0 , and suppose we have an estimator that can estimate densities in D L ( G ) at rate O ( n -1 / (2+ r -ε ) ) for some ε > 0 . Let p ∈ D r,L and X 1 , . . . , X r ∼ p . We will construct an estimator for p using the aforementioned estimator. Define ˜ X 1 , . . . , ˜ X d ∼ ˜ p as follows:

where ( ˜ X 1 , . . . , ˜ X r ) and ( ˜ X r +1 , . . . , ˜ X d ) are independent.

Note that ˜ p ∈ D L ( G ) . Let ˆ p be an estimator for ˜ p such that

Let L denote the law of a random vector. It is well known that for any function f from one Euclidean space to another, ∥L ( f ( X )) -L ( f ( Y )) ∥ 1 ≤ ∥L ( X ) -L ( Y ) ∥ 1 (see Devroye and Lugosi (2001), Section 5.4). We will use f : ( x 1 , . . . , x d ) ↦→ ( x 1 , . . . , x r ) . Let Y 1 , . . . , Y d ∼ ˆ p . Using L (( Y 1 , . . . , Y r )) as an estimator for p , we have:

This shows we can estimate all densities in D r,L at rate O ( n -1 / ( r +2 -ε ) ) , while the known optimal rate is O ( n -1 / ( r +2) ) . Thus, for any r and d , there exists a graph with d vertices and resilience r such that estimation of densities in D L ( G ) can be no faster than O ( n -1 / ( r +2) ) .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided the full proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: We do not have experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: We do not have experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: We do not have experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We do not have experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: We do not have experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our result is a theoretical result.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposed a new summary of graph, called graph resilience, which measures the complexity of density estimation. This concept is different from more its well-known counterparts, including rank, sparsity, manifold dimension etc. Quite a few concrete examples of graph along with their resilience are given. Both the concept and the theory are sound, the contribution to the literature of density estimation is solid.

STRENGTHS:
The concept of resilience is novel. The definition is clear, esp Figure 3. 
Section 4 is very well-written, which helps the audience a lot to understand the concept of resilience.

WEAKNESSES:
More intuitions are expected, to define $r$. The concept is beautiful, but I am curious how the authors come up with the definition. Is there is any clear intuition to define it, or the authors start from the proof of the density estimation rate and then identify this key quantity?

QUESTIONS:
Given an arbitrary graph, is there any concrete computational algorithm to calculate the exact resilience? If so, what’s the computational cost? I suppose the exact calculation is very expensive, that’s why the authors focused more on how to bound it.

LIMITATIONS:
Can the authors discuss some possible extensions? For example, what does the resilience look like for directed graphs, or for non-Markovian graphs?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of estimating a multivariate density, assuming that the density is Markov w.r.t an underlying undirected graph $G$. It is shown that the sample complexity for estimating such a density scales with the resilience of the graph, as opposed to  the dimension d. Several examples of G are provided for which the resilience can be exactly computed or bounded.\n\nSTRENGTHS:\n1. The paper is written very well with a clear exposition which makes it easy to follow. The problem is well defined and motivated, along with clear notation throughout. The related work is also thorough and provides a good overview of the literature.\n\n2. The results are to my knowledge novel, I am not aware of similar results for this problem setting. I believe the paper would be of interest for researchers in high dimensional statistics, and related problems.\n\nWEAKNESSES:\n1. It would have been helpful to provide a proof-sketch of the main theorems in the main text, this is particularly relevant here given that this is a theoretical paper. Moreover, I think it is important to outline the algorithm in the main text and not relegate it to the appendix. To handle space constraints, I think parts of Section 4 could be moved to the appendix as they are relatively less important as opposed to the aforementioned details.\n\n2. It would have also been useful to provide some simulation results on synthetic data as this would have helped empirically validate the theoretical results.\n\nQUESTIONS:\n1. What is the running time of the algorithm w.r.t n, d and r? This is important to discuss in the main text and is related to the point I had raised earlier.\n\n2. The density is assumed to be Lipschitz for the analysis, but can the results be extended to handle higher order smoothness (s times continuous differentiability)?\n\n3. Can something be said about the optimality of the bounds? In particular, is the graph resilience the right quantity for this problem? \n\n4. In Corollary 4.12, why not show the dependence on t explicitly in the bound?\n\nMinor remarks:\n\n- I think there is a typo towards the end of line 182. Another typos in line 252 (\"... tend to have ...\")\n\n- In the statement of Theorem 3.1, it might be good to specify that $G$ is known.\n\nLIMITATIONS:\nI do not see a dicussion on limitations in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

comments: SUMMARY:
The authors, through the formalism of graph models for probability density functions, identify the “graph resilience” as a key quantity to estimate the sample complexity of computing the density.

STRENGTHS:
The work is original and technically sound, claims are generally well supported (with one exception that I will mention later on). 

The ideas of graphs resilience and disintegration are interesting and are nicely defined.

WEAKNESSES:
The only claim that I consider not well supported is related to lines 160-161, more concretely to the lower dimensional manifold. In order to support this claim, the authors should prove this claim by finding an example where the resilience r is lower than the dimension d and that cannot be mapped into a lower dimensional manifold.    

The paper is not clearly written and in a way that only specialists in the field of graph modelling can understand. The relationship with classical methods like kernel density estimation is not clear.  

No practical applications are shown.  

From certain point of view, the novelty is not high: The authors shown that, for structured data, the Markov property is enough to simplify the complexity of density estimation. However, the structure in the data is, by itself, a correlation measure, and therefore, it is not surprising that it reduces the effect of the curse of dimensionality.

QUESTIONS:
Why is the manifold assumption violated in structured data if the effective dimension is lower than the embedding dimension? 

Are you willing to provide a code that allows to use your findings in practical data?

LIMITATIONS:
No limitations are discussed. Aspects like the limited application field and the difficulties when trying to use the findings in real data should be addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13