Title: Hardness of Learning Neural Networks under the Manifold Hypothesis
Venue: NeurIPS 2024
Accepted: yes
Keywords: Manifold hypothesis

Learning theory

Hardness of learning

Neural network hardness

Manifold Learning
Abstract: The manifold hypothesis presumes that high-dimensional data lies on or near a low-dimensional manifold. 
While the utility of encoding geometric structure has been demonstrated empirically, rigorous analysis of its impact on the learnability of neural networks is largely missing. Several recent results have established hardness results for learning feedforward and equivariant neural networks under i.i.d. Gaussian or uniform Boolean data distributions. In this paper, we investigate the hardness of learning under the manifold hypothesis. We ask, which minimal assumptions on the curvature and regularity of the manifold, if any, render the learning problem efficiently learnable. We prove that learning is hard under input manifolds of bounded curvature by extending proofs of hardness in the SQ and cryptographic settings for boolean data inputs to the geometric setting. On the other hand, we show that additional assumptions on the volume of the data manifold alleviate these fundamental limitations and guarantee learnability via a simple interpolation argument. Notable instances of this regime are manifolds which can be reliably reconstructed via manifold learning. 
Looking forward, we comment on and empirically explore intermediate regimes of manifolds, which have heterogeneous features commonly found in real world data.

Hardness of Learning Neural Networks under the Manifold Hypothesis

Abstract

The manifold hypothesis presumes that high-dimensional data lies on or near a low-dimensional manifold. While the utility of encoding geometric structure has been demonstrated empirically, rigorous analysis of its impact on the learnability of neural networks is largely missing. Several recent results have established hardness results for learning feedforward and equivariant neural networks under i.i.d. Gaussian or uniform Boolean data distributions. In this paper, we investigate the hardness of learning under the manifold hypothesis. We ask which minimal assumptions on the curvature and regularity of the manifold, if any, render the learning problem efficiently learnable. We prove that learning is hard under input manifolds of bounded curvature by extending proofs of hardness in the SQ and cryptographic settings for Boolean data inputs to the geometric setting. On the other hand, we show that additional assumptions on the volume of the data manifold alleviate these fundamental limitations and guarantee learnability via a simple interpolation argument. Notable instances of this regime are manifolds which can be reliably reconstructed via manifold learning. Looking forward, we comment on and empirically explore intermediate regimes of manifolds, which have heterogeneous features commonly found in real world data. 4

1 Introduction

High-dimensional data is often thought to have low-dimensional structure, which may stem, for instance, from symmetries in the underlying system. This observation has given rise to the manifold hypothesis , which presumes that high-dimensional data lies on or near a low-dimensional manifold. Empirical studies have confirmed this hypothesis across domains [20, 77, 37, 65]. A plethora of algorithms for geometric data analysis [44, 8, 80, 21] and, more recently, machine learning [30, 15, 23] seek to leverage such structure. While these methods have shown empirical promise, few formal results on the benefits of encoding geometric structure have been established. Here, we study the impact of the manifold hypothesis on the computational complexity of learning algorithms for neural networks. Specifically, we investigate, under which geometric assumptions feedforward neural networks are efficiently learnable.

The computational hardness of learning shallow, fully-connected neural networks under i.i.d. Gaussian data distributions has been established by [36, 26, 48, 32], who show that the complexity of learning even single hidden layer neural networks grows at least exponentially with the input size. Some of this analysis utilizes the correlational statistical query (CSQ) framework, of which learnability via gradient decent is a notable instance. Recently, [57] have shown similar hardness results

for equivariant neural networks, a class of geometric architectures that explicitly encode symmetries. Both lines of work indicate that additional assumptions on the neural network architecture or the data geometry are needed to establish learnability. Here, we focus on the latter.

A separate body of literature investigates nonlinear, low-dimensional structure in data. Approaches for estimating intrinsic dimension [44, 63, 87] and curvature [1, 86, 46] from data seek to characterize the geometry of low-dimensional manifolds. Manifold Learning aims to identify lowdimensional structure by reconstructing low-dimensional manifolds from data. Methods such as Multi-Dimensional Scaling [62], Isomap [89] or Diffusion Maps [31] have shown empirical success in learning low-dimensional data manifolds. For some of these approaches, formal guarantees on their reliability with respect to the geometric characteristics and sample size of the data are known [9]. More generally, the manifold learning problem and its complexity have been formally studied in [43, 42, 3]. However, the impact of data geometry on the complexity of learning neural networks in downstream tasks remains open.

In this paper, we ask: Which assumptions on the data geometry guarantee learnability of neural networks? We show that the manifold hypothesis on its own does not guarantee learnability. In particular, we give hardness results for a class of low-dimensional manifolds in the statistical query (SQ) model or under cryptographic hardness assumptions. Our work follows an established line of proof techniques in the SQ literature [32, 48, 26], extending hardness results for neural network training in the Boolean and Gaussian input models to more general geometries. We further show that additional assumptions on the volume and curvature of the data manifold alleviate the fundamental limitations and guarantee learnability through a rather simple interpolation argument. In particular, manifolds which can be reliably reconstructed via manifold learning are in this regime. We further discuss geometric regimes in which our results do not directly apply, and in which provable learnability remains an open question. We illustrate our learnability results through computational experiments on neural network training in the learnable and provably hard regimes. We further complement our theoretical analysis with a brief computational study of the intrinsic dimension of image data manifolds, with the goal of testing the geometric assumptions in our framework.

2 Background

2.1 Basic Notation

We denote scalars, vectors, and matrices as v, v , and V respectively. We consider ambient spaces R n equipped with the usual inner product ⟨· , ·⟩ and associated ℓ 2 norm ∥·∥ . Submanifolds M⊂ R n considered in this study have intrinsic dimension d ≤ n and are compact and connected (unless otherwise stated). The tangent space T p M at a point p ∈ M denotes the d -dimensional linear subspace of R n spanned by velocity vectors of smooth curves incident at p . Given a subset S ⊂ R n , we denote by d ( z , S ) = inf p ∈ S ∥ z -p ∥ the distance of a point z to S . We denote Vol d as the d -dimensional volume measure inherited from the d -dimensional Hausdorff measure and denote ω d ( r ) and σ d ( r ) as the volume of the d -dimensional ball and d -dimensional sphere of radius r respectively. For a point p ∈ M in a given manifold M , Vol M ( p , r ) denotes the volume of the ball of radius r around the point p with respect to the Riemannian distance metric.

2.2 Learning Setting

We consider the task of learning feedforward neural networks f : R n → R composed of L hidden layers f ( ℓ ) : R d ℓ → R d ℓ -1 taking the form

where v ∈ R d L , W ℓ ∈ R d ℓ × d ℓ -1 and b ∈ R d ℓ are trainable weights. The input dimension d 0 is set to the ambient dimension n and the output is scalar-valued. Throughout we will consider the setting where the weight entries and hidden widths are bounded by O (poly( n )) . When guaranteeing that a class of networks is learnable, we will assume that the number of layers L = O (1) is constant with respect to the input dimension. Our formal hardness results will apply for single hidden layer networks ( L = 1 ).

We consider learnability in the distribution-specific probably approximately correct (PAC) setting where the goal is to produce an algorithm which can learn a target function given polynomial time and samples [68].

Definition 2.1 (Efficiently PAC Learnable) . A concept class C consisting of functions c : X → Y is efficiently PAC-learnable over distribution D on X ⊆ R n , if there exists an algorithm such that for any ϵ > 0 and δ > 0 , and for any target concept c ∗ ∈ C , the algorithm takes at most m = poly(1 /ϵ, 1 /δ, n ) samples drawn i.i.d. from ( x , c ∗ ( x )) with x ∼ D , and returns a function f satisfying ∥ f -c ∗ ∥ D := √ E x ∈D [( f ( x ) -c ∗ ( x )) 2 ] ≤ ϵ with probability at least 1 -δ in time at most poly(1 /ϵ, 1 /δ, n ) .

Note, that the above is a distribution specific instance of PAC learning as we require the algorithm to work only for a given distribution and not all distributions. In Section 3.2, we will also show a hard class of functions which is likely not efficiently PAC learnable. Hardness results are proven in the restricted statistical query (SQ) setting, a query complexity based model for proving hardness capturing most algorithms in practice [55, 78]. Given a joint distribution D on input/output space X × Y , any SQ algorithm is composed of a set of queries. Each query takes as input a function g : X × Y → [ -1 , 1] and tolerance parameter τ > 0 , and returns a value SQ( g, τ ) in the range:

Gradient based algorithms can be queried by, for example, setting g ( x , y ) = C ∂ ∂θ (NN θ ( x ) -y ) 2 to estimate the gradient of a neural network NN θ with respect to parameter θ for the MSE loss (constant C chosen so that outputs of g are bounded in [ -1 , 1] forming a valid query). Hardness is quantified in the number of queries needed to learn a function c ∗ drawn from function class C .

Manifold smoothness restrictions. To conform to practical settings where input features are normalized (e.g. image pixel values in the range [0 , 1] ), input distributions are assumed to be supported on smooth d -dimensional sub-manifolds M ⊂ [0 , 1] n of the n -dimensional hypercube. For any given manifold M , we will assume that data distributions D M supported on that manifold have a smooth density f with respect to the volume measure and are appropriately bounded such that ρ max := max x ∈M f ( x ) = O (poly( n )) .

We will also place curvature restrictions on the manifold by bounding its reach, a global smoothness quantity introduced by [41] and commonly studied in the manifold learning community [1, 3, 43, 47]. Informally, it is the largest number D , such that any point in ambient space at distance less than D has a unique nearest neighbor in the manifold M . It is defined more formally from descriptions of the medial axis (Figure 1).

Definition 2.2 (Reach from medial axis) . Given a closed subset S ⊂ R n , the medial axis Med( S ) of S consists of the set of points with no unique nearest neighbor:

̸

The reach Rch( S ) of S is the minimal distance from S to Med( S ) :

Bounds on the reach imply corresponding bounds on the radius of curvature (related to the geodesic and normal curvature) at any point in the manifold since Rch( M ) -1 ≥ ∥ γ ′′ γ ( t ) ,γ ′ ( t ) ( t ) ∥ for any unit-speed geodesic γ : R → M where ∥ γ ′ γ ( t ) ( t ) ∥ = 1 for all t ∈ R [3]. We will encounter a second classical curvature notion, Ricci curvature , which is a local, intrinsic curvature notion

Medial Axis

that characterizes the volume growth of geodesic balls (see sec. B.1 for a more formal definition). Positive lower bounds on the Ricci curvature imply that the manifold has a bounded diameter, a fact that we will use below.

2.3 Related works

Here, we briefly summarize the most relevant prior work to our study and reserve Appendix A for a more detailed discussion. Our study lies at the intersection of research in manifold learning complexity and neural network learnability. In manifold learning, various works have analyzed the complexity of learning tasks over input manifolds. In one line of work, estimation of smooth manifolds in Hausdorff loss has been shown to require sample complexity of O ( ϵ -d/ 2 log(1 /ϵ )) , which is independent of the ambient dimension n [14, 47, 58]; later works also provided algorithms that run in time linear in n [4, 38]. In a learning setting similar to our work, [3] provide upper and lower bounds on the complexity of manifold reconstruction in SQ settings. In a separate context, [71, 43] show that the sample complexity for determining whether a dataset is within a class of manifolds of specified intrinsic dimension, curvature, and volume bounds grows exponentially with the intrinsic dimension and polynomially in the reach and volume. [71] also categorizes the sample complexity for binary classification over smooth cuts on a data manifold where smoothness is defined by the condition number of the classification boundary of the manifold (a quantity closely related to reach).

The hardness of learning neural networks has a long history that we detail further in Appendix A. Under the i.i.d. Gaussian input model, superpolynomial [48] and exponential [36] lower bounds for learning single hidden layer networks in CSQ settings have been shown. For uniformly random Boolean inputs, [32] reduce the problem of learning single hidden layer neural networks to a cryptographically hard problem, a technique, which we also use in Appendix C.3. They also show how to 'Booleanize' Gaussian inputs to show hardness for three hidden layer networks, which was later extended to SQ and cryptographic hardness of learning two hidden layer networks in [26]. [57] used similar techniques to give hardness results for equivariant neural networks. To the best of our knowledge, the hardness of learning feedforward neural networks under more general data geometries, such as under the manifold hypothesis, has not been studied previously. Various works have found efficient learning algorithms under i.i.d. input assumptions when the weights of the networks are restricted in their rank, condition number, positivity, and other criteria [90, 27, 7, 36]. We consider the generic setting where weight matrices are bounded in width and magnitude by O (poly( n )) , but otherwise unrestricted.

3 Learnability results

Our main results prove the existence of a learnable and hard to learn class of input data manifolds illustrated in Figure 2. The learnable setting is the class of efficiently sampleable manifolds, which form the basis for algorithms that can provably reconstruct manifolds [9, 28, 70, 42]. As expected, we find that a simple interpolation argument guarantees learnability of neural networks over these manifolds. Below and in Appendix B.3, we comment on instances of data geometries commonly assumed in machine learning and data science applications that place manifolds within this regime.

The provably hard setting are manifolds without bounds on volume but with bounds on curvature quantified globally by the reach. When the bound on the reach grows no faster than o ( √ n ) ( n denoting the input size), we construct input data manifolds that feature curvature no larger than the stated bounds but for which learning neural networks is exponentially hard. Our proofs construct curves that cover exponentially many quadrants of the Boolean cube, which allows us to extend classical hardness results for learning Boolean functions expressible by neural networks.

Within the manifold regimes we study, our results are relatively tight with respect to bounds on the reach, since whenever the reach grows as ω ( √ n ) , such manifolds fall within the first class of efficiently sampleable manifolds. We leave as an open question the learnability of manifolds whose reach grows exactly as Θ( √ n ) (see Appendix C.4 for more details). Looking beyond our setting, real-world data manifolds feature heterogeneous properties that may not conform to the global bounds on curvature, intrinsic dimension, etc. that we set in our study. Better characterizing these heteregeneous features is a first step to extending our learnability results to more realistic settings. We conduct some preliminary empirical analysis in analyzing this heterogeneity in our experiments (Section 4.2).

(Learnable)

Efficiently Sampleable Regime

Example: Positive Bounded Ricci Curvature Manifold

(Potentially Learnable) Heterogeneous Regime

Example: Manifolds with varying intrinsic dimension (e.g. as in real world data)

(Provably Hard)

Bounded Curvature, Unbounded Volume Regime

Example: Space filling manifold

?

3.1 Sampleable regime

In the manifold reconstruction literature, the goal is to draw samples from a distribution P (typically supported on a given manifold M ) and find a manifold M ′ which closely approximates the sample distribution or target manifold in some appropriate error metric such as the Haussdorf distance [89, 9, 2, 43, 28, 70, 42]. Runtime and sample complexity for these algorithms are generally at least linear in the volume of the manifold, polynomial in smoothness parameters such as the reach, and exponential in the intrinsic dimension, though the dependence on the ambient dimension can often be removed [42]. Manifold reconstruction algorithms offer a direct means for learning data from a target neural network f ∗ as one can apply such algorithms to the graph of the function consisting of points ( x , f ∗ ( x )) ∈ X × R . As we will show here, guarantees of learning in the manifold reconstruction literature directly imply guarantees for learning neural networks via a simple interpolation argument. However, caution must be taken in assuming this situation holds in practice. For this procedure to be efficient, the volume of the manifold should be at worst polynomial in the intrinsic dimension d and not growing significantly with the ambient dimension n . Some empirical evidence suggests that manifolds of real-world data may not be in this regime [74, 16].

Essential to many of the manifold reconstruction algorithms is the requirement that one can efficiently cover the manifold with samples [28, 43, 9]. This requirement comes in various technical forms and we frame our results here assuming the ability to form an epsilon net with samples.

Definition 3.1 ( ( ϵ, δ ) -net) . Given a distribution D M over a manifold M , a subset S ⊂ M of points forms an ( ϵ, δ ) -net if with probability at least 1 -δ over draws x ∼ D M , there exists a point x ′ ∈ S where ∥ x -x ′ ∥ ≤ ϵ .

We denote a sequence of manifolds indexed by ambient dimensions as efficiently sampleable if for a fixed intrinsic dimension d , one can draw polynomially many samples in n and error 1 /ϵ to form an epsilon net over the manifold M .

Definition 3.2 (Efficiently sampleable manifold) . Let {M n } denote a sequence of manifolds in ambient dimension n with fixed intrinsic dimension d = O (1) . Let {D M n } be a corresponding sequence of distributions over points on the manifolds. We denote this sequence of manifolds as efficiently sampleable if with at most ˜ O (poly( n, 1 /ϵ )) samples drawn i.i.d. from D M n , one can form an ( ϵ, δ ) -net of the manifold M n for any δ = Ω(poly( n ) -1 ) .

Note that classical manifold learning algorithms typically implicitly assume efficiently sampleable manifolds as they assume that the algorithms run efficiently or have runtimes that depend polynomially on the manifold's volume Vol( M ) (see Example B.8 for details). With standard regularity assumptions on the manifold, such a volume assumption typically implies that an epsilon net can be formed efficiently with ˜ O (Vol( M ) /ϵ d ) samples by a coupon collector argument (see, e.g., Proposition 3.5).

Remark 3.3 . Given a target function f ∗ with inputs on a manifold M⊆X , we could treat the graph of the function M f ∗ = { ( x , f ∗ ( x )) : x ∈ M} as a manifold in X × R . Then, one can apply manifold reconstruction algorithms directly to the graph M f ∗ , which lies in an ambient space of

dimension n + 1 . This likely works in practice, though it runs into some technical issues due to discontinuities introduced by ReLU activations. We apply a more direct method here.

Proposition 3.4. Let n -dimensional inputs be drawn from a sequence of efficiently sampleable manifolds M n with intrinsic dimension d = O (1) and distributions D M n over the manifold. Denote H n as the function class of constant depth ReLU networks on n inputs with weights bounded in magnitude by B = O (poly( n )) and O (poly( n )) width. Then, one can learn f ∗ ∈ H n up to error ϵ in runtime and sample complexity O (poly( n, B, 1 /ϵ )) .

Proof. From the efficiently sampleable property of the manifold, we form an ( ϵ ′ , δ ) -net using ˜ O (poly( n, 1 /ϵ )) many samples where for all but a δ -fraction of the manifold any point is within ϵ ′ of a given point within the net. Any network of L layers and O (poly( n )) bounded width and weight magnitude has the Lipschitz property that [92]

where B ′ = O (poly( n, B )) is a bound on the Frobenius norm of the weight matrices. Therefore, we can interpolate the value of f ∗ for any x ∈ M n from a sample by setting ϵ ′ = O ( ϵB ′-1 n -L ) . For the δ -fraction that is not in this net, note that | f ( x ) | ≤ O (poly( n )) for all x in the hypercube so setting δ = o ( | f ( x ) | -1 ) will guarantee that the contribution from these points decays. Finally, since L = O (1) and B = O (poly( n )) by assumption, we have the resulting polynomial complexity.

Restrictions on the curvature or convexity properties of a manifold can often render the manifold efficiently sampleable. To illustrate that many manifolds implicitly fall within the regime of efficiently sampleable manifolds, we give an example below. Additional examples can be found in Appendix B.3.

Proposition 3.5 (Bounded Ricci curvature (Isoperimetric setting, see [52, 17] for motivation) ) . The set of distributions P M over complete manifolds with bounded Ricci curvature Ric ( M ) ≥ ( d -1) K for a constant K > 0 5 are efficiently sampleable.

Proof. The bound on the Ricci curvature guarantees that the diameter of the manifold is at most 2 π/ √ K and the manifold is contained within a d -dimensional ball of radius 1 / √ K [69] (see also Theorem 6.3.3 of [75]). Thus, we can form an ( ϵ, δ ) -net over the manifold M by inducing it from an ϵ -cover on the ball of radius 1 / √ K (see Definition B.1 for definition). To achieve an ϵ -cover of such a ball in d dimensions, it suffices to have N ϵ = O (1 /ϵ d ) points (see, e.g., Lemma 5.2 of [91]). We denote the balls in the cover as b 1 , . . . , b N ϵ . Let τ denote a sampling ratio where we will take τ -2 = O (poly( n )) samples to form the ( ϵ, δ ) -net. By a coupon-collector argument (see Lemma B.5), τ -2 samples suffices to guarantee with high probability that any ball b i with probability at least P M ( b i ) ≥ τ has a sample within it. δ then is at most the probability that a randomly drawn sample falls within a ball b i with probability P M ( b i ) < τ . Therefore,

Thus, setting τ = δ/N ϵ suffices to achieve δ = Ω(poly( n ) -1 ) .

We remark that the efficiently sampleable setting does not necessarily require that the manifold be fully connected. In fact, it is straightforward to extend proofs of learnability such as in Proposition 3.5 to settings where there are multiple such disconnected manifolds as long as the number of disconnected components does not grow superpolynomially in n .

3.2 Hard regime with bounded curvature

The learnability of the networks in the manifold learning setting relied crucially on the fact that such settings implicitly place bounds on the volume of such manifolds. Here, we lift that restriction and consider a setting where manifolds are smooth and bounded only in their curvature and intrinsic dimension. More formally, we consider studying inputs drawn from manifolds with bounded reach (see Definition 2.2).

In this section, we construct a sequence of manifolds {M n } with sufficiently bounded reach that are exponentially hard to learn in the SQ model. These manifolds resemble space filling curves which wrap around exponentially many quadrants of the Boolean cube (see Appendix C.1). Given a bound on the reach Rch( M n } ) = O ( n α ) for α < 0 . 5 , the space filling curve wraps around 2 n 1 -2 α = 2 n Ω(1) quadrants allowing one to extend standard hardness results for learning Boolean functions to data supported on this manifold.

Theorem 3.6. Let D M n denote the uniform distribution over manifolds M n constructed in Definition C.4 where Rch( M n ) = O ( n α ) for α < 0 . 5 . Any SQ algorithm A capable of learning the class of linear width single hidden layer ReLU neural networks under this sequence of distributions up to mean squared error sufficiently small ( ϵ/ 8 suffices) with queries of tolerance τ must use at least Ω( τ 2 2 n Ω(1) ) queries.

Proof sketch. We first form manifolds conforming to the stated bounds on the reach and intrinsic dimension while also 'looping' around exponentially many quadrants of the hypercube. This is done by forming space-filling curves whose paths follow the indexing of a Gray code [51]. Inputs on this manifold are real-valued, but by carefully selecting a portion of the input dimensions, we obtain inputs that with high probability are approximately distributed as uniform over Boolean inputs { 0 , 1 } n b . We then extend previous proofs of the hardness of learning networks approximating parity functions under i.i.d. Boolean inputs proven in [32, 26] to our setting. See Appendix C for complete proofs.

Remark 3.7 . In Theorem C.12, we show an equivalent proof of hardness in the cryptographic setting following the techniques in [32]. Namely, in Appendix C.3, we show that the class of functions in Theorem 3.6 is also hard to learn conditional on cryptographic assumptions related to the hardness of learning a class of pseudorandom functions.

The above result is relatively tight with respect to the bound on the reach. Namely, whenever the reach Rch( M n ) = ω ( n α ) is growing faster than √ n , one can show that the volume of such manifolds is at most O (poly( n )) and fitting within the regime of efficiently sampleable manifolds studied in the previous subsection.

Proposition 3.8. Given a sequence of manifolds {M n } of intrinsic dimension d (fixed and independent of n ) and reach bounded by Rch( M n ) = ω ( n 0 . 5 ) , the volume of the manifolds grows at most Vol d ( M n ) = O (poly( n )) .

We refer the reader to Appendix C.4 for further details and formal proofs of this tightness.

4 Experiments

4.1 Empirical verification of main findings

To verify that neural networks are learnable in the sampleable regime (Section 3.1) and hard to learn in the bounded curvature regime (Section 3.2), we train neural networks over input data manifolds taken from these regimes to confirm the formal theoretical results. We consider target networks, which are single hidden layer neural networks of O ( n ) width and train overparameterized neural networks of larger width. For the sampleable regime, we draw inputs from a d -dimensional hypersphere supported over d orthogonal dimensions in the n -dimensional ambient space. This is an instance of a complete manifold of bounded curvature as in Proposition 3.5. For the hard to learn regime, we draw inputs from the 1-dimensional manifold constructed in Appendix C.1 with reach R = 0 . 5 . Target functions in this hard regime correspond to the parity functions described in our proofs in Appendix C.

0

20

40

60

80

100

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Test MSE

20

50

100

200

(a) Learnable Setting

0

5000

10000

15000

Step

0.0

0.2

0.4

0.6

0.8

1.0

Test MSE

10

20

50

(b) Hard to Learn Setting

The results in Figure 3 empirically confirm our main findings. When inputs are drawn from the d -dimensional hypersphere (Figure 3a), the trained neural network achieves low test error with a fixed training set of size 1000 for all n . Here, target functions are drawn from those in [36] who provided a class of hard to learn functions in the i.i.d. Gaussian input model. Once the input model is changed to the d -dimensional hypersphere, this class of functions is no longer hard to learn (see Appendix E for similar results over random targets).

In contrast, when inputs are drawn from the hard to learn manifold (Figure 3b), learning is only possible when the ambient dimension is small. Here, target functions are randomly chosen parity functions over the inputs (see Appendix E). At each step of training, we provide the algorithm with a fresh batch of i.i.d. samples. In Figure 8, we replicate these results when training networks overparameterized in depth (i.e. three hidden layers as opposed to one) as well. We refer the reader to Appendix E for further details.

4.2 Empirical study of geometry of data manifolds

Weempirically investigate the intrinsic dimension of real-world data manifolds as a first step towards testing the manifold regularity assumptions of our framework on real data. Largely, our results corroborate findings in other works highlighting the heterogeneous nature of real-world data manifolds [77, 87, 16].

Experimental Setup. We estimate intrinsic dimension using samples generated by a diffusion model, closely following the approach of [87]. The use of a diffusion model allows us to generate arbitrarily dense samples in a neighborhood of a given point from the data manifold. Given these samples, we perform PCA on the collection of score vectors { s i } of the diffusion model at each of these samples, which point towards the direction of de-noising and hence towards the manifold itself. Estimating the rank of such a collection of score vectors recovers an estimate of the normal and intrinsic dimensions. We defer a detailed description of our approach, its implementation, and hyperparameter choices to Appendix E.1.





















































































































































































Results. Estimated intrinsic dimensions for three image data manifolds are given in Table 3. Score spectra for the three manifolds are shown in Figure 4 confirming that the score vectors do indeed cluster in a linear subspace spanning the normal dimensions. Furthermore, our approach gives a similar estimate for the intrinsic dimension of MNIST as reported in [87]. To the best of our knowledge, FMNIST and KMNIST were not investigated in prior work. Comparing estimated intrinsic dimension across classes indicates significant heterogeneity of the real data manifolds, which would place them in the potentially learnable regime in our proposed framework. This may seem surprising as images drawn from different classes can have different properties and symmetries. However, it corroborates our initial hypothesis that real-world data may not conform to global bounds on intrinsic dimension. Furthermore, we find that upon re-sizing the images to a smaller scale, the ambient dimension decreases faster in comparison to the intrinsic dimension for some datasets (see Table 3 in Appendix) lending some limited evidence to the underlying assumption in the manifold hypothesis that bounds on intrinsic dimension do not grow in tandem with ambient dimension. Additional experimental results and further discussion can be found in Appendix E.3.

5 Discussion

In this paper we have investigated whether regularity assumptions on the data geometry can render feedforward neural networks efficiently learnable. We show that bounding curvature in lowdimensional data manifolds, a common assumption in high-dimensional learning, does not suffice to alleviate the fundamental hardness of learning such architectures. However, we establish learning guarantees under geometric assumptions common in the manifold reconstruction literature that allow manifolds to be efficiently covered with O (poly( n )) samples. Our results contribute to a recent body of literature that seeks to establish learnability guarantees and provable hardness results for neural network architectures [12, 26, 36, 57, 48, 90].

While our results establish guarantees for a wide range of data manifolds, including manifolds that can be provably reconstructed via manifold learning, there are geometric regimes in which the question of learnability remains open. We hypothesize that the heterogeneity of real data manifolds might place them in this more uncertain regime and find some evidence of this heterogeneity in our experiments. A further theoretical and empirical investigation of this class of data manifolds is an important direction for future work.

The experimental results on estimating the intrinsic dimension of data manifolds are merely a starting point for understanding realistic assumption on data geometry in machine learning architectures. In future work we hope to look at a wider range of synthetic and real data sets, as well as a wider range of geometric characteristics and estimation techniques.

Broadly, our results indicate that assumptions beyond global smoothness of data manifolds are needed to guarantee learnability of neural networks in real-world settings. To circumvent hardness results, the particular form of the networks and their weights can be changed to for example incorporate symmetries via equivariant architectures or place restrictions on weights such as bounds on condition number or rank [27, 7]. Furthermore, some datasets live in a discrete input space (e.g., language data) which does not conform directly to Riemannian analysis. In these settings, adaptations of the manifold hypothesis to analysis via graphs and other combinatorial objects presents an interesting direction for future work [81, 15].

References

E. Aamari, J. Kim, F. Chazal, B. Michel, A. Rinaldo, and L. Wasserman. Estimating the reach of a manifold. 2019.

E. Aamari and A. Knop. Statistical query complexity of manifold estimation. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 116-122, 2021.

E. Aamari and A. Knop. Adversarial manifold estimation. Foundations of Computational Mathematics , pages 1-97, 2022.

E. Aamari and C. Levrard. Stability and minimax optimality of tangential delaunay complexes for manifold reconstruction. Discrete & Computational Geometry , 59:923-971, 2018.

E. Abbe, E. B. Adsera, and T. Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory , pages 2552-2623. PMLR, 2023.

Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701 , 2021.

A. Bakshi, R. Jayaram, and D. P. Woodruff. Learning two layer rectified neural networks in polynomial time. In Conference on Learning Theory , pages 195-268. PMLR, 2019.

M. Belkin and P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation , 15(6):1373-1396, 2003.

M. Bernstein, V. De Silva, J. C. Langford, and J. B. Tenenbaum. Graph approximations to geodesics on embedded manifolds. Technical report, Citeseer, 2000.

R. L. Bishop. A relation between volume, mean curvature and diameter. In Euclidean Quantum Gravity , pages 161-161. World Scientific, 1964.

A. Blum, M. Furst, J. Jackson, M. Kearns, Y. Mansour, and S. Rudich. Weakly learning dnf and characterizing statistical query learning using fourier analysis. In Proceedings of the twenty-sixth annual ACM symposium on Theory of computing , pages 253-262, 1994.

A. Blum and R. Rivest. Training a 3-node neural network is np-complete. Advances in neural information processing systems , 1, 1988.

A. Bogdanov and A. Rosen. Pseudorandom functions: Three decades later. In Tutorials on the Foundations of Cryptography: Dedicated to Oded Goldreich , pages 79-158. Springer, 2017.

J.-D. Boissonnat and A. Ghosh. Manifold reconstruction using tangential delaunay complexes. In Proceedings of the twenty-sixth annual symposium on Computational geometry , pages 324333, 2010.

M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine , 34(4):18-42, 2017.

B. C. Brown, A. L. Caterini, B. L. Ross, J. C. Cresswell, and G. Loaiza-Ganem. Verifying the union of manifolds hypothesis for image data. In The Eleventh International Conference on Learning Representations , 2022.

S. Bubeck and M. Sellke. A universal law of robustness via isoperimetry. Journal of the ACM , 70(2):1-18, 2023.

S. Buchanan, D. Gilboa, and J. Wright. Deep networks and the multiple manifold problem. arXiv preprint arXiv:2008.11245 , 2020.

F. Camastra and A. Vinciarelli. Estimating the intrinsic dimension of data with a fractal-based method. IEEE Transactions on pattern analysis and machine intelligence , 24(10):1404-1407, 2002.

G. Carlsson. Topology and data. Bulletin of the American Mathematical Society , 46(2):255308, 2009.

L. Cayton et al. Algorithms for manifold learning . eScholarship, University of California, 2008.

M. Chen, K. Huang, T. Zhao, and M. Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. arXiv preprint arXiv:2302.07194 , 2023.

R. T. Chen and Y. Lipman. Riemannian flow matching on general geometries. arXiv preprint arXiv:2302.03660 , 2023.

S. Chen, S. Chewi, J. Li, Y. Li, A. Salim, and A. Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations , 2023.

S. Chen, Z. Dou, S. Goel, A. R. Klivans, and R. Meka. Learning narrow one-hidden-layer relu networks. arXiv preprint arXiv:2304.10524 , 2023.

S. Chen, A. Gollakota, A. Klivans, and R. Meka. Hardness of noise-free learning for twohidden-layer neural networks. Advances in Neural Information Processing Systems , 35:1070910724, 2022.

S. Chen and S. Narayanan. A faster and simpler algorithm for learning shallow networks. arXiv preprint arXiv:2307.12496 , 2023.

S.-W. Cheng, T. K. Dey, and E. A. Ramos. Manifold reconstruction from point samples. In SODA , volume 5, pages 1018-1027, 2005.

T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha. Deep learning for classical japanese literature, 2018.

T. Cohen and M. Welling. Group equivariant convolutional networks. In ICML , 2016.

R. R. Coifman and S. Lafon. Diffusion maps. Applied and Computational Harmonic Analysis , 21(1):5-30, 2006. Special Issue: Diffusion Maps and Wavelets.

A. Daniely and G. Vardi. From local pseudorandom generators to hardness of learning. In Conference on Learning Theory , pages 1358-1394. PMLR, 2021.

S. Dasgupta and Y. Freund. Random projection trees and low dimensional manifolds. In Proceedings of the fortieth annual ACM symposium on Theory of computing , pages 537-546, 2008.

V. De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.05314 , 2022.

L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

I. Diakonikolas, D. M. Kane, V. Kontonis, and N. Zarifis. Algorithms and sq lower bounds for pac learning one-hidden-layer relu networks. In Conference on Learning Theory , pages 1514-1539. PMLR, 2020.

J. J. DiCarlo, D. Zoccolan, and N. C. Rust. How does the brain solve visual object recognition? Neuron , 73(3):415-434, 2012.

V. Divol. Minimax adaptive estimation in manifold inference. Electronic Journal of Statistics , 15(2):5888-5932, 2021.

P. Erd˝ os and A. R´ enyi. On a classical problem of probability theory. Magyar Tud. Akad. Mat. Kutat´ o Int. K¨ ozl , 6(1):215-220, 1961.

E. Facco, M. d'Errico, A. Rodriguez, and A. Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports , 7(1):12140, 2017.

H. Federer. Curvature measures. Transactions of the American Mathematical Society , 93(3):418-491, 1959.

C. Fefferman, S. Ivanov, Y. Kurylev, M. Lassas, and H. Narayanan. Fitting a putative manifold to noisy data. In Conference On Learning Theory , pages 688-720. PMLR, 2018.

C. Fefferman, S. Mitter, and H. Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society , 29(4):983-1049, 2016.

K. Fukunaga and D. R. Olsen. An algorithm for finding intrinsic dimensionality of data. IEEE Transactions on computers , 100(2):176-183, 1971.

O. Ganea, G. B´ ecigneul, and T. Hofmann. Hyperbolic neural networks. Advances in neural information processing systems , 31, 2018.

N. Garcia Trillos and M. Weber. Continuum limits of Ollivier's Ricci curvature on data clouds: pointwise consistency and global lower bounds. arXiv preprint arXiv:2307.02378 , 2023.

C. R. Genovese, M. Perone Pacifico, V. Isabella, L. Wasserman, et al. Minimax manifold estimation. Journal of machine learning research , 13:1263-1291, 2012.

S. Goel, A. Gollakota, Z. Jin, S. Karmalkar, and A. Klivans. Superpolynomial lower bounds for learning one-layer neural networks using gradient descent. In International Conference on Machine Learning , pages 3587-3596. PMLR, 2020.

O. Goldreich. Candidate one-way functions based on expander graphs. Studies in Complexity and Cryptography. Miscellanea on the Interplay between Randomness and Computation: In Collaboration with Lidor Avigad, Mihir Bellare, Zvika Brakerski, Shafi Goldwasser, Shai Halevi, Tali Kaufman, Leonid Levin, Noam Nisan, Dana Ron, Madhu Sudan, Luca Trevisan, Salil Vadhan, Avi Wigderson, David Zuckerman , pages 76-87, 2011.

V. V. Goncharov and G. E. Ivanov. Strong and weak convexity of closed sets in a hilbert space. Operations Research, Engineering, and Cyber Security: Trends in Applied Mathematics and Technology , pages 259-297, 2017.

F. Gray. Pulse code communication. United States Patent Number 2632058 , 1953.

M. Gromov. Isoperimetric inequalities in riemannian manifolds. Asymptotic Theory of Finite Dimensional Spaces , 1200:114-129, 1986.

J. S. Judd. Learning in networks is hard. In Proc. of 1st International Conference on Neural Networks, San Diego, California, June 1987 . IEEE, 1987.

N. Kambhatla and T. Leen. Fast non-linear dimension reduction. Advances in neural information processing systems , 6, 1993.

M. Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) , 45(6):983-1006, 1998.

B. K´ egl. Intrinsic dimension estimation using packing numbers. Advances in neural information processing systems , 15, 2002.

B. Kiani, T. Le, H. Lawrence, S. Jegelka, and M. Weber. On the hardness of learning under symmetries. In The Twelfth International Conference on Learning Representations , 2024.

A. K. Kim and H. H. Zhou. Tight minimax rates for manifold estimation under hausdorff loss. 2015.

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

D. E. Knuth. The art of computer programming , volume 3. Pearson Education, 1997.

M. Kossaczk´ a and J. Vyb´ ıral. Entropy numbers of finite-dimensional embeddings. Expositiones Mathematicae , 38(3):319-336, 2020.

J. B. Kruskal. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika , 29(1):1-27, 1964.

E. Levina and P. Bickel. Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems , 17, 2004.

H. Liu, M. Chen, T. Zhao, and W. Liao. Besov function approximation and binary classification on low-dimensional manifolds using convolutional residual networks. In International Conference on Machine Learning , pages 6770-6780. PMLR, 2021.

Y. Ma and Y. Fu. Manifold learning theory and applications . CRC press, 2011.

A. Maloney, D. A. Roberts, and J. Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.16859 , 2022.

T. Minka. Automatic choice of dimensionality for pca. Advances in neural information processing systems , 13, 2000.

M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machine learning . MIT press, 2018.

S. B. Myers. Riemannian manifolds with positive mean curvature. 1941.

H. Narayanan and S. Mitter. Sample complexity of testing the manifold hypothesis. Advances in neural information processing systems , 23, 2010.

H. Narayanan and P. Niyogi. On the sample complexity of learning smooth cuts on a manifold. In COLT , 2009.

A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. 2019.

W. Peng, T. Varanka, A. Mostafa, H. Shi, and G. Zhao. Hyperbolic deep neural networks: A survey. IEEE Transactions on pattern analysis and machine intelligence , 44(12):10023-10044, 2021.

P. Petersen. Riemannian geometry , volume 171. Springer, 2006.

J. Pidstrigach. Score-based generative models detect manifolds. Advances in Neural Information Processing Systems , 35:35852-35865, 2022.

P. Pope, C. Zhu, A. Abdelkader, M. Goldblum, and T. Goldstein. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2104.08894 , 2021.

L. Reyzin. Statistical queries and statistical algorithms: Foundations and applications. arXiv preprint arXiv:2004.00557 , 2020.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

S. T. Roweis and L. K. Saul. Nonlinear dimensionality reduction by locally linear embedding. science , 290(5500):2323-2326, 2000.

P. Rubin-Delanchy. Manifold structure in graph embeddings. Advances in neural information processing systems , 33:11687-11699, 2020.

D. Scieur, T. Kerdreux, A. d'Aspremont, S. Pokutta, et al. Strong convexity of sets in riemannian manifolds. arXiv e-prints , pages arXiv-2312, 2023.

U. Sharma and J. Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004.10802 , 2020.

J. Skilling. Programming the hilbert curve. In AIP Conference Proceedings , volume 707, pages 381-387. American Institute of Physics, 2004.

L. Song, S. Vempala, J. Wilmes, and B. Xie. On the complexity of learning neural networks. Advances in neural information processing systems , 30, 2017.

D. Sritharan, S. Wang, and S. Hormoz. Computing the riemannian curvature of image patch and single-cell rna sequencing data manifolds using extrinsic differential geometry. Proceedings of the National Academy of Sciences , 118(29):e2100473118, 2021.

J. Stanczuk, G. Batzolis, T. Deveney, and C.-B. Sch¨ onlieb. Your diffusion model secretly knows the dimension of the data manifold. arXiv preprint arXiv:2212.12611 , 2022.

B. Tahmasebi and S. Jegelka. The exact sample complexity gain from invariances for kernel regression, 2023.

J. B. Tenenbaum, V. d. Silva, and J. C. Langford. A global geometric framework for nonlinear dimensionality reduction. science , 290(5500):2319-2323, 2000.

S. Vempala and J. Wilmes. Gradient descent for one-hidden-layer neural networks: Polynomial convergence and sq lower bounds. In Conference on Learning Theory , pages 3115-3117. PMLR, 2019.

R. Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 , 2010.

A. Virmaux and K. Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems , 31, 2018.

M. Weber, M. Zaheer, A. S. Rawat, A. Menon, and S. Kumar. Robust large-margin learning in hyperbolic space. In Advances in Neural Information Processing Systems 34 , 2020.

N. Whiteley, A. Gray, and P. Rubin-Delanchy. Statistical exploration of the manifold hypothesis, 2023.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets. Advances in neural information processing systems , 30, 2017.

A Extended Related Works

Hardness of learning neural networks. The first results showing the hardness of learning neural networks are those of [53, 12] proving that proper learning of neural networks is an NP complete task. As mentioned earlier, a number of works prove hardness results for learning feedforward ReLU networks under the Gaussian i.i.d. data model [48, 36, 85, 26, 32]. Many of these works prove hardness results in the statistical query model [78, 55]. Similar hardness results, also in the SQ model, have also been shown for classes of symmetric or equivariant neural networks [57]. These results show that additional assumptions on the network class are needed to prove learnability results. Such assumptions which lead to provable learnability include those on the condition number or positivity of weights [7, 36], polynomial approximation guarantees [90], or bounds on width [27, 25]. When the target is not necessarily a feedforward network, [5] show that single hidden layer ReLU networks can efficiently learn functions with low so-called leap complexity categorized by the growth in the degrees of polynomials over a linear subspace for i.i.d. Gaussian or uniform Boolean inputs.

Intrinsic dimension estimation. There are many algorithms, dating back decades, which estimate the intrinsic dimension of a given set of data points including approaches based on PCA [44, 67], nearest neighbor methods [63], and others [19, 56]. In the context of modern large image datasets such as Imagenet, [77] perform estimates of the intrinsic dimension which are noticeably smaller than the input dimension. Various works study scaling laws arguing that the intrinsic dimension may be a useful indicator of the complexity of a dataset correlating to the rate at which neural network performance scale with increased data or training [83, 6]. These works estimate the intrinsic dimension via a maximum likelihood estimator calculating the intrinsic dimension in relation to the distances of a point to its nearest neighbors [63, 40]. In later work which estimates the intrinsic dimension using diffusion models, [87] empirically show that this MLE estimator can underestimate the intrinsic dimension for various toy datasets where the intrinsic dimension is known. For this reason, our experiments follow the approach in [87].

Learning algorithms over data manifolds. Among the earliest algorithms for learning over data manifolds are those for dimensionality reduction and representation learning [44, 8, 54, 33, 80]. As mentioned earlier, a number of works provide algorithms and statistical analysis for manifold reconstruction or learning where the goal is to construct a manifold that closely fits a target [21, 3, 42]. Perhaps closest to our work are studies of the hardness of learning functions defined over input manifolds. [71] categorize the sample complexity for binary classification over smooth cuts on a data manifold where smoothness is defined by the condition number of the classification boundary of the manifold (a quantity closely related to the reach). When no restrictions are placed on the probability distribution of data on the manifold, they show that the VC dimension is unbounded when this condition number exceeds the reach of the manifold. For nicer distributions where there is an upper bound on the density of the distribution on the manifold, a covering number argument shows that the sample complexity is dependent on the intrinsic dimension, reach, and condition number but independent of the ambient dimension. [64] show that for a certain class of convolutional networks trained on data lying in a low-dimensional manifold, the sample complexity depends weakly on the ambient dimension. [88] study the sample complexity of learning a target dataset over inputs taken from manifolds in kernel settings where the target function is invariant over some group operation acting on the input data manifold. The goal of their work was to quantify the gains in sample complexity from enforcing invariance and their sample complexity bounds depend on the volume of the manifold and dimension of the quotient space.

In recent years, various works have empirically tested and analyzed various aspects and implications of the manifold hypothesis. [16] argue that manifolds of data may have many connected components of potentially varying intrinsic dimension and provide some empirical estimates of intrinsic dimension by image class supporting this argument. Our empirical analysis of the intrinsic dimension also supports this point. [18] consider the task of learning a binary classification task where the two classes are two separate disconnected one dimensional manifolds (curves) on the sphere. Under regularity conditions on the extrinsic curvature and Riemannian distance properties of the curves, they prove that sufficiently wide and deep feedforward networks can learn to separate the manifolds. Their proofs are based on a neural tangent kernel approach. [66] theoretically and empirically study kernel models trained under settings where the latent dimension of the data can be varied. They

show that the rate of decay in the spectrum of the kernel is more strongly related to the scaling of the loss as opposed to the intrinsic dimension of the data with some discussion of potential connections between the two measures. [94] propose a statistical model of real-world data where low dimensional features and manifold structures can naturally emerge. [71] categorize the sample complexity for binary classification over smooth cuts on a data manifold where smoothness is defined by the condition number of the classification boundary of the manifold (a quantity closely related to reach). When no restrictions are placed on the probability distribution of data on the manifold, they show that the VC dimension is unbounded when this condition number exceeds the reach of the manifold. For nicer distributions where there is an upper bound on the density of the distribution on the manifold, a covering number argument shows that the sample complexity is dependent on the intrinsic dimension, reach, and condition number but independent of the ambient dimension.

Generative modeling. From the perspective of generative modeling, various works have recently considered the task of sampling from low-dimensional manifolds embedded in high dimensional space. [76] consider distributions supported over a data manifold of low intrinsic dimension. They provide sufficient conditions for score-based generative models to identify the support of the learned distribution matching that of the target manifold. [34] prove convergence results in Wasserstein distance between score-based models and target distributions which are supported on a manifold under the assumption that the score estimator is accurate in the L ∞ norm. These results were subsequently improved in [24] which only required a score estimator is accurate in the L 2 norm, among other improvements. For data supported on a low-dimensional linear subspace, [22] provide an end-to-end sampling guarantee avoiding the curse of dimensionality and showing that the score function can be efficiently estimated. The resulting sampling algorithm in their work thus approaches the true distribution in total variation distance and recovers the linear subspace appropriately.

Geometric Machine Learning. We briefly mention a related body of work, which studies machine learning algorithms, which directly leverage geometric structure in data. Such methods have shown empirical promise in a variety of domains [15]. Examples of such architectures are translation- and rotation-equivariant neural networks [30], graph neural networks [59] and DeepSets for permutation-invariant inputs [96], as well as hyperbolic machine learning algorithms [45, 93], which assume that data lies on a manifold of constant negative curvature.

B Deferred Proofs

B.1 Auxiliary statements

For completeness, we recall several classical notions from metric geometry.

Definition B.1 (Packing and covering numbers) . Given a set S ⊆ R n , the covering number cv S ( ϵ ) is the minimum number of balls of size ϵ needed to cover every point in S :

The packing number pk S ( ϵ ) is the maximum number of disjoint ϵ -balls that can be contained in S :

̸

Lemma B.2 (Packing and covering duality) . For any compact set S ⊂ R n and ϵ > 0 :

Proof. For the first inequality, assume by contradiction that cv S (2 ϵ ) < pk S (2 ϵ ) . Then, there must exist two points x i , x j in the maximal packing, which are within the same 2 ϵ -ball in the cover and thus have distance ∥ x i -x j ∥ ≤ 2 ϵ contradicting the assumption.

For the second inequality, note that any maximal packing with balls of radius ϵ is a 2 ϵ -covering, because otherwise, there would exist a point x ∈ S which is at least 2 ϵ distance away from all the packing points contradicting the definition of a packing.

Before stating the next result, we give a brief definition of Ricci curvature , which locally characterizes the curvature of M in a neighborhood of a point x ∈ M .

Definition B.3 (Ricci Curvature) . Let v ∈ T x M denote a unit vector and { u 1 , . . . , u m -1 , v } an orthonormal basis of T x M ; g x denotes the inner product on T x M . Then the Ricci curvature at x along the direction v is defined as

where R ( u, v ) w := ∇ u ∇ v w - ∇ v ∇ u w - ∇ [ u,v ] w is the Riemann curvature tensor, with [ u, v ] denoting the Lie Bracket between u and v .

Theorem B.4 (Corollary of Bishop-Gromov Theorem [10] (see [75] Lemma 7.1.3)) . Let Vol M ( p , r ) denote the volume of the ball of radius r around the point p with respect to the Riemannian distance metric. If a manifold M of intrinsic dimension d has bounded Ricci curvature Ric ( M ) ≥ ( d -1) K , then for all r > 0 and p ∈ M , Vol M ( p , r ) ≤ Vol S K d ( r ) where Vol S K d ( r ) denotes the volume of a ball of radius r around an arbitrary point p ∈ S K d in the space S K d with constant sectional curvature K ( d -sphere if K > 0 or d -dimensional Hyperbolic space if K < 0 ).

LemmaB.5 (Coupon collector bound [39]) . Let ξ 1 , . . . , ξ N be i.i.d. random variables taking values in [ n ] with uniform probability: P [ ξ i = k ] = 1 n for all k ∈ [ n ] . Let T n be the minimum value T such that for all k ∈ [ n ] , there exists at least one i ∈ [ T ] where ξ i = k (i.e. every value in [ n ] is covered). Then, E [ T n ] = Θ( n log( n )) and

B.2 SQ lower bounds

To show lower bounds in the statistical query model, we follow the technique employed in [26, 13], which constructs hardness from statistical independence properties of a function class. Since our setting differs slightly from those in [26, 13], we provide an adapted version of their constructions below.

Definition B.6 ( (1 -η ) -pairwise independent; see also Definition C.1 of [26]) . Let C be a function class consisting of functions f : X → Y . Let D be a distribution on X . C is (1 -η ) -pairwise independent if there exists a finite subset ̂ Y ⊆ Y such that with probability (1 -η ) over x , x ′ ∈ X drawn independently from D , the distribution of ( f ( x ) , f ( x ′ )) for f ∼ Unif( C ) drawn uniformly at random from C is the product distribution Unif( ̂ Y ) ⊗ Unif( ̂ Y ) .

The above agrees with Definition C.1 of [26] apart from the extension that here we define the independence with respect to a finite subset ̂ Y ⊆ Y of the output space.

Theorem B.7 (Theorem C.4 of [26]) . Let the function class C of functions f : X → Y be a (1 -η ) -pairwise independent function family with respect to a distribution D on X . For any f ∈ C , let D f denote the distribution of ( x , f ( x )) where x ∼ D . Let D Unif( C ) denote the distribution of ( x , y ) where x ∼ D and y = f ( x ) for f ∼ Unif( C ) . Any SQ learner able to distinguish the labeled distribution D f ∗ for an unknown f ∗ ∈ C from the randomly labeled distribution D Unif( C ) using bounded queries of tolerance τ requires at least τ 2 2 η such queries.

Proof. We follow the proof in [26]. Let ϕ : X ×Y → [ -1 , 1] be any query made by the learner and set ϕ [ f ] = E x ∼D [ ϕ ( x , f ( x ))] . Then,

In the last line, we use the fact that for any (1 -η ) -pairwise independent class C , the inner expectation is zero with probability at least 1 -η over the choice of x , x ′ ∼ D and at most 2 otherwise.

Since any SQ algorithm must work with any given value of the noise within the stated tolerance, consider the adversarial strategy where the SQ oracle responds to a query with ϕ = E f ∼ Unif( C ) [ ϕ [ f ]] whenever possible. By Chebyshev's inequality,

So each such query only allows the learner to rule out at most a 2 η τ 2 fraction of C . Thus to distinguish D f ∗ from D Unif( C ) , the learner requires at least τ 2 2 η queries.

B.3 Additional examples of learnable manifolds

Here, we provide additional examples of manifolds and their properties that render such manifolds learnable as detailed in Section 3.1. There, learnability of the distribution of the manifold corresponded to the property that one can efficiently approximate the manifold with an epsilon net.

As mentioned in the main text, many algorithms in the manifold reconstruction literature implicitly assume that the manifold is efficiently sampleable. We give one such example below.

Example B.8 (Manifold Learning Setting [9]) . [9] provide provable guarantees for the Isomap algorithm in the sense of recovering geodesic distances on a manifold M via shortest-path distance on a similarity graph G under the following assumptions (listed in Main Theorem A in [9]), which informally require among other things: (1) the algorithm is given a set of samples { x i } , which form a δ -net over the manifold M ; (2) M is geodesically convex; (3) the reach (equivalently encoded as bounds on the radius of curvature and branch separation) is bounded as Rch( M ) = O ( δ ) . Assumption (1) suffices to guarantee learnability in our setting.

We also extend Proposition 3.5 to incorporate manifolds with negative curvature. Negatively curved manifolds have volume that can grow exponentially with respect to the radius around a given point. However, as long as the radius of a sequence of manifolds does not grow with the ambient dimension, the Bishop-Gromov inequality bounds the volume of such manifolds rendering them efficiently sampleable. As far as we are aware, estimating the diameter of real-world manifolds is a challenging task and it is unknown whether real-world data manifolds can fit within this regime.

Example B.9 (Manifolds with bounded curvature and radius) . Any sequence of distributions {P M n } supported over manifolds {M n } where each manifold M n has bounded Ricci curvature Ric ( M n ) ≥ ( d -1) K for an arbitrary constant K and is contained within a ball of radius r = O n (log( n )) around a point p n ∈ M n is efficiently sampleable.

Proof. By the Bishop-Gromov theorem (Theorem B.4), the volume of any such manifold is bounded by Vol S K d ( r ) which is the volume of a ball of radius r in the d -dimensional Hyperbolic space of constant curvature K . This means that

Thus, whenever r = O n (log( n )) , this allows for an ϵ -cover of size O n (poly( n )) . The rest of the proof follows directly from that of Proposition 3.5.

Another example of a class of manifolds that fall in the learnable regime are those which are α -strongly convex [82], a property that can be leveraged in instances of Riemannian optimization. This property is extended from the standard notion of strong convexity on Hilbert spaces.

Definition B.10 ( α -strong convexity of a Hilbert space [50]) . Given a real Hilbert space H with inner product ⟨· , ·⟩ and corresponding induced norm ∥ · ∥ , denote by B R ( c ) the closed ball of radius R centered around c ∈ H . A subset A ⊂ H is α -strongly convex if there exists a set C ⊂ H such that

where R = 1 2 α .

The above definition captures the equivalent notion in the Euclidean space R d that an α -strongly convex set A ⊂ R d is one where for any x , y ∈ A and unit norm z ∈ R d such that ∥ z ∥ = 1 , we have that [82]

Riemannian manifolds are α -strongly convex if the image of the exponential map over inputs in a set A living in the tangent space are α -strongly convex in the Euclidean sense.

Definition B.11 ( α -strong convexity of a Riemannian manifold [82]) . Let M be a Riemannian manifold that is uniquely geodesic (i.e. any two points x , y ∈ M can be connected by a unique geodesic). Then, M is Riemannian α -strongly convex if for any x ∈ M , the set

is α -strongly convex with respect to the inner product ∥·∥ x on T x M in the sense of Definition B.10.

The above definition places restrictions on the diameter of manifolds, essentially placing them in the setting of bounded radius manifolds studied in Example B.9.

Example B.12 (Strongly convex manifolds) . As stated in Definition B.10 for any α -strongly convex manifold M , the diameter of Exp -1 x ( M ) is at most 2 r = α -1 . For a sequence of such Riemannian α -strongly convex manifolds {M n } , as long as the metric over the manifold in the normal coordinates at T x M is bounded everywhere (or say the curvature is bounded), then such sequences of manifolds are efficiently sampleable similar to the setting of Example B.9.

C Proofs for Section 3.2

C.1 Space-filling manifold

To construct a manifold which achieves the desired lower bound, we form a customized manifold which acts as a space-filling curve touching exponentially many quadrants of the n -dimensional hypercube. This space-filling curve is constructed by maneuvering around a Gray code which enumerates Boolean strings in such a fashion that successive strings differ by Hamming distance one. This procedure is motivated by and reminiscent of techniques used in constructing Hilbert curves from Gray codes [84].

Lemma C.1 (Gray code [51]) . For every integer k > 0 , there exists a bijective function G : [2 k ] → { 0 , 1 } k enumerating length k bitstrings such that successive bitstrings G ( i ) and G ( i +1) differ in only one coordinate, i.e., | G ( i ) -G ( i +1) | H = 1 where | · | H denotes the Hamming distance.

As a simple example, for k = 3 , the sequence [000 , 001 , 011 , 010 , 110 , 111 , 101 , 100] is one such Gray code which can be constructed recursively as a binary-reflected Gray code [60]. We also visualize the k = 7 bit Gray code in Figure 5.

Given an intrinsic dimension d , we will construct manifolds M as products of P d = { x ∈ [0 , 1] d -1 } , the ( d -1) -dimensional hypercube, and a one-dimensional submanifold over the remaining dimensions, which resembles a one-dimensional space-filling curve. The submanifold is constructed to touch the corners of the hypercube following a Gray code.

One-dimensional space-filling curve construction. Our goal is to construct a one-dimensional sub-manifold M⊂ [0 , 1] n -d +1 that covers as many quadrants of the hypercube as possible. Given a radius of curvature R , which we will later set to be within the bounds of the stated reach, let n R = ⌊ ( n -d +1) /δ R ⌋ and δ R = ⌈ 4 R 2 ⌉ . In our construction, we will use the Gray code on n R bits and map these to bitstrings in dimension n by copying the bitstring of the Gray code δ R times.

1d Manifold ( n =3)

x

1

0.0

0.5

1.0

x

2

0.0

0.5

1.0

0.0

0.5

1.0

x

This construction will allow the manifold to have radius of curvature conforming to the given bound on the reach.

Specifically, let G : [2 n R ] →{ 0 , 1 } n R be a Gray code over n R bits, which maps indices 1 , . . . , 2 n R to the corresponding bitstring in the Gray code (see Lemma C.1). Then, for i ∈ [2 n R ] let b i = G ( i ) ⊕ δ R ⊕ 0 n -n R where G ( i ) ⊕ δ R is the bitstring G ( i ) repeated δ R times and 0 k is the bitstring of length k with each entry equal to 0 . 0 k is concatenated at the end to ensure the input is of dimension n . Then, the space-filling sub-manifold takes the form:

As an example, the manifold M 3 is visualized in Figure 6. Indexing of the bitstrings b k is assumed circular and taken k mod 2 n .

Lemma C.2. The manifold M n R defined in Equation (18) has Rch( M n R ) = R . Furthermore, defining

as the operation which projects each point onto the nearest corner corresponding to bitstrings of the hypercube, we have that | Round( M n R ) | = 2 n R .

Proof. Each segment of the manifold in Equation (18) is an arc of a circle centered at b k -1 + b k +1 2 and of radius √ δ R / 2 . For two segments indexed by k ′ such that | k -k ′ | ≥ 2 , the segments differ maximally in at least δ R locations so the medial axis between these segments is at least √ δ R / 2 ≥ R distance away.

Between neighboring segments indexed by k and k +1 , we note that we have a curve which consists of circular arcs from [ 0 δ R , 1 δ R / 2 , 1 δ R ] ⊕ s to [ 0 δ R , 1 δ R , 1 δ R / 2] ⊕ s and then to [ 1 δ R / 2 , 1 δ R , 0 δ R ] ⊕ s for some bitstring s which is shared between all of the points in the curve. Here, for simplicity, we have permuted the distinct elements in the neighboring segments to the beginning of the bitstrings. Consider now the medial axis of this curve and set δ R = 1 for simplicity as the general case will directly follow. Here, the medial axis takes the form of points [ a, 1 -b, c ] ⊕ s where 0 ≤ a, b ≤ 1 / 2 and 0 ≤ c ≤ 1 . For any such point where c > 0 . 5 or c < 0 . 5 , the nearest point in the manifold will be a point in the first or second segment respectively. For such points, the medial axis will thus be the center of the corresponding circle which the arc spans. When c = 0 . 5 , if a = b < 0 . 5 , then the unique nearest point is always [0 , 1 , 1 / 2] ⊕ s . Otherwise, if c = 0 . 5 and a < b or b < a , the nearest point in the manifold will be a point in the first or second segment respectively as in the previous case. Putting this all together, the reach of this manifold is √ δ R / 2 ≥ R .

3

Finally, to count the number of elements in Round( M n R ) , note that setting t = π/ 4 for segment k in Equation (18) obtains a point of the form

The above is rounded to b k . Applying this procedure for all k gives 2 n R total rounded points.

Corollary C.3. For a sequence of manifolds {M n } of intrinsic dimension d = O (1) and reach Rch( M n ) = O ( n α ) for α < 0 . 5 taking the form of Lemma C.2, the manifold covers exponentially many quadrants of the Boolean cube, i.e. | Round( M n ) | = 2 Ω( n ) .

In our proofs, we will consider the setting where we have a sequence of manifolds of increasing intrinsic dimension n and bounded reach. We choose the parameters of this sequence as follows.

Definition C.4 ( ( α, d ) -sequence of manifolds) . For given intrinsic dimension d and bound on the reach R = O ( n α ) for some α < 0 . 5 , set n = ⌈ 4 R 2 ⌉ n b + d -1 . We construct a sequence of manifolds {M n } of the form given in Equation (18) with the given values of R,n,d where n is incremented by increasing values n b .

C.2 SQ hardness

Wepresent here the hardness results in the statistical query (SQ) model by mapping a class of hard to learn Boolean functions to real-valued neural network functions, which approximate those Boolean functions well. This follows a commonly used set of proof techniques employed in [36, 32, 26, 48]. We follow most closely the line of reasoning in [26] for showing hardness results for learning two hidden layer ReLU networks though the nature of the distribution in our case will allow us to show SQ hardness results for single hidden layer networks. Similar hardness results based on reductions to cryptographically hard problems as shown in the work of [32] are also provided in Appendix C.3.

Hardness in our setting is based on classic results of the SQ hardness of learning parity functions over Boolean inputs [55, 11]. A parity function χ S : { 0 , 1 } d →{ 0 , 1 } is a Boolean function taking the form

which determines whether the summation of the input Boolean string over indices in the set S ⊆ [ d ] is even or odd. The class of such parity functions indexed by subsets S ⊆ [ d ] is exponentially hard to learn in the SQ model.

Theorem C.5 (SQ hardness of parities [11, 55], see also Theorem 4.3 of [26]) . Any SQ algorithm given SQ access to the distribution of labeled pairs ( x b , y ) where x b ∼ Unif( { 0 , 1 } d ) and y = χ S ( x b ) for an unknown S ⊆ [ d ] capable of learning χ S up to classification error ϵ sufficiently small with queries of tolerance τ requires Ω( τ 2 2 d ) queries.

For Boolean inputs x b ∈ { 0 , 1 } d , the parity function χ S ( x b ) can be constructed as a single hidden layer neural network with O ( d ) nodes. Ideally, one would want to replicate this over real-valued inputs, but since ReLU networks are continuous, this construction cannot be made exactly. [32] show that, in practice, one can form a network which is equivalent to the sign function for all but a small percentage of inputs. Combined with an indicator function that zeros out the output wherever such inputs cannot be rounded exactly, they show that equivalent hardness results can be obtained in real-valued settings. This construction was extended to SQ settings by [26].

In our setting where the underlying distribution is uniformly drawn from the space-filling manifold described in Appendix C.1, we can resort to a simpler construction given that most input values are either 0 or 1 at any point in the space-filling manifold. This lets us identify approximate parity functions which for all but an exponentially small fraction of inputs is equivalent to the Boolean parity function.

To achieve this goal, we need to map Boolean parity functions χ S : { 0 , 1 } n b → { 0 , 1 } to realvalued neural network functions f : R n → R that approximate the parity functions appropriately. In this construction, for given intrinsic dimension d and bound on the reach R = O ( n α ) for some α < 0 . 5 , we set n = ⌈ 4 R 2 ⌉ n b + d -1 in the construction given in Appendix C.1 ( n chosen so

that the number of bits in the Gray code n R is equal to n b ). Each input entry is repeated ⌈ 4 R 2 ⌉ times in this construction, so for a given input x ∈ R n , we define P : R n → R n b as an operation, which selects one of the ⌈ 4 R 2 ⌉ many equivalent inputs for each of the n b potentially unique values. That is, for segments of the manifold taking the form of Equation (18), each segment is a sum of elements b i = G ( i ) ⊕ δ b ⊕ 0 n -n b where G : [2 n b ] → { 0 , 1 } n b is a Gray code over n b bits. Setting P x = [ x ] : n b to capture the first n b elements then suffices to perform such an operation.

The intuition for our construction is as follows: We have to show that at most points on the manifold, inputs are with high probability equal to some Boolean string. If we choose the marginal distribution over the first k input locations of P x with k < n b sufficiently smaller than n b , then with high probability, these will always be constantly set to either value 0 or 1 in the segments of the manifold (Equation (18)). This is because the reflected Gray code is formed by changing the right-most bits first whenever possible so the left-most bits are constant along most segments. We formalize this below.

̸

Lemma C.6. Let D M n denote the uniform distribution over the ( α, d ) -sequence of manifolds M n constructed in Definition C.4 for given bounds on the reach R = O ( n α ) and intrinsic dimension d . Given x ∈ M n constructed over the binary code over n b bits, let [ P x ] : n b -t denote the vector consisting of the first n b -t entries of P x for t ∈ [ n b ] . Then, with probability 1 -O (2 -t ) , a draw of x ∼ D M n will have the property that [ P x ] : n b -t ∈ { 0 , 1 } n b -t . Additionally, with probability 2 t -n b + O (2 -t ) over independent draws x , x ′ ∼ D M n , [ P x ] : n b -t = [ P x ′ ] : n b -t and the resulting values will differ in at least one location.

Proof. In the binary reflected Gray code G : [2 n b ] →{ 0 , 1 } n b over n b bits, at any location i ∈ [2 n b ] , there exists a span of integers [ a, b ) containing i of length b -a = 2 t such that [ G ( j )] n b -t = [ G ( j ′ )] n b -t is equal for all j, j ′ ∈ [ a, b ) . Furthermore, by removing the endpoints where for any j, j ′ ∈ [ a + 1 , b -1) , any entry of [ G ( j )] k for k ∈ [ n b -t ] will also have the property that [ G ( j )] k = [ G ( j ′ )] k (see Figure 5 for example for a visualization of this property). The value at a given point in a segment of the manifolds in Equation (18) is not equal to zero or one only when bitstring indices are differing in three adjacent points in the Gray code. Given a random point x ∼ D M n , it will fall within a given segment of the Gray code spanned over elements i -1 , i, i +1 of the Gray code for i ∼ Unif([2 n b ]) . Thus, the probability that [ P x ] : n b -t ∈ { 0 , 1 } n b -t is at least 1 -O (2 -t ) .

To prove the additional fact, note that for any z ∈ { 0 , 1 } n b -t , we have that

This follows from the previously stated property that at any location i ∈ [2 n b ] in the gray code, there exists a span of integers [ a, b ) containing i of length b -a = 2 t such that [ G ( j )] k = [ G ( j ′ )] k is equal for all j, j ′ ∈ [ a + 1 , b -1) and all k ∈ [ n b -t ] . From this span [ a, b ) we remove the first and last three indices to get the span [ a + 4 , b -4) over which [ G ( i )] n b -t = [ G ( i + k )] n b -t for all i ∈ [ a + 4 , b -4) and k ∈ [ -2 , 2] . I.e. this span is chosen to ensure that x n b -t is constant in Equation (18) over those segments of the Gray code. From here, we have that over independent draws x , x ′ ∼ D M n :

The second line above follows from H¨ older's inequality. The third line above applies Equation (22). Noting that the last term is 2 t -n b + O (2 -t ) completes the proof.

The above shows that with exponentially high probability, samples drawn from the distribution over the space-filling manifold will be equivalent to those drawn uniformly over Boolean inputs. This lets us complete the proof of hardness.

Theorem C.7. Let D M n denote the uniform distribution over the ( α, d ) sequence of manifolds M n constructed in Definition C.4 where Rch( M n ) = O ( n α ) for α < 0 . 5 . Any SQ algorithm A capable of learning the class of linear width single hidden layer ReLU neural networks under this sequence of distributions up to mean squared error sufficiently small ( ϵ/ 8 suffices) with queries of tolerance τ must use at least Ω( τ 2 2 n Ω(1) ) queries.

Proof. Given the sequence of manifolds {M n } from the construction in Definition C.4, we have n b = Ω( n 1 -2 α ) which is chosen to fit the bounds on the reach of the manifold.

Given a parity function χ S : { 0 , 1 } d →{ 0 , 1 } taking the form

consider its continuous approximation ˜ χ S : [0 , 1] d →{ 0 , 1 } defined as

Note, that the above is valid for d even and a similar equation can be obtained for d odd. The above is also piecewise linear and can be constructed as a single hidden layer ReLU network with O ( d ) width. We consider learning the class C of single hidden layer networks consisting of

̸

Setting t = n b / 2 suffices by Lemma C.6 to guarantee that [ P x ] : n b / 2 is a Boolean string with probability 1 -O (2 -Ω( n b ) ) . Furthermore, for x , x ′ drawn i.i.d. from D M n , [ P x ] : n b / 2 = [ P x ′ ] : n b / 2 with probability 1 -O (2 -Ω( n b ) ) . Conditioned on this event, the distribution of ( f S ( x ) , f S ( x ′ )) is equal to Unif( { 0 , 1 } 2 ) . Since n b = Ω( n 1 -2 α ) , we can apply Theorem B.7 noting that learning the function class up to MSE ϵ sufficiently small suffices to distinguish the distribution task in Theorem B.7.

C.3 Cryptographic hardness reduction

Here, we will prove hardness results for learning single hidden layer networks under cryptographic hardness assumptions following the methodology in [32]. First, we detail the construction of Goldreich's pseudorandom generator (PRG) [49] which are used as the basis for the hard to learn class of single hidden layer neural networks in [32]. In fact, [32] show that given random Boolean inputs, a neural network with width ω (1) suffices to construct these hard to learn functions. From here, we show that the manifold constructed in Section 3.1 can produce inputs that are approximately uniformly distributed over the Boolean cube when restricted to a certain set of input locations. This lets us directly apply the results of [32] to this distribution.

Pseudorandomness assumption The pseudorandom functions are constructed over an ( n, m, k ) -hypergraph with n vertices [ n ] with m ordered hyperedges S 1 , . . . , S m , each having cardinality k and no repeated entries. Hypergraphs are drawn from the Erd˝ os-R´ enyi distribution G n,m,k where each hyperedge is drawn i.i.d. from the set of n ! / ( n -k )! possible hyperedges (equivalent to ordered sets).

Definition C.8 (Goldreich's pseudorandom generator (PRG) [49]) . Given constant integer k > 0 , predicate P : { 0 , 1 } k → { 0 , 1 } , and a ( n, m, k ) -hypergraph G , Goldreich's pseudorandom generator (PRG) is the function f P,G : { 0 , 1 } n → { 0 , 1 } m , such that given input x ∈ { 0 , 1 } n returns f P,G ( x ) = ( P ( x S 1 ) , . . . , P ( x S m )) . The PRG has polynomial stretch if m = n a for some a > 1 .

Denoting the collection of functions f P,G over ( n, m, k ) -hypergraphs as F P,n,m then F P,n,m is an ε -pseudorandom generator ( ε -PRG) if every polynomial-time probabilistic algorithm A has advantage at most ϵ in distinguishing pairs ( G,f P,G ( x )) from random pairs ( G, y ) where y is drawn i.i.d. from the uniform distribution on { 0 , 1 } m . This is formalized below.

Definition C.9 ( ε -pseudorandom generator ( ε -PRG) [32]) . F P,n,m is an ε -pseudorandom generator ( ε -PRG) if for every polynomial-time probabilistic algorithm A , it holds that

where B k denotes the uniform distribution over bitstrings { 0 , 1 } k .

The cryptographic hardness assumption posits that there does exist such a class as above for which it is hard to distinguish between the cases where y is completely random or the output of a randomly drawn function f P,G .

Assumption C.10 (Existence of ϵ -PRG [32]) . For every constant s > 1 , there exists a constant k and a predicate P : { 0 , 1 } k →{ 0 , 1 } , such that F P,n,n s is 1 3 -PRG.

Hard to learn neural network function class Wefollow the procedure in the previous subsection to map hard to learn Boolean functions to the task of learning real-valued functions where inputs are drawn from the data manifold. In this procedure, we must show that instances where a given input does not correspond to a Boolean string occur with a vanishingly small probability. Following the previous construction, for given intrinsic dimension d and bound on the reach R = O ( n α ) for some α < 0 . 5 , we again set n = ⌈ 4 R 2 ⌉ n b + d -1 in the construction given in Appendix C.1 ( n chosen so that the number of bits in the Gray code n R is equal to n b ). As each input entry is repeated ⌈ 4 R 2 ⌉ times in this construction, we recall the definition of P : R n → R n b selecting one of the ⌈ 4 R 2 ⌉ many equivalent inputs for each of the n b potentially unique values. The following lemma then guarantees upon that taking the first n b -t entries, with probability 1 -O (2 -t ) the distribution of [ P x ] : n b -t will be equivalent to the uniform distribution over Boolean strings.

Lemma C.11. Let D M n denote the uniform distribution over the ( α, d ) -sequence of manifolds M n constructed in Definition C.4 for given bounds on the reach R = O ( n α ) and intrinsic dimension d . Given x ∈ M n constructed over the binary code over n b bits, let [ P x ] : n b -t denote the vector consisting of the first n b -t entries of P x for t ∈ [ n b ] . Denoting PD M n as the distribution of [ P x ] : n b -t for x ∼ D M n , we have that with probability at least 1 -2 3 -t , PD M n returns a uniformly random Boolean string:

where B k denotes the uniform distribution over bitstrings { 0 , 1 } k and D rem is an arbitrary distribution handling the event where draws from the distribution PD M n are not uniform over the Boolean string.

Proof. It suffices to show that there exists an event E occuring with probability at least 1 -2 3 -t such that conditioned on this event, for every z ∈ { 0 , 1 } n b -t :

i.e. every bitstring z ∈ { 0 , 1 } n b -t is returned with equal probability. In fact, previously in Lemma C.6 and specifically Equation (22), we showed that

Therefore, we can construct the event E by taking the union over the bitstrings z , each of probability 2 t -8 2 n b ignoring a portion of the distribution whenever a given P [[ P x ] : n b -t = z ] exceeds this value. This implies that

Theorem C.12 (Cryptographic hardness) . Let D M n denote the uniform distribution over manifolds M n constructed in Definition C.4 where Rch( M n ) = O ( n α ) for α < 0 . 5 . Set a > 0 to be constant. Then under Assumption C.10, there does not exist any polynomial time algorithm A that with probability 1 -δ for 0 < δ < 1 / 2 is capable of learning the class of width O ( n a ) single hidden layer ReLU neural networks under this sequence of distributions up to mean squared error sufficiently small ( ϵ/ 40 suffices).

Proof. Theorem 8 (part 3) of [32] states that under Assumption C.10, there does not exist an efficient algorithm to learn the class of O ( n a ) width neural networks under the uniform input distribution B m over bitstrings { 0 , 1 } m . Namely, they show that there exist a set of neural network functions f : R m → [0 , 1] that have the property that they recover a hard to learn Boolean DNF formula

̸

f ( x ) ∈ { 0 , 1 } for any x ∈ { 0 , 1 } m . They then show that any algorithm which with probability 1 -δ returns a hypothesis h b : { 0 , 1 } m →{ 0 , 1 } achieving classification error P x ∼B m [ h b ( x ) = f ( x )] ≤ 1 / 10 suffices to contradict Assumption C.10.

̸

Any efficient algorithm can only take O (poly( n )) samples. From Lemma C.11, we can set m = n b / 2 with t = n b / 2 so that with probability at least 1 -exp( -n Ω(1) ) a given sample is drawn from B n b / 2 . By union bounding over O (poly( n )) samples, we are guaranteed that all samples are drawn i.i.d. from B n b / 2 with probability 1 -δ ′ for δ ′ vanishing exponentially in n . Given any input x b ∈ { 0 , 1 } m , we can convert it to a random point on the corresponding quadrant in the manifold x = T x b ∈ M n by drawing t ∼ Unif([0 , π/ 2]) and applying Equation (18). This recovers the distribution D M n . Now, assume that the algorithm for learning single hidden layer neural networks under the distribution D M n returns a hypothesis h : R n → R . We can convert this to a booleanoutput function h ′ : R m →{ 0 , 1 } by setting h ′ ( x ) = sign( h ( x ) -1 / 2) . Note that (see also Lemma 23 of [32]) whenever h ′ ( T x b ) = f ( x b ) then ( h ( T x b ) = f ( x b )) 2 ≥ 1 / 4 and

̸

̸

The right hand side above contains the mean squared error loss over random Boolean inputs. This is equivalent to the mean squared error loss under the distribution D M n up to an exponentially small correction for the event where [ P x ] : n b -t is not Boolean. Thus, learning the function f up to mean squared error ϵ/ 40 suffices to output a function h ′ achieving classification error ϵ/ 10 (up to these exponentially vanishing corrections). Applying Theorem 8 of [32] finishes the proof.

C.4 Volume bounds based on reach

The reach of a manifold lower bounds the radius of curvature of a manifold. In our setting where manifolds are contained within the hypercube [0 , 1] n , restrictions on the reach can intuitively limit the space-filling capacity of a manifold resulting in upper bounds on the volume of the manifold. Namely, as we will show below, for a sequence of manifolds {M n } of intrinsic dimension d , if the reach is bounded by Rch( M n ) = ω ( n 0 . 5 ) , then the volume of the manifold is Vol d ( M n ) = poly( n ) . This implies that the results proven in the previous subsections are tight up to Rch( M n ) = Θ( n 0 . 5 ) , where similar exponential hardness results may apply though are not immediately obtained from our proofs. We formally detail the tightness of our results in what follows below.

Our volume bounds will follow from Lemma B.1 of [3] which locally bound the volume of a manifold of bounded reach contained within a ball.

Lemma C.13 (Adapted from Lemma B.1 of [3]) . Consider a Riemannian manifold M ⊂ R n of intrinsic dimension d with reach bounded by R ≤ Rch( M ) . Let D be a measure on R n supported on M with density function f with respect to the volume measure Vol d where f is Lipschitz smooth and bounded by 0 < f min ≤ f ( x ) ≤ f max for all x ∈ M . Denote B ( x , h ) = { x ′ : ∥ x -x ′ ∥ ≤ h } as the ball centered around x of radius h . Then for any x 0 ∈ R n and h ≤ R/ 8 , it holds that

Lemma C.14. Denote B ( x , h ) = { x ′ : ∥ x -x ′ ∥ ≤ h } as the ball centered around x of radius h . Given a manifold M ⊂ R n of intrinsic dimension d bounded in reach by Rch( M ) ≥ R , the volume of the intersection of the manifold and a ball B ( x , h ) with h ≤ R/ 8 is bounded by

where ω d is the volume of the d -dimensional unit ball.

Proof. This is a consequence of Lemma B.1 of [3] recalled in Lemma C.13 setting f min = f max = 1 to recover the standard volume measure, i.e. for a given point x ∈ R n and h ≤ Rch( M ) / 8 , Lemma C.13 states

Applying the bound h ≤ Rch( M ) / 8 and noting that (5 / 4) d/ 2 2 d (1 / 8) d ≤ (1 / 2) d completes the proof.

The above lemma provides an immediate bound on the volume as one can note that for large enough n , any manifold M⊂ [0 , 1] n is contained within a single ball B ( x, h ) for h = ω ( n 0 . 5 ) .

Proposition C.15. Given a sequence of manifolds {M n } of intrinsic dimension d (fixed and independent of n ) and reach bounded by Rch( M n ) = ω ( n 0 . 5 ) , the volume of the manifolds grows at most Vol d ( M n ) = O (poly( n )) .

Proof. There exists large enough n such that [0 , 1] n is contained within a ball of radius equal to the bound on the reach Rch( M n ) = ω ( n 0 . 5 ) . Applying Lemma C.14 with h = √ n/ 2 and x = 1 / 2 then implies that for large enough n :

Remark C.16 . The above implies that manifolds with reach bounded as Rch( M n ) = ω ( n 0 . 5 ) fall within the class of efficiently sampleable manifolds defined in Definition 3.2. The proof above does not extend to the setting where Rch( M n ) = Ω( n 0 . 5 ) . Covering number bounds (e.g. see Theorem 2 of [61]) show that exponentially many (in n ) balls of radius cn 0 . 5 are needed to cover the space [0 , 1] n when c is sufficiently small. Therefore, Proposition C.15 does not guarantee that volume is polynomially bounded in n specifically for the setting where Rch( M n ) = Θ( n 0 . 5 ) .

D Experiments confirming learnability results

0

50

100

150

200

Epoch

0.0

0.2

0.4

0.6

0.8

Test MSE

20

50

100

200

0

5000

10000

15000

Step

0.0

0.5

1.0

1.5

2.0

Test MSE

10

20

50

For Figure 3a, the data is generated by drawing random unit norm vectors from a subspace of dimension d = 10 with ambient space of dimension n . The random d -dimensional subspace is obtained by drawing a random n × n orthogonal matrix and taking its first d = 10 columns. Target functions for Figure 3a are drawn from the class in Equation (5) in [36], which form a hard class of single hidden layer neural networks to learn under the i.i.d. Gaussian input model. We set the hidden layer width k = n/ 4 = O ( n ) for this construction. To corroborate these findings, we also show in Figure 7 that the results are consistent when the objective is to learn a target function which is a single hidden layer ReLU neural network with randomly chosen weights of the same form. To normalize all target functions to have a consistent MSE benchmark, we divide by the norm of the function which is approximated by taking the average norm over a batch of 100 randomly chosen inputs.

We use the Pytorch package for our neural network experiments [73]. Training a single neural network on the tasks shown in Figure 3 took no longer than a few minutes on a single GPU. For the learnable setting in Figure 3a, we use a training set of size 1000 and a neural network of a single hidden layer and width 100 to learn the target function. For the hard setting of Figure 3b, we provide the algorithm with a fresh randomly drawn batch of data each training step. The trained neural network is overparameterized with width 2 n in these experiments. For the hard setting in Figure 3b, we also attempted to learn the target function with an overparameterized network with three hidden layers and width 2 n . The results, shown in Figure 8, are consistent with those shown in Figure 3b. In all experiments, the learning rate for the optimizer was varied by the default value times a random multiplicative factor exp( c ) where c ∼ Unif([ -2 , 1]) ; results were largely consistent across all ranges of the tuned values.

E Estimating Intrinsic Dimension

E.1 Details on Experimental Setup

Following [87], to calculate the intrinsic dimension at a given point p ∈ M , we sample a local neighborhood of points around p by noising p a small amount. Then, from a trained diffusion model, we collect a series of score vectors { s i } N i =1 at the sampled neighborhood which point in the direction of the de-noised sample. We then apply PCA upon this collection of N score vectors (treated as a matrix S ∈ R n × N ) which gives us a basis for the tangent space T p M and normal space N p M . Given potential instabilities inherent in training diffusion models and performing statistical estimation on manifolds, we use the stable rank to estimate the rank of S [91]. Given a matrix M , the stable rank is given by

where ∥· ∥ F and ∥· ∥ ∞ denote the Frobenius norm and maximum singular value respectively. Since S is actually high rank, we estimate instead the rank of s max I -S where s max is the max singular value of M to convert this into a low rank estimation problem. We find the stable rank above to be a more robust measure of estimated rank in comparison to other such metrics such as the maximum difference in singular values.

We note that this approach can also be taken by estimating the rank over a neighborhood of sampled points as well. That is, we find that collecting difference vectors between p and the denoised points as samples also estimates the dimension of the tangent space d accurately. Such a method may be more suited to higher dimensional inputs. However, to be consistent with prior work, we perform our analysis on collections of score vectors here. [87] have confirmed this overall methodology on synthetic datasets where the intrinsic dimension is known, showing that in general, it outperforms other statistical means to estimate intrinsic dimension.

Diffusion model and training. For Euclidean datasets, we train a fully-connected 5-layer ReLU network of width 2048 on 100000 samples for 30 epochs, batch size 64, and a learning rate of 4 × 10 -4 .

For image datasets, we pre-process the images to have pixel values in [ -1 , 1] . We train a DDPM U-Net 2D model [79] on 10000 samples for 30 epochs, batch size 64, and a learning rate of 4 × 10 -4 . The architecture is held the same across all three datasets: a ResNet block, a ResNet downsampling

block with spatial self-attention, and a ResNet block with 64, 128, and 256 channels each (and the reverse for upsampling).

We use a noise scheduler in the DDPM paradigm with 1000 timesteps between the image and pure Gaussian noise, with the squared cosine beta schedule introduced by [72]. This training was done on an NVIDIA L4 24GB GPU.

We investigate synthetic unit hyperspheres to sanity check our method and proceed to three realworld image datasets: MNIST [35], Fashion MNIST (FMNIST) [95], and Kuzushiji-MNIST (KMNIST) [29]. For each of the three image datasets, we consider two sizes: the original 28 × 28 and a downsampled 12 × 12 .

E.2 Results on synthetic data manifolds

Following [87], we verify our intrinsic dimension estimation procedure on unit hyperspheres of different dimension, randomly projected into a higher-dimensional ambient space. We vary intrinsic and ambient dimension and report estimates of intrinsic dimension measurements (Table 2). We observe that, generally, the estimated and true intrinsic dimension align well. Our results further suggest an overall higher accuracy if the intrinsic dimension is small compared to the ambient dimension. Figure 9 shows the singular values of the score vector for different configurations; the curves, resembling a step function, distinguish normal from tangent directions.

2-Sphere in 20 Dimensions

10-Sphere in 20 Dimensions

18-Sphere in 20 Dimensions

300

0OE

Intrinsic Dim: 10

200

Intrinsic Dim: 17

250

175

250

150

Singular Value

Singular vlue

200

Singular Value

200

125

150

100

150

100

100

50

50

50

25

0

0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

5.0

7.5

10.0

12.5

15.0

17.5

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Dimension

Dimension

Dimension

10-Sphere in 100 Dimensions

50-Sphere in 100 Dimensions

90-Sphere in 100 Dimensions

Intrinsic Dim: 47

300

Intrinsic Dim: 82

300

0OE

250

250

250

200

Singular Value

200

Singular Value

200

150

150

100

100

100

50

50

50

Intrinsic Dim: 12

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Dimension

Dimension

Dimension

E.3 Results on real-world data manifolds

As stated in the main text, we observe that our estimate of MNIST's intrinsic dimension matches that obtained in [87]. We observe slightly lower values, which we believe to be due to different

input sizes. In [87], MNIST images were upscaled to size 32 × 32 , which differs from our input sizes ( 28 × 28 , 12 × 12 ). In addition to MNIST, we also analyze FMNIST and KMNIST, which have not been considered in previous studies. Our results suggest that among the three datasets, FMNIST has the largest intrinsic dimension. Images in FMNIST cover part of the image in comparison to MNIST and KMNIST; we believe that much of this increase in intrinsic dimension is due to the larger set of possible perturbations of the image within this area.

Wealso explore how resizing the image impacts the data manifold's intrinsic dimension in Figure 10 and Table 3. We see that for MNIST and KMNIST, resizing to a smaller ambient dimension can reduce the intrinsic dimension though at a slightly slower rate than that of the ambient dimension. We note that all the images we consider here have relatively intrinsic dimension, where the effects of resizing may be more prominent.















































































































































































NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In this work, we support formal claims with proofs and include experimental evidence of the main findings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have described limitations of our work at various points. Given our work describes a theoretical setting where we can prove hardness under data manifold assumptions, many of these limitations relate to the assumptions fo the work. For example, in the discussion section, we mention various places where our formal statements do not apply and mention future areas of work to address these limitations. In our experiments, we also outline areas of differences between practice and theory.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All formal statements are proven either in the main text or Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Code is supplied which replicates all experiments. We include additional details of experiments in the main text and in appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is supplied and details are provided in appendix for experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details of optimizers, hyperparameters, etc. are provided in Appendices D and E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For experiments where we train neural networks (Appendix D), we include error bars over random instantiations of the training.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Computational resources were relatively limited for this largely theoretical work. All experiments required only a single GPU (see Appendices D and E).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work fits within the scope of foundational research and is theoretical in nature. We do not introduce any new models or techniques, instead focusing on understanding data manifolds and learning in a more theoretical framework. Although this work can advance our knowledge of machine learning, there are no clear negative societal impacts that directly arise from this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is largely theoretical and apart from some small datasets like MNIST, we do not perform any experiments on data that has a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The real-world datasets used in our paper are cited with their respective licenses explicitly named in the README and are properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIt is widely believed that the low-dimensional structure in natural data is the key to the success of modern machine learning methods. Characterizing this structure and designing algorithms that leverage it, and the complementary problem of understanding when learning under this structure is hard, are thus important problems. In this work the authors provide a hardness result for learning data lying on manifolds using a shallow neural network in the statistical query model. This result is based on reductions from learning Boolean functions, by constructing space-filling manifolds that cover exponentially many quadrants of the Boolean cube.\n\nSTRENGTHS:\nThe construction of manifolds that are hard to learn is novel, and the paper is clearly written.\n\nWEAKNESSES:\nIn terms of novelty and impact, I am concerned that it is not particularly surprising that one can construct space-filling manifolds that are hard to learn, while at the same time other highly structured classes of manifolds can be learned easily. \n\nIn terms of the empirical results, it's unclear to what extent they corroborate the theoretical results. There is no attempt for instance to check empirically whether the scaling of the sample complexities in the hardness result matches the experiment. The novelty of the experiments estimating the intrinsic dimension of standard image datasets is also unclear.\n\nOne weakness of the hardness results is that they only apply to shallow networks. Note that for data lying on curves, there are results proving that networks of sufficient depth and width can generalize when trained with gradient descent [1]. These results essentially show that when depth is large compared to a scale set by the curvature of the class manifolds and the distance between them, they can be distinguished by a neural network trained in the NTK regime (with polynomial training time and sample complexity). It would be interesting to better understand how these results can be reconciled, and I suspect either shallow depth or the additional noise inherent in SQ access make the setting considered in the submission harder. \n\n[1] Wang, Tingran, et al. \"Deep networks provably classify data on curves.\" Advances in neural information processing systems 34 (2021): 28940-28953.\n\nQUESTIONS:\nI'm curious about whether SQ access is crucial for obtaining such hardness results, or whether the authors believe they can be strengthened to the PAC setting.\n\nCould the authors comment about whether their results should still hold for deeper networks. I am particularly curious given the results of [1] mentioned above.\n\nLIMITATIONS:\nLimitations have been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
The paper establishes bounds on the learning hardness for a class of low-dimensional manifolds in the statistical query (SQ) model. It extends existing hardness results for neural network training in Boolean and Gaussian input models to more general geometries. These theoretical results are validated through synthetic experiments. Additionally, the proposed method offers a framework for studying the geometry of data manifolds.

STRENGTHS:
1. The paper is clearly written and easy to read. The authors effectively state their motivations and results before delving into the mathematical details, enhancing the paper's readability.
2. The paper makes theoretical contributions by extending the proofs of hardness in the Statistical Query (SQ) for Boolean and Gaussian input to the geometric setting. This broadens the understanding of the learnability of neural networks beyond traditional data distributions.
3. The theoretical results are validated through experiments, and the paper also provides new inspiration for studying the geometry of data manifolds.

WEAKNESSES:
1. The synthetic experiment itself does not seem convincing enough. The authors only use hypersphere as the input data. It remains unclear whether the results hold for other geometries besides the hypersphere.

QUESTIONS:
The results appear very similar to those in papers discussing the sample complexity of learning a manifold via a ReLU network. This paper, however, reformulates the problem from a Statistical Query (SQ) perspective. How do the authors differentiate their results from those existing studies of sample (or network) complexity?

LIMITATIONS:
See weaknesses. Overall, this is an interesting paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present work considers the hardness of learning under the manifold hypothesis, i.e., the identification of low-dimensional structure by reconstructing low-dimensional manifolds from data and how the latter impacts on the computational complexity of learning algorithms. The authors investigate the data geometry in order to derive minimal assumptions that render the learning problem, e.g., for the class of feedforward neural network, efficiently learnable. Hardness results for a class of low-dimensional manifolds are provided in the framework of statistical query models or under cryptographic hardness assumptions. The latter shows that assuming the manifold hypothesis solely is insufficient for guaranteeing learnability and learning is proved hard under input manifolds of bounded curvature. However, given additional assumptions, e.g., on the volume of the manifold can alleviate the fundamental limitations and is proven to ensure learnability.\nThe authors conclude by providing empirical evidence that illustrate the learnability results through computational experiments on neural network training in the learnable and provably hard regimes.\n\nSTRENGTHS:\nTo the best of my knowledge, this work is novel and can be described as a valuable contribution to the field of manifold learning opening up new research perspectives. It provides interesting insights and results of a theoretical and empirical nature on the problem of the learnability of neural networks from a data geometry perspective by drawing connections to statistical query and cryptographic settings. The results assumptions are based on intrinsic manifold properties which is another plus. Interesting examples for the learnable and provably hard case increase the value of this work.\nThe submission is clearly written, well organized and shows a complete status which made reviewing an enjoyable task.\n\nWEAKNESSES:\nAny of the following concerns can be assumed mediocre and the first two are also discussed transparently.\n\n(1)\nThe sampleable approach restricts the class of included manifolds.\n\n(2)\nThe presented approach requires prior knowledge about the distributions over the sequence of efficiently sampleable manifolds. This is a disadvantage in real-world data problems and is prone to biases, e.g., when the true unknown manifold is inferred from the data.\n\n(3) Only one class, namely feedforward neural networks, is considered.\n\n### Minor\n- l 115: to to\n\nQUESTIONS:\n- The learnability results are provided with respect to the broad class of feedforward neural networks. Can you provide arguments that open or prevent your approach from analyzing other neural network classes, such as the class of residual or recurrent neural networks?\n\n- l. 80: You state that your formal hardness results apply to single-layer networks. Does the hardness result automatically apply to multi-layer networks or is the single-layer regime a necessary restriction?\n\n- Does your analysis capture hyperbolic spaces, i.e., $n-$dimensional Riemannian manifold of constant sectional curvature equal to $-1$ since the latter are shown to be adequate for image learning (see Khrulkov et al.: Hyberbolic Image Embeddings, CVPR, 2020)?\n\nLIMITATIONS:\nLimitations are provided regarding real data manifolds that may fall into the heterogeneous regime and those regarding the sampleable manifold approach that restrict the class of included manifolds. However, I would have liked to see an additional discussion – if any – of the limitations imposed by the chosen class of feedforward neural networks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper relates the sample complexity required for learning a manifold to its curvature – the curvature is characterized by its “reach” that is defined as the minimum distance from a point on the manifold to some point that has multiple nearest neighbors on the manifold. For a manifold in a n-dimensional hypercube they show that if the reach is \\omega(sqrt n) then the manifold can be learned with polynomial sample complexity. On the other hand if it is o(sqrt n) then expotentially many samples may be required. This is proved by constructing a space filling curve that has exponential complexity. They obtaining lower bounds under SQ and cryptographic hardness assumptions.\n\nSTRENGTHS:\nThe paper presents a nice clean connection between the \"worst case\" curvature and sample complexity of learning a manifold. There is a tight split of sqrt(n) for the curvature around which the problem becomes hard.\n\nIt is well written with clearly defined theory that flows smoothly.\n\nWEAKNESSES:\nThat at high curvature exponentially many samples would be required is not too surprising.\n\nThe worst case curvature may not be the main determinant of the hardness of learning a manifold – there may be real world manifolds with very high curvature somewhere in the manifold but still learnable with low sample complexity.\n\nQUESTIONS:\nDo you really need the SQ and the cryptographic hardness assumptions for the exponential lower bound? Cant you just get unconditional hardness bounds as with a space filling curve that goes to a large fraction of the quadrants, the number of degrees of freedom is exp(n) -- so number of such functions is exp(exp(n)). So shouldn't automatically the sample complexity be exp(n)?\n\nThe worst case curvature may not be the main determinant of the hardness of learning a manifold – there may be real world manifolds with very high curvature somewhere in the manifold but still learnable with low sample complexity. For example if you take a V-shaped manifold with the very small angle at the vertex then this has a high curvature. Any thoughts on a tighter characterization of the complexity of learning a manifold?\n\nMight there be other simpler ways of lower bounding the number of functions of high curvature for example maybe if you take polynomials of high degree then the number of coefficients could be exponentially many and the curvature would increase with the degree?\n\nLIMITATIONS:\nYes they have.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}