Deep Networks and the Multiple Manifold Problem

1 INTRODUCTION . Data in many applications in machine learning and computer vision exhibit low-dimensional structure ( Fig . 1a ) . Although deep neural networks achieve state-of-the-art performance on tasks in these areas , rigorous explanations for their performance remain elusive , in part due to the complex interaction between models , architectures , data , and algorithms in neural network training . There is a need for model problems that capture essential features of applications ( such as low dimensionality ) , but are simple enough to admit rigorous end-to-end performance guarantees . In addition to helping to elucidate the mechanisms by which deep networks succeed , this approach has the potential to clarify the roles of various network properties and how these should reflect the properties of the data . These considerations lead us to formulate the multiple manifold problem ( Fig . 1b ) , a binary classification problem in which the classes are two disjoint submanifolds of the unit sphere Sn0−1 , and the classifier is a deep fully-connected ReLU network of depth L and width n trained on N i.i.d . samples from a distribution supported on the manifolds . The goal is to articulate conditions on the network architecture and number of samples under which the learned classifier provably separates the two manifolds , guaranteeing perfect generalization to unseen data . The difficulty of an instance of the multiple manifold problem is controlled by the dimension of the manifolds d0 , their separation ∆ , and their curvature κ , allowing us to study the constraints imposed by these intrinsic properties of the data on the settings of the neural network ’ s architectural hyperparameters such that the two manifolds can be separated by training with a gradient-based method . Our main result is an analysis of the one-dimensional case of the multiple manifold problem , which reduces the analysis of the gradient descent dynamics to the construction of a certificate—showing that a certain deterministic integral equation involving the network architecture and the structure of the data admits a solution of small norm . We construct such a certificate for the simple geometry in Fig . 3 , guaranteeing generalization in this setting . Theorem 1 ( informal ) . Let d0 = 1 . Suppose a certificate forM exists . Then if the network depth satisfies L ≥ poly ( κ , Cρ , log ( n0 ) ) , the width satisfies n ≥ poly ( L , log ( Ln0 ) ) , and the number of training samples satisfies N ≥ poly ( L ) , randomly-initialized gradient descent on N i.i.d . samples rapidly learns a network that separates the two manifolds with overwhelming probability . The constants Cρ , κ depend only on the data density and the regularity of the manifolds . In addition , if L & ∆−1 , then a certificate exists for the configuration ofM shown in Fig . 3 . Theorem 1 gives a provable generalization guarantee for a model classification problem with deep networks on structured data that depends only on the architectural hyperparameters and properties of the data . In addition , it provides an interpretable tradeoff between the architectural settings necessary to separate the two manifolds : the network depth needs to be set according to the intrinsic difficulty of the problem , and the network width needs to grow with the depth . Our analysis gives further insight into the independent roles played by each of these parameters in solving the problem , with the depth acting as a ‘ fitting resource ’ , making the network ’ s output more regular and easier to change , and the width acting as a ‘ statistical resource ’ , granting concentration of the network over the random initialization around a well-behaved object that we can analyze . Moreover , the sample complexity of Theorem 1 is dictated by the intrinsic difficulty of the problem instance which is set by the geometry of the data . As a consequence , we avoid any dependence of the width of the network on the number of samples , which is common in deep network convergence results in the literature ( e.g . ( Allen-Zhu et al. , 2019b ; Du et al. , 2019 ) , ( Chen et al. , 2021 , Theorem 3.4 ) ) . As is the case in practice , given a fixed architecture , more data doesn ’ t have a detrimental effect on fitting 1 . Theorem 1 is modular , in the sense that a generalization guarantee is ensured for any geometry for which one can construct a certificate . The key to our approach will be to approximate the gradient 1When using data augmentation , for example , the number of samples is effectively infinite yet highly structured , enabling convergence and generalization . descent dynamics with a linear discrete dynamical system defined in terms of the so-called neural tangent kernel Θ ( x , x′ ) defined on the manifolds . Due to the structure in the data , diagonalizing the operator corresponding to this kernel is intractable in general , but we show that constructing a certificate—arguably an easier task , because it requires producing a bound on the norm of a solution to an equation rather than producing the solution itself—suffices to guarantee that the error decreases rapidly during training given a suitably structured network . We summarize the primary contributions of this work below . • Generalization in deep networks : There are few generalization results for deep networks trained efficiently with gradient descent available in the literature.2 Theorem 1 provides such a guarantee that does not depend on any property of the trained network ( e.g. , norms of final weights ) that is not readily available before training . In this context , the certificate condition is equivalent to the initial network function having a controlled norm in a certain RKHS ; this condition is natural in the training regime we consider , and appears ubiquitously in works on generalization in shallower networks ( Ghorbani et al. , 2020 ; Ji & Telgarsky , 2020 ; Nitanda & Suzuki , 2021 ) . • Uniform concentration of the neural tangent kernel for deep ReLU networks : As an intermediate step in the proof of Theorem 1 , we establish essentially optimal rates of uniform concentration for the neural tangent kernel of an arbitrarily deep network ( Theorem 2 ) using martingale concentration , where we require the width to grow only linearly with the depth . We expect this martingale approach to be applicable to essentially any other compositionally-structured network architecture . Our uniform result generalizes prior results on pointwise concentration ( Arora et al. , 2019b ; Allen-Zhu et al. , 2019b ) , analogous to our Theorem B.3 , and proves useful in establishing generalization . • Strong regularity estimates for random ReLU networks : As a further consequence of the uniform concentration framework we have developed , we obtain depth-logarithmic Lipschitz estimates for random ReLU networks of arbitrary depth and linear width , as well as ( for still wider networks ) a uniform approximation for the network output by a constant which improves with depth , both with overwhelming probability ( Section 3.3 ) . We also control the evolution of the Lipschitz constant during NTK regime training ( Lemma B.7 ) , showing that it scales polynomially in the depth . These results may be of interest in applications where guaranteeing a Lipschitz property for networks is important , such as GAN training ( Miyato et al. , 2018 ) or denoising ( Ryu et al. , 2019 ; Sun et al. , 2020 ) . 1.1 RELATED WORK . Deep networks and low-dimensional structure . The notion of modeling data as low-dimensional submanifolds has been widely considered in the context of clustering ( Wang et al. , 2015 ) and manifold learning ( Donoho & Grimes , 2005 ; Fefferman et al. , 2016 ) . Goldt et al . ( 2020 ) independently proposed the “ hidden manifold model ” , a model problem for learning shallow neural networks for binary classification of structured data with motivations very similar to ours and which admits a mean-field analysis ( Gerace et al. , 2020 ) . The data model consists of gaussian samples from a low-dimensional subspace passed through a nonlinear function acting coordinatewise in the standard basis ; although this models statistical variations around a base domain , a feature of real data that the model we study here lacks , we believe that the study of an arbitrary density supported on two Riemannian manifolds lends our data model increased structural generality . In the context of kernel regression with the kernel given by the NTK of a two-layer neural network , Ghorbani et al . ( 2020 ) study a data generating model that consists of uniform samples from a low-dimensional subsphere corrupted additively by independent uniform samples from a subsphere in the orthogonal complement , and a target mapping that depends only on the low-dimensional part . The authors obtain asymptotic generalization guarantees for this data model that reveal conditions under which the corruption degrades the performance of neural tangent methods . 2The closest result we are aware of is ( Chen et al. , 2021 , Theorem 3.4 ) ; this result involves a-priori assumptions on the trained network weights , which are only resolved for two-layer networks , and entails an unnatural relationship between n and N and a possible exponential dependence of N on L , which Theorem 1 avoids . Analyses of neural network training . To reason analytically about the complicated training process , we adopt the neural tangent kernel approach ( Jacot et al. , 2018 ) . The first works to instantiate these ideas in a nonasymptotic setting obtained convergence guarantees for training deep neural networks on finite datasets ( Allen-Zhu et al. , 2019b ; Du et al. , 2019 ) . By exploiting more structure in the data , generalization results have been obtained ( Allen-Zhu et al. , 2019a ; Arora et al. , 2019a ; Ji & Telgarsky , 2020 ; Oymak et al. , 2019 ; Cao & Gu , 2019 ; Suzuki , 2020 ; Li et al. , 2020 ; AllenZhu & Li , 2020 ) that apply to shallow networks , teacher-student learning scenarios , and/or hold conditional on the existence of certain small-norm interpolators . Other works have obtained generalization guarantees using generalization bounds for kernel methods ( Ghorbani et al. , 2019 ; Liang et al. , 2020 ; Ghorbani et al. , 2020 ; Montanari & Zhong , 2020 ) using the fact that the linearized predictor in the NTK regime can be linked to a kernel method ( Arora et al. , 2019b ) . A parallel line of works ( Mei et al. , 2018 ; Tzen & Raginsky , 2020 ; Mei et al. , 2019 ; Chizat & Bach , 2020 ; Fang et al. , 2020 ) approach the problem by studying an infinite-width limit of neural network training that yields a different training dynamics . Approaches of this type are of interest because there is no restriction to short-time dynamics , and the limit of the dynamics can often be characterized in terms of a well-structured object , such as a max-margin classifier ( Chizat & Bach , 2020 ) . On the other hand , it is often difficult to prove finite-time convergence to the limit . 2 PROBLEM FORMULATION AND MAIN RESULTS . 2.1 DATA MODEL AND NETWORK DEFINITIONS . We consider data supported on the union of two class manifoldsM =M+ ∪M− , whereM+ and M− are two disjoint smooth regular simple curves taking values in Sn0−1 , with n0 ≥ 3 . We denote the data measure supported onM that generates our samples as µ∞ , with corresponding density ρ , and write ρmin = infx∈M ρ ( x ) and ρmax = supx∈M ρ ( x ) . We denote by κ a uniform bound on the ( extrinsic ) curvature of the two curves , we write ∆ = minx∈M+ , x′∈M− ∠ ( x , x ′ ) for the separation between class manifolds , where ∠ ( x , x′ ) = cos-1〈x , x′〉 for unit vectors , and to have a quantitative characterization of ‘ how simple ’ the curves are , we assume there exist constants 0 < cλ ≤ 1 , Kλ ≥ 1 such that for every 0 < s ≤ cλ/κ and every x , x′ in a common connected component of M , one has distM ( x , x′ ) ≤ Kλs if ∠ ( x , x′ ) ≤ s , where distM denotes the Riemannian distance . Our target function is the signed indicator for each class manifold f ? ( x ) = 1M+ ( x ) − 1M− ( x ) . The model we consider is a fully-connected neural network with ReLU activations and access to i.i.d . samples from µ∞ and their corresponding labels . We parameterize our neural network with weights W 1 ∈ Rn×n0 , W ` ∈ Rn×n if ` ∈ { 2 , . . . , L } , and WL+1 ∈ R1×n , which we collect as θ = ( W 1 , . . . , WL+1 ) , and write the iterates of the forward pass as α ` θ ( x ) = [ W ` α ` −1θ ( x ) ] + for ` = 1 , . . . , L with α0θ ( x ) = x , which we also refer to as features or activations , with the network output written as fθ ( x ) = WL+1αLθ ( x ) , and the prediction error as ζθ ( x ) = fθ ( x ) − f ? ( x ) . For an i.i.d . sample ( x1 , . . . , xN ) from µ∞ , we write µN = 1N ∑N i=1 δxi for the empirical measure associated to the sample , and we consider the training objective LµN ( θ ) = 12 ∫ M ( ζθ ( x ) ) 2 dµN ( x ) , i.e . the empirical risk evaluated with the square loss . We train with vanilla gradient descent with constant step size τ > 0 : after randomly initializing the parameters θN0 as W ` ∼i.i.d . N ( 0 , 2/n ) if ` ∈ [ L ] andWL+1 ∼i.i.d . N ( 0 , 1 ) we consider the sequence of iterates θNk+1 = θNk −τ∇̃LµN ( θNk ) , where ∇̃LµN represents a ‘ formal gradient ’ of the empirical loss , which we define in detail in Appendix A.1 . We say the parameters obtained at iteration k of gradient descent separate the manifolds M if the classifier implemented by the neural network with the parameters θNk labels the two manifolds correctly , i.e . if f ? ( x ) sign ( fθNk ( x ) ) = 1 for every x ∈ M. As a shorthand , we will denote quantities evaluated at θNk with a subscript k ; an omitted subscript will denote k = 0 , and we will write explicitly θ0 = θN0 . Additional notation is provided in Appendix A.5.1 .

Summary: The paper studies the conditions for a deep fully-connected network to separate low-dimensional data classes. A binary classification setting is considered, where the two classes are modelled as two different manifolds. The manifolds are assumed to be one-dimensional for the ease of analysis. It is shown that the network depth should be sufficiently large so as to adapt to the geometrical properties of data (e.g. the manifold curvature); the network width should increase polynomially with the network depth; and the number of data samples should also scale at a polynomial rate with the network depth. The authors show that if these conditions are met, with high probability a randomly initialized network converges to a classifier that separates the two class manifolds. The proof technique relies on conditioning the network parameters of the l-th layer on the parameters of the previous layers using a Martingale model, which gives sharp concentration guarantees.