Deep Networks and the Multiple Manifold Problem

1 INTRODUCTION . Data in many applications in machine learning and computer vision exhibit low-dimensional structure ( Fig . 1a ) . Although deep neural networks achieve state-of-the-art performance on tasks in these areas , rigorous explanations for their performance remain elusive , in part due to the complex interaction between models , architectures , data , and algorithms in neural network training . There is a need for model problems that capture essential features of applications ( such as low dimensionality ) , but are simple enough to admit rigorous end-to-end performance guarantees . In addition to helping to elucidate the mechanisms by which deep networks succeed , this approach has the potential to clarify the roles of various network properties and how these should reflect the properties of the data . These considerations lead us to formulate the multiple manifold problem ( Fig . 1b ) , a binary classification problem in which the classes are two disjoint submanifolds of the unit sphere Sn0−1 , and the classifier is a deep fully-connected ReLU network of depth L and width n trained on N i.i.d . samples from a distribution supported on the manifolds . The goal is to articulate conditions on the network architecture and number of samples under which the learned classifier provably separates the two manifolds , guaranteeing perfect generalization to unseen data . The difficulty of an instance of the multiple manifold problem is controlled by the dimension of the manifolds d0 , their separation ∆ , and their curvature κ , allowing us to study the constraints imposed by these intrinsic properties of the data on the settings of the neural network ’ s architectural hyperparameters such that the two manifolds can be separated by training with a gradient-based method . Our main result is an analysis of the one-dimensional case of the multiple manifold problem , which reduces the analysis of the gradient descent dynamics to the construction of a certificate—showing that a certain deterministic integral equation involving the network architecture and the structure of the data admits a solution of small norm . We construct such a certificate for the simple geometry in Fig . 3 , guaranteeing generalization in this setting . Theorem 1 ( informal ) . Let d0 = 1 . Suppose a certificate forM exists . Then if the network depth satisfies L ≥ poly ( κ , Cρ , log ( n0 ) ) , the width satisfies n ≥ poly ( L , log ( Ln0 ) ) , and the number of training samples satisfies N ≥ poly ( L ) , randomly-initialized gradient descent on N i.i.d . samples rapidly learns a network that separates the two manifolds with overwhelming probability . The constants Cρ , κ depend only on the data density and the regularity of the manifolds . In addition , if L & ∆−1 , then a certificate exists for the configuration ofM shown in Fig . 3 . Theorem 1 gives a provable generalization guarantee for a model classification problem with deep networks on structured data that depends only on the architectural hyperparameters and properties of the data . In addition , it provides an interpretable tradeoff between the architectural settings necessary to separate the two manifolds : the network depth needs to be set according to the intrinsic difficulty of the problem , and the network width needs to grow with the depth . Our analysis gives further insight into the independent roles played by each of these parameters in solving the problem , with the depth acting as a ‘ fitting resource ’ , making the network ’ s output more regular and easier to change , and the width acting as a ‘ statistical resource ’ , granting concentration of the network over the random initialization around a well-behaved object that we can analyze . Moreover , the sample complexity of Theorem 1 is dictated by the intrinsic difficulty of the problem instance which is set by the geometry of the data . As a consequence , we avoid any dependence of the width of the network on the number of samples , which is common in deep network convergence results in the literature ( e.g . ( Allen-Zhu et al. , 2019b ; Du et al. , 2019 ) , ( Chen et al. , 2021 , Theorem 3.4 ) ) . As is the case in practice , given a fixed architecture , more data doesn ’ t have a detrimental effect on fitting 1 . Theorem 1 is modular , in the sense that a generalization guarantee is ensured for any geometry for which one can construct a certificate . The key to our approach will be to approximate the gradient 1When using data augmentation , for example , the number of samples is effectively infinite yet highly structured , enabling convergence and generalization . descent dynamics with a linear discrete dynamical system defined in terms of the so-called neural tangent kernel Θ ( x , x′ ) defined on the manifolds . Due to the structure in the data , diagonalizing the operator corresponding to this kernel is intractable in general , but we show that constructing a certificate—arguably an easier task , because it requires producing a bound on the norm of a solution to an equation rather than producing the solution itself—suffices to guarantee that the error decreases rapidly during training given a suitably structured network . We summarize the primary contributions of this work below . • Generalization in deep networks : There are few generalization results for deep networks trained efficiently with gradient descent available in the literature.2 Theorem 1 provides such a guarantee that does not depend on any property of the trained network ( e.g. , norms of final weights ) that is not readily available before training . In this context , the certificate condition is equivalent to the initial network function having a controlled norm in a certain RKHS ; this condition is natural in the training regime we consider , and appears ubiquitously in works on generalization in shallower networks ( Ghorbani et al. , 2020 ; Ji & Telgarsky , 2020 ; Nitanda & Suzuki , 2021 ) . • Uniform concentration of the neural tangent kernel for deep ReLU networks : As an intermediate step in the proof of Theorem 1 , we establish essentially optimal rates of uniform concentration for the neural tangent kernel of an arbitrarily deep network ( Theorem 2 ) using martingale concentration , where we require the width to grow only linearly with the depth . We expect this martingale approach to be applicable to essentially any other compositionally-structured network architecture . Our uniform result generalizes prior results on pointwise concentration ( Arora et al. , 2019b ; Allen-Zhu et al. , 2019b ) , analogous to our Theorem B.3 , and proves useful in establishing generalization . • Strong regularity estimates for random ReLU networks : As a further consequence of the uniform concentration framework we have developed , we obtain depth-logarithmic Lipschitz estimates for random ReLU networks of arbitrary depth and linear width , as well as ( for still wider networks ) a uniform approximation for the network output by a constant which improves with depth , both with overwhelming probability ( Section 3.3 ) . We also control the evolution of the Lipschitz constant during NTK regime training ( Lemma B.7 ) , showing that it scales polynomially in the depth . These results may be of interest in applications where guaranteeing a Lipschitz property for networks is important , such as GAN training ( Miyato et al. , 2018 ) or denoising ( Ryu et al. , 2019 ; Sun et al. , 2020 ) . 1.1 RELATED WORK . Deep networks and low-dimensional structure . The notion of modeling data as low-dimensional submanifolds has been widely considered in the context of clustering ( Wang et al. , 2015 ) and manifold learning ( Donoho & Grimes , 2005 ; Fefferman et al. , 2016 ) . Goldt et al . ( 2020 ) independently proposed the “ hidden manifold model ” , a model problem for learning shallow neural networks for binary classification of structured data with motivations very similar to ours and which admits a mean-field analysis ( Gerace et al. , 2020 ) . The data model consists of gaussian samples from a low-dimensional subspace passed through a nonlinear function acting coordinatewise in the standard basis ; although this models statistical variations around a base domain , a feature of real data that the model we study here lacks , we believe that the study of an arbitrary density supported on two Riemannian manifolds lends our data model increased structural generality . In the context of kernel regression with the kernel given by the NTK of a two-layer neural network , Ghorbani et al . ( 2020 ) study a data generating model that consists of uniform samples from a low-dimensional subsphere corrupted additively by independent uniform samples from a subsphere in the orthogonal complement , and a target mapping that depends only on the low-dimensional part . The authors obtain asymptotic generalization guarantees for this data model that reveal conditions under which the corruption degrades the performance of neural tangent methods . 2The closest result we are aware of is ( Chen et al. , 2021 , Theorem 3.4 ) ; this result involves a-priori assumptions on the trained network weights , which are only resolved for two-layer networks , and entails an unnatural relationship between n and N and a possible exponential dependence of N on L , which Theorem 1 avoids . Analyses of neural network training . To reason analytically about the complicated training process , we adopt the neural tangent kernel approach ( Jacot et al. , 2018 ) . The first works to instantiate these ideas in a nonasymptotic setting obtained convergence guarantees for training deep neural networks on finite datasets ( Allen-Zhu et al. , 2019b ; Du et al. , 2019 ) . By exploiting more structure in the data , generalization results have been obtained ( Allen-Zhu et al. , 2019a ; Arora et al. , 2019a ; Ji & Telgarsky , 2020 ; Oymak et al. , 2019 ; Cao & Gu , 2019 ; Suzuki , 2020 ; Li et al. , 2020 ; AllenZhu & Li , 2020 ) that apply to shallow networks , teacher-student learning scenarios , and/or hold conditional on the existence of certain small-norm interpolators . Other works have obtained generalization guarantees using generalization bounds for kernel methods ( Ghorbani et al. , 2019 ; Liang et al. , 2020 ; Ghorbani et al. , 2020 ; Montanari & Zhong , 2020 ) using the fact that the linearized predictor in the NTK regime can be linked to a kernel method ( Arora et al. , 2019b ) . A parallel line of works ( Mei et al. , 2018 ; Tzen & Raginsky , 2020 ; Mei et al. , 2019 ; Chizat & Bach , 2020 ; Fang et al. , 2020 ) approach the problem by studying an infinite-width limit of neural network training that yields a different training dynamics . Approaches of this type are of interest because there is no restriction to short-time dynamics , and the limit of the dynamics can often be characterized in terms of a well-structured object , such as a max-margin classifier ( Chizat & Bach , 2020 ) . On the other hand , it is often difficult to prove finite-time convergence to the limit . 2 PROBLEM FORMULATION AND MAIN RESULTS . 2.1 DATA MODEL AND NETWORK DEFINITIONS . We consider data supported on the union of two class manifoldsM =M+ ∪M− , whereM+ and M− are two disjoint smooth regular simple curves taking values in Sn0−1 , with n0 ≥ 3 . We denote the data measure supported onM that generates our samples as µ∞ , with corresponding density ρ , and write ρmin = infx∈M ρ ( x ) and ρmax = supx∈M ρ ( x ) . We denote by κ a uniform bound on the ( extrinsic ) curvature of the two curves , we write ∆ = minx∈M+ , x′∈M− ∠ ( x , x ′ ) for the separation between class manifolds , where ∠ ( x , x′ ) = cos-1〈x , x′〉 for unit vectors , and to have a quantitative characterization of ‘ how simple ’ the curves are , we assume there exist constants 0 < cλ ≤ 1 , Kλ ≥ 1 such that for every 0 < s ≤ cλ/κ and every x , x′ in a common connected component of M , one has distM ( x , x′ ) ≤ Kλs if ∠ ( x , x′ ) ≤ s , where distM denotes the Riemannian distance . Our target function is the signed indicator for each class manifold f ? ( x ) = 1M+ ( x ) − 1M− ( x ) . The model we consider is a fully-connected neural network with ReLU activations and access to i.i.d . samples from µ∞ and their corresponding labels . We parameterize our neural network with weights W 1 ∈ Rn×n0 , W ` ∈ Rn×n if ` ∈ { 2 , . . . , L } , and WL+1 ∈ R1×n , which we collect as θ = ( W 1 , . . . , WL+1 ) , and write the iterates of the forward pass as α ` θ ( x ) = [ W ` α ` −1θ ( x ) ] + for ` = 1 , . . . , L with α0θ ( x ) = x , which we also refer to as features or activations , with the network output written as fθ ( x ) = WL+1αLθ ( x ) , and the prediction error as ζθ ( x ) = fθ ( x ) − f ? ( x ) . For an i.i.d . sample ( x1 , . . . , xN ) from µ∞ , we write µN = 1N ∑N i=1 δxi for the empirical measure associated to the sample , and we consider the training objective LµN ( θ ) = 12 ∫ M ( ζθ ( x ) ) 2 dµN ( x ) , i.e . the empirical risk evaluated with the square loss . We train with vanilla gradient descent with constant step size τ > 0 : after randomly initializing the parameters θN0 as W ` ∼i.i.d . N ( 0 , 2/n ) if ` ∈ [ L ] andWL+1 ∼i.i.d . N ( 0 , 1 ) we consider the sequence of iterates θNk+1 = θNk −τ∇̃LµN ( θNk ) , where ∇̃LµN represents a ‘ formal gradient ’ of the empirical loss , which we define in detail in Appendix A.1 . We say the parameters obtained at iteration k of gradient descent separate the manifolds M if the classifier implemented by the neural network with the parameters θNk labels the two manifolds correctly , i.e . if f ? ( x ) sign ( fθNk ( x ) ) = 1 for every x ∈ M. As a shorthand , we will denote quantities evaluated at θNk with a subscript k ; an omitted subscript will denote k = 0 , and we will write explicitly θ0 = θN0 . Additional notation is provided in Appendix A.5.1 .

Summary: The authors consider a binary classification task. As a model the authors use a deep fully-connected neural network and train it to separate the submanifolds, representing different classes. They assume that sub-manifolds belong to the unit sphere. Also, the authors restrict their analysis to a one-dimensional case. The main claim is that by increasing depth we can improve model generalization of a network, trained by SGD.