How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?

1 INTRODUCTION . Deep neural networks have become one of the most important and prevalent machine learning models due to their remarkable power in many real-world applications . However , the success of deep learning has not been well-explained in theory . It remains mysterious why standard optimization algorithms tend to find a globally optimal solution , despite the highly non-convex landscape of the training loss function . Moreover , despite the extremely large amount of parameters , deep neural networks rarely over-fit , and can often generalize well to unseen data and achieve good test accuracy . Understanding these mysterious phenomena on the optimization and generalization of deep neural networks is one of the most fundamental problems in deep learning theory . Recent breakthroughs have shed light on the optimization and generalization of deep neural networks ( DNNs ) under the over-parameterized setting , where the hidden layer width is extremely large ( much larger than the number of training examples ) . It has been shown that with the standard random initialization , the training of over-parameterized deep neural networks can be characterized by a kernel function called neural tangent kernel ( NTK ) ( Jacot et al. , 2018 ; Arora et al. , 2019b ) . In the neural tangent kernel regime ( or lazy training regime ( Chizat et al. , 2019 ) ) , the neural network function behaves similarly as its first-order Taylor expansion at initialization ( Jacot et al. , 2018 ; Lee et al. , 2019 ; Arora et al. , 2019b ; Cao and Gu , 2019 ) , which enables feasible optimization and generalization analysis . In terms of optimization , a line of work ( Du et al. , 2019b ; Allen-Zhu et al. , 2019b ; Zou et al. , 2019 ; Zou and Gu , 2019 ) proved that for sufficiently wide neural networks , ( stochastic ) gradient descent ( GD/SGD ) can successfully find a global optimum of the training loss function . For generalization , Allen-Zhu et al . ( 2019a ) ; Arora et al . ( 2019a ) ; Cao and Gu ( 2019 ) established generalization bounds of neural networks trained with ( stochastic ) gradient descent , and showed that the neural networks can learn target functions in certain reproducing kernel Hilbert space ( RKHS ) or the corresponding random feature function class . Although existing results in the neural tangent kernel regime have provided important insights into the learning of deep neural networks , they require the neural network to be extremely wide . ∗Equal contribution . The typical requirement on the network width is a high degree polynomial of the training sample size n and the inverse of the target error ´1 . As there still remains a huge gap between such network width requirement and the practice , many attempts have been made to improve the overparameterization condition under various conditions on the training data and model initialization ( Oymak and Soltanolkotabi , 2019 ; Zou and Gu , 2019 ; Kawaguchi and Huang , 2019 ; Bai and Lee , 2019 ) . For two-layer ReLU networks , a recent work ( Ji and Telgarsky , 2020 ) showed that when the training data are well separated , polylogarithmic width is sufficient to guarantee good optimization and generalization performances . However , their results can not be extended to deep ReLU networks since their proof technique largely relies on the fact that the network model is 1-homogeneous , which can not be satisfied by DNNs . Therefore , whether deep neural networks can be learned with such a mild over-parameterization is still an open problem . In this paper , we resolve this open problem by showing that polylogarithmic network width is sufficient to learn DNNs . In particular , unlike the existing works that require the DNNs to behave very close to a linear model ( up to some small approximation error ) , we show that a constant linear approximation error is sufficient to establish nice optimization and generalization guarantees for DNNs . Thanks to the relaxed requirement on the linear approximation error , a milder condition on the network width and tighter bounds on the convergence rate and generalization error can be proved . We summarize our contributions as follows : • We establish the global convergence guarantee of GD for training deep ReLU networks based on the so-called NTRF function class ( Cao and Gu , 2019 ) , a set of linear functions over random features . Specifically , we prove that GD can learn deep ReLU networks with width m “ polypRq to compete with the best function in NTRF function class , where R is the radius of the NTRF function class . • We also establish the generalization guarantees for both GD and SGD in the same setting . Specifically , we prove a diminishing statistical error for a wide range of network width m P prΩp1q,8q , while most of the previous generalization bounds in the NTK regime only works in the setting where the network widthm is much greater than the sample size n. Moreover , we establish rOp ´2q rOp ´1q sample complexities for GD and SGD respectively , which are tighter than existing bounds for learning deep ReLU networks ( Cao and Gu , 2019 ) , and match the best results when reduced to the two-layer cases ( Arora et al. , 2019b ; Ji and Telgarsky , 2020 ) . • We further generalize our theoretical analysis to the scenarios with different data separability assumptions in the literature . We show if a large fraction of the training data are well separated , the best function in the NTRF function class with radius R “ rOp1q can learn the training data with error up to . This together with our optimization and generalization guarantees immediately suggests that deep ReLU networks can be learned with network width m “ rΩp1q , which has a logarithmic dependence on the target error and sample size n. Compared with existing results ( Cao and Gu , 2020 ; Ji and Telgarsky , 2020 ) which require all training data points to be separated in the NTK regime , our result is stronger since it allows the NTRF function class to misclassify a small proportion of the training data . For the ease of comparison , we summarize our results along with the most related previous results in Table 1 , in terms of data assumption , the over-parameterization condition and sample complexity . It can be seen that under data separation assumption ( See Sections 4.1 , 4.2 ) , our result improves existing results for learning deep neural networks by only requiring a polylogpn , ´1q network width . Notation . For two scalars a and b , we denote a^ b “ minta , bu . For a vector x P Rd we use } x } 2 to denote its Euclidean norm . For a matrix X , we use } X } 2 and } X } F to denote its spectral norm and Frobenius norm respectively , and denote by Xij the entry of X at the i-th row and j-th column . Given two matrices X and Y with the same dimension , we denote xX , Yy “ ř i , jXijYij . Given a collection of matrices W “ tW1 , ¨ ¨ ¨ , WLu P bLl “ 1Rmlˆm 1 l and a function fpWq over bLl “ 1Rmlˆm 1 l , we define by∇WlfpWq the partial gradient of fpWq with respect to Wl and denote ∇WfpWq “ t∇WlfpWquLl “ 1 . We also denote BpW , τq “ W1 : maxlPrLs } W1l ´Wl } F ď τ ( for τ ě 0 . For two collection of matrices A “ tA1 , ¨ ¨ ¨ , Anu , B “ tB1 , ¨ ¨ ¨ , Bnu , we denote xA , By “ řn i “ 1xAi , Biy and } A } 2F “ řn i “ 1 } Ai } 2F . Algorithm 1 Gradient descent with random initialization Input : Number of iterations T , step size η , training set S “ tpxi , yiqni “ 1u , initialization Wp0q for t “ 1 , 2 , . . . , T do Update Wptq “ Wpt´1q ´ η ¨∇WLSpWpt´1qq . end for Output : Wp0q , . . . , WpT q . Given two sequences txnu and tynu , we denote xn “ Opynq if |xn| ď C1|yn| for some absolute positive constant C1 , xn “ Ωpynq if |xn| ě C2|yn| for some absolute positive constant C2 , and xn “ Θpynq if C3|yn| ď |xn| ď C4|yn| for some absolute constants C3 , C4 ą 0 . We also use rOp¨q , rΩp¨q to hide logarithmic factors inOp¨q and Ωp¨q respectively . Additionally , we denote xn “ polypynq if xn “ OpyDn q for some positive constant D , and xn “ polylogpynq if xn “ polyplogpynqq . 2 PRELIMINARIES ON LEARNING NEURAL NETWORKS . In this section , we introduce the problem setting in this paper , including definitions of the neural network and loss functions , and the training algorithms , i.e. , GD and SGD with random initialization . Neural network function . Given an input x P Rd , the output of deep fully-connected ReLU network is defined as follows , fWpxq “ m1 { 2WLσpWL´1 ¨ ¨ ¨σpW1xq ¨ ¨ ¨ q , where W1 P Rmˆd , W2 , ¨ ¨ ¨ , WL´1 P Rmˆm , WL P R1ˆm , and σpxq “ maxt0 , xu is the ReLU activation function . Here , without loss of generality , we assume the width of each layer is equal to m. Yet our theoretical results can be easily generalized to the setting with unequal width layers , as long as the smallest width satisfies our overparameterization condition . We denote the collection of all weight matrices as W “ tW1 , . . . , WLu . Loss function . Given training dataset txi , yiui “ 1 , ... , n with input xi P Rd and output yi P t´1 , ` 1u , we define the training loss function as LSpWq “ 1 n n ÿ i “ 1 LipWq , where LipWq “ ` ` yifWpxiq ˘ “ log ` 1 ` expp´yifWpxiqq ˘ is defined as the cross-entropy loss . Algorithms . We consider both GD and SGD with Gaussian random initialization . These two algorithms are displayed in Algorithms 1 and 2 respectively . Specifically , the entries in Wp0q1 , ¨ ¨ ¨ , W p0q L´1 are generated independently from univariate Gaussian distributionNp0 , 2 { mq and the entries in Wp0qL are generated independently from Np0 , 1 { mq . For GD , we consider using the full gradient to update the model parameters . For SGD , we use a new training data point in each iteration . Note that our initialization method in Algorithms 1 , 2 is the same as the widely used He initialization ( He et al. , 2015 ) . Our neural network parameterization is also consistent with the parameterization used in prior work on NTK ( Jacot et al. , 2018 ; Allen-Zhu et al. , 2019b ; Du et al. , 2019a ; Arora et al. , 2019b ; Cao and Gu , 2019 ) . Algorithm 2 Stochastic gradient desecent ( SGD ) with random initialization Input : Number of iterations n , step size η , initialization Wp0q for i “ 1 , 2 , . . . , n do Draw pxi , yiq from D and compute the corresponding gradient∇WLipWpi´1qq . Update Wpiq “ Wpi´1q ´ η ¨∇WLipWpi´1qq . end for Output : Randomly choose xW uniformly from tWp0q , . . . , Wpn´1qu .

Summary: The paper studies optimization and generalization properties of deep relu networks trained with (stochastic) gradient descent on the logistic loss in the neural tangent kernel (NTK) regime. By using a new analysis that makes the "linearized" approximation as well as the L2 norm of the model in the approximate "random feature" kernel more explicit, the authors obtain results where the width only depends poly-logarithmically on the number of samples and 1/epsilon, for a test 0-1 loss of epsilon. This improves on previous analysis for deep networks, although it is similar to the two-layer result of Ji & Telgarsky.