Nonparametric Learning of Two-Layer ReLU Residual Units

[ ( A∗x ) + + x ] , where ground-truth network parametersA∗ ∈ Rd×d represent a nonnegative full-rank matrix andB∗ ∈ Rm×d is full-rank with m ≥ d and for c ∈ Rd , [ c+ ] i = max { 0 , ci } . We design layer-wise objectives as functionals whose analytic minimizers express the exact ground-truth network in terms of its parameters and nonlinearities . Following this objective landscape , learning residual units from finite samples can be formulated using convex optimization of a nonparametric function : for each layer , we first formulate the corresponding empirical risk minimization ( ERM ) as a positive semi-definite quadratic program ( QP ) , then we show the solution space of the QP can be equivalently determined by a set of linear inequalities , which can then be efficiently solved by linear programming ( LP ) . We further prove the statistical strong consistency of our algorithm , and demonstrate its robustness and sample efficiency through experimental results . 1 INTRODUCTION . Neural networks have achieved remarkable success in various fields such as computer vision ( LeCun et al. , 1998 ; Krizhevsky et al. , 2012 ; He et al. , 2016a ) and natural language processing ( Kim , 2014 ; Sutskever et al. , 2014 ) . This success is largely due to the strong expressive power of neural networks ( Bengio & Delalleau , 2011 ) , where nonlinear activation units , such as rectified linear units ( ReLU ) ( Nair & Hinton , 2010 ) and hyperbolic tangents ( tanh ) play a vital role to ensure the large learning capacity of the networks ( Maas et al. , 2013 ) . However , the nonlinearity of neural networks makes them significantly more difficult to train than linear models ( Livni et al. , 2014 ) . Therefore , with the development of neural network applications , finding efficient algorithms with provable properties to train such nontrivial neural networks has become an important and a relatively new goal . A ReLU B Residual networks , or ResNets ( He et al. , 2016a ) , are a class of deep neural networks that adopt skip connections to feed values between nonadjacent layers , where skipped layers may contain nonlinearities in between . Without loss of expressivity , ResNets avoid the vanishing gradient problem by directly passing gradient information from previous layers to current layers where otherwise gradients might vanish without skipping . In practice , ResNets have shown strong learning efficiency in several tasks , e.g . achieving at least 93 % test accuracy on CIFAR-10 classification , lowering single-crop error to 20.1 % on the 1000-class ImageNet dataset ( Russakovsky et al. , 2015 ; He et al. , 2016b ) . Common ResNets are often aggregated by many repeated shallow networks , where each network acts as a minimal unit with this kind of skip propagation , named a residual unit ( He et al. , 2016b ) . Given the flexibility and simplicity of residual units , much theoretical work has been devoted to study them and develop training algorithms for them in a way that sidesteps from the standard backpropagation regime and provides guarantees on the quality of estimation ( see subsection 1.1 ) . In this paper , we propose algorithms that can learn a general class of single-skip twolayer residual units with ReLU activation as shown on the right by the equation : y = B [ ( Ax ) + + x ] , where for a scalar c , c+ = max { 0 , c } ( for a vector , this maximization is applied coordinate-wise ) , x is a random vector as the network input with support space Rd , andA ∈ Rd×d andB ∈ Rm×d are weight matrices of layer 1 and layer 2 , respectively . Compared to previous work ( Ge et al. , 2018 ; 2017 ; Zhang et al. , 2018 ; Wu et al. , 2019 ; Tian , 2017 ; Du et al. , 2017 ; Brutzkus & Globerson , 2017 ; Soltanolkotabi , 2017 ; Li & Yuan , 2017 ; Zhong et al. , 2017 ) , the introduction of residual connections simplifies the recovery of the network parameters by removing the permutation and scaling invariance . However , the naive mean square error minimization used in estimating the parameters remains nonconvex . Unlike most previous work , we do not assume a specific input distribution nor that the distribution is symmetric ( Ge et al. , 2018 ; Du & Goel , 2018 ) . We show that under regularity conditions on the weights of the residual unit , the problem of learning the unit can be formulated through quadratic programming ( QP ) . We use nonparametric estimation ( Guntuboyina et al. , 2018 ) to estimate the ReLU function values in the networks . We further rewrite our constructed quadratic programs to linear programs ( LPs ) . The LP formulation is simpler to optimize and has the same solution space as the QP for the network parameters . 1.1 RELATED WORK . Provable learning of neural networks has become an important topic of research . Arora et al . ( 2014 ) recover a multi-layer generative network with sparse connections and Livni et al . ( 2014 ) study the learning of multi-layer neural networks with polynomial activation . Goel et al . ( 2018 ) learns a one-layer convolution network with a perceptron-like rule . They prove the correctness of an iterative algorithm for exact recovery of the target network . Rabusseau et al . ( 2019 ) describe a spectral algorithm for two-layer linear networks . Recent work has connected optimization and two-layer network learning ( Ergen & Pilanci , 2021 ; Sahiner et al. , 2020 ; Ergen & Pilanci , 2020 ; Pilanci & Ergen , 2020 ) and has showed how to optimize networks layer by layer ( Belilovsky et al. , 2019 ) . For learning a one-layer ReLU network , Wu et al . ( 2019 ) optimize the norm and direction of neural network weight vectors separately , and Zhang et al . ( 2018 ) use gradient descent with a specific initialization . For learning a two-layer ReLU network , Ge et al . ( 2017 ) redesign the optimization landscape such that it is more amenable to theoretical analysis and Ge et al . ( 2018 ) use a momentbased method for estimating a neural network Janzamin et al . ( 2015 ) . Many others have studied ReLU networks in various settings ( Tian , 2017 ; Du et al. , 2017 ; Brutzkus & Globerson , 2017 ; Soltanolkotabi , 2017 ; Li & Yuan , 2017 ; Zhong et al. , 2017 ; Goel & Klivans , 2017 ) . The study of ReLU networks with two hidden layers has also been gaining attention ( Goel & Klivans , 2017 ; Allen-Zhu et al. , 2019 ) , with a focus on PAC learning ( Valiant , 1984 ) . In relation to our work , Allen-Zhu & Li ( 2019 ) examined the PAC-learnable function of a specific three-layer neural network with residual connections . Their work differs from ours in two aspects . First , their learnable functions include a smaller ( in comparison to the student network ) three-layer residual network . Second , the assumptions they make on their three-layer model are rather different than ours . In relation to nonparametric estimation , Guntuboyina et al . ( 2018 ) treat the final output of a shaperestricted regressor as a parameter , placing some restrictions on the type of function that can be estimated ( such as convexity ) . They provide solutions for estimation with isotonic regression ( Brunk , 1955 ; Ayer et al. , 1955 ; van Eeden , 1956 ) , convex regression ( Seijo et al. , 2011 ) , shape-restricted additive models ( Meyer , 2013 ; Chen & Samworth , 2016 ) and shape-restricted single index models ( Kakade et al. , 2011 ; Kuchibhotla et al. , 2017 ) . 1.2 MAIN RESULTS . We design quadratic objective functionals with a linear bounded feasible domain that take network parameters and functions as variables to estimate the ground-truth network parameters and nonlinearities . The values of the objectives are moments over the input distribution . Thm . 1.1 summarizes the landscapes of the objectives . Theorem 1.1 ( objective landscape , informal ) . Suppose a ground-truth residual unit has nondegenerate weights in both layers and nonnegative weights in layer 1 . Then there exist quadratic functionals defined in linear-constrained domains whose minimizers a ) are unique , and are the exact ground-truth , or b ) are not unique , but can be adjusted to the exact ground-truth . In practice , the exact moments over unobserved variables are not available . We can only construct the empirical risk minimization ( ERM ) of the moment-valued objectives by generated samples . With functions as variables in moment-valued objectives being optimized nonparametrically , the empirical objectives become quadratic functions with linear constraints ( QP ) . We further show the convexity of our QP which guarantees its solution in polynomial time w.r.t . sample size and dimension . With the solution to the QP available , the strong consistency of our network learner is guaranteed : Theorem 1.2 ( strong consistency , informal ) . Suppose samples are generated by a ground-truth residual unit that has nondegenerate weights in both layers and nonnegative weights in layer 1 . Then there exists an algorithm that learns a network a.s.−−→ the exact ground-truth as sample size grows . Roadmap : AssumeA andB are student network weights of layer 1 and 2 . We first give a warm-up vanilla linear regression approach only knowing A is in Rd×d and B is in Rm×d ( section 3 ) . We then move to the details of our nonparametric learning for layer 2 ( section 4 ) , and similarly , layer 1 ( section 5 ) . We formalize Thm . 1.1 , showing how nonparametric learning allows us to select the values of A and B from reduced spaces that are seeded by A∗ and B∗ , under which LR is faster than vanilla LR . In section 6 , we describe the strong consistency of our methods for respective layers , formalizing Thm . 1.2 using the continuous mapping theorem ( Mann & Wald , 1943 ) . 2 PRELIMINARIES . We describe the notations used in this paper , introduce the model and its underlying assumptions , and state conditions which simplify the problem but which can be removed without loss of learnability . Notation The ReLU residual units we use take a vector x ∈ Rd as input and return a vector y ∈ Rm . We use A∗ ∈ Rd×d and B∗ ∈ Rm×d to denote the ground-truth network parameters for layer 1 and 2 , respectively . We use circumflex to denote predicted terms ( e.g . an empirical objective function f̂ , estimated layer 1 weights Â ) . We use n to denote the number of i.i.d . samples available for the training algorithm , { x ( i ) , y ( i ) } to denote the i-th sample drawn . For an integer k , We define [ k ] to be { 1 , 2 , . . . , k } , and e ( j ) as the standard basis vector with a 1 at position j . All scalar-based operators are element wise in the case of vectors or matrices unless specified otherwise . We use x and y to refer to the input and output vectors , respectively , as random variables . Linear Regression : Linear regression ( LR ) in this paper refers to the problem of estimating L for unbiased model y = Lx . In this case , estimation is done by minimizing the empirical risk R̂ ( L ) = 12n ∑ i∈ [ n ] ∥∥Lx ( i ) − y ( i ) ∥∥2 – using linear least squares ( LLS ) . Its solution has a closed form which we use as given . We refer the reader to Hamilton ( 1994 ) for more information . Models , Assumptions and Conditions Following previous work ( Livni et al. , 2014 ) , we assume a given neural network structure specifies a hypothesis class that contains all the networks conforming to this structure . Learning such a class means using training samples to find a set of weights such that the neural networks predictions generalize well to unseen samples , where both the training and the unseen samples are drawn from an unobserved ground-truth distribution . In this paper , the hypothesis class is given by ReLU residual units and we assume it has sufficient expressive power to fit the ground-truth model . More specifically , we discuss the realizable case of learning , in which the ground-truth model is set to be a residual unit taken from the hypothesis class with the form : y = B∗ [ ( A∗x ) + + x ] , ( 1 ) and is used to draw samples for learning . Unlike other multi-layer ReLU-affine models which do not apply skip connections , we can not permute the weight matrices of the residual unit and retain the same function because of skip-adding x ( it breaks symmetry ) . This helps us circumvent issues of identifiability1 , and allows us to precisely estimate the ground-truth weight matricesA∗ andB∗ . Our general approach for residual unit layer 2 learns a scaled ground-truth weight matrix that also minimizes the layer 2 objective . The existence of such scaled equivalence of our layer 2 approach comes from what is defined below . 1Here , we are referring to the ability to identify the true model parameters using infinite samples . Definition 2.1 ( component-wise scale transformation ) . A matrix A ∈ Rd×d is said to be a scale transformation w.r.t . the j-th component if ( Aj , : ) > = Aj , j · e ( j ) . Additionally , estimation is more complex when the layer 2 weightsB∗ is nonsquare . For simplicity of our algorithm presentation for layer 2 , we stick to Cond . 2.1 in the following sections2 . Condition 2.1 ( layer 2 objective minimizer unique ) . A∗ is not a scale transformation w.r.t . any components andB∗ is a square matrix , i.e . m = d .

Summary: This paper gives an algorithm for learning two-layer neural networks with ReLU activation (realizable case). The network also consists of skip connections to avoid the problems of identifiability. As a warm up, the authors first give a simple algorithms based on linear regression, whenever enough samples could be observed with all-positive and all-negative entries.   They start with an algorithm for recovering the layer two parameters. For this step, they use the $\ell_2$ loss and show that its minimizer is unique and matches with the ground truth parameters under certain condition. Subsequently, the authors replace the expectation in the objective with an average over the samples (based on the empirical risk minimization principle), which they could write as a convex quadratic program without making any parametric assumption. Finally, they show that the quadratic program can be equivalently written as a linear program, which lets them recover the parameters of the second layer (in fact the inverse matrix of this weight matrix).  Once layer 2 is learnt, the original problem reduces to that of learning the single layer non-linearity. They write the objective as a sum of weight estimator and a function estimator. They show that the minimizer of this loss is achieved when the estimated weight matrix is a diagonal matrix times the ground truth and the function measures the difference between the true nonlinear function and the scaled wright matrix. Again, they learn the function non-parametrically using a quadratic program, which they equivalently write as a linear program to get the scaled weight matrix. The scaling factors in the diagonal matrix can then be learnt using a linear regression.  Finally, the authors conduct extensive experimentation to validate that their algorithms performs superior to the vanilla SGD-based algorithm. They also show that their algorithm, especially the QP algorithm is robust to noise.