On the Margin Theory of Feedforward Neural Networks

1 INTRODUCTION . In deep learning , over-parametrization refers to the widely-adopted technique of using more parameters than necessary ( Krizhevsky et al. , 2012 ; Livni et al. , 2014 ) . Both computationally and statistically , over-parametrization is crucial for learning neural nets . Controlled experiments demonstrate that over-parametrization eases optimization by smoothing the non-convex loss surface ( Livni et al. , 2014 ; Sagun et al. , 2017 ) . Statistically , increasing model size without any regularization still improves generalization even after the model interpolates the data perfectly ( Neyshabur et al. , 2017b ) . This is surprising given the conventional wisdom on the trade-off between model capacity and generalization . In the absence of an explicit regularizer , algorithmic regularization is likely the key contributor to good generalization . Recent works have shown that gradient descent finds the minimum norm solution fitting the data for problems including logistic regression , linearized neural networks , and matrix factorization ( Soudry et al. , 2018 ; Gunasekar et al. , 2018b ; Li et al. , 2018 ; Gunasekar et al. , 2018a ; Ji & Telgarsky , 2018 ) . Many of these proofs require a delicate analysis of the algorithm ’ s dynamics , and some are not fully rigorous due to assumptions on the iterates . To the best of our knowledge , it is an open question to prove analogous results for even two-layer relu networks . ( For example , the technique of Li et al . ( 2018 ) on two-layer neural nets with quadratic activations still falls within the realm of linear algebraic tools , which apparently do not suffice for other activations . ) We propose a different route towards understanding generalization : making the regularization explicit . The motivations are : 1 ) with an explicit regularizer , we can analyze generalization without fully understanding optimization ; 2 ) it is unknown whether gradient descent provides additional implicit regularization beyond what ` 2 regularization already offers ; 3 ) on the other hand , with a sufficiently weak ` 2 regularizer , we can prove stronger results that apply to multi-layer relu networks . Additionally , explicit regularization is perhaps more relevant because ` 2 regularization is typically used in practice . Concretely , we add a norm-based regularizer to the cross entropy loss of a multi-layer feedforward neural network with relu activations . We show that the global minimizer of the regularized objective achieves the maximum normalized margin among all the models with the same architecture , if the regularizer is sufficiently weak ( Theorem 2.1 ) . Informally , for models with norm 1 that perfectly classify the data , the margin is the smallest difference across all datapoints between the classifier score for the true label and the next best score . We are interested in normalized margin because its inverse bounds the generalization error ( see recent work ( Bartlett et al. , 2017 ; Neyshabur et al. , 2017a ; 2018 ; Golowich et al. , 2017 ) or Proposition 3.1 ) . Our work explains why optimizing the training loss can lead to parameters with a large margin and thus , better generalization error ( see Corollary 3.2 ) . We further note that the maximum possible margin is non-decreasing in the width of the architecture , and therefore the generalization bound of Corollary 3.2 can only improve as the size of the network grows ( see Theorem 3.3 ) . Thus , even if the dataset is already separable , it could still be useful to increase the width to achieve larger margin and better generalization . At a first glance , it might seem counterintuitive that decreasing the regularizer is the right approach . At a high level , we show that the regularizer only serves as a tiebreaker to steer the model towards choosing the largest normalized margin . Our proofs are simple , oblivious to the optimization procedure , and apply to any norm-based regularizer . We also show that an exact global minimum is unnecessary : if we approximate the minimum loss within a constant factor , we obtain the max-margin within a constant factor ( Theorem 2.2 ) . To better understand the neural network max-margin , in Section 4 we compare the max-margin two-layer network obtained by optimizing both layers jointly to kernel methods corresponding to fixing random weights for the hidden layer and solving a 2-norm max-margin on the top layer . We design a simple data distribution ( Figure 1 ) where neural net margin is large but the kernel margin is small . This translates to an Ω ( √ d ) factor gap between the generalization error bounds for the two approaches and demonstrates the power of neural nets compared to kernel methods . We experimentally confirm that a gap does indeed exist . In the setting of two-layer networks , we also study how over-parametrization helps optimization . Prior works ( Mei et al. , 2018 ; Chizat & Bach , 2018 ; Sirignano & Spiliopoulos , 2018 ; Rotskoff & Vanden-Eijnden , 2018 ) show that gradient descent on two-layer networks becomes Wasserstein gradient flow over parameter distributions in the limit of infinite neurons . For this setting , we prove that perturbed Wasserstein gradient flow finds a global optimizer in polynomial time . Finally , we empirically validate several claims made in this paper . First , we confirm that neural networks do generalize better than kernel methods . Second , we show that for two-layer networks , the test error decreases and margin increases as the hidden layer grows , as predicted by our theory . 1.1 ADDITIONAL RELATED WORK . Zhang et al . ( 2016 ) and Neyshabur et al . ( 2017b ) show that neural network generalization defies conventional explanations and requires new ones . Neyshabur et al . ( 2014 ) initiate the search for the “ inductive bias ” of neural networks towards solutions with good generalization . Recent papers ( Hardt et al. , 2015 ; Brutzkus et al. , 2017 ; Chaudhari et al. , 2016 ) study inductive bias through training time and sharpness of local minima . Neyshabur et al . ( 2015a ) propose a new steepest descent algorithm in a geometry invariant to weight rescaling and show that this improves generalization . Morcos et al . ( 2018 ) relate generalization in deep nets to the number of “ directions ” in the neurons . Other papers ( Gunasekar et al. , 2017 ; Soudry et al. , 2018 ; Nacson et al. , 2018 ; Gunasekar et al. , 2018b ; Li et al. , 2018 ; Gunasekar et al. , 2018a ) study implicit regularization towards a specific solution . Ma et al . ( 2017 ) show that implicit regularization can help gradient descent avoid overshooting optima . Rosset et al . ( 2004a ; b ) study logistic regression with a weak regularization and show convergence to the max margin solution . We adopt their techniques and extend their results . A line of work initiated by Neyshabur et al . ( 2015b ) has focused on deriving tighter norm-based Rademacher complexity bounds for deep neural networks ( Bartlett et al. , 2017 ; Neyshabur et al. , 2017a ; Golowich et al. , 2017 ) and new compression based generalization properties ( Arora et al. , 2018b ) . Dziugaite & Roy ( 2017 ) manage to compute non-vacuous generalization bounds from PAC-Bayes bounds . Neyshabur et al . ( 2018 ) investigate the Rademacher complexity of two-layer networks and propose a bound that is decreasing with the distance to initialization . Liang & Rakhlin ( 2018 ) and Belkin et al . ( 2018 ) study the generalization of kernel methods . On the optimization side , Soudry & Carmon ( 2016 ) explain why over-parametrization can remove bad local minima . Safran & Shamir ( 2016 ) show that over-parametrization can improve the quality of the random initialization . Haeffele & Vidal ( 2015 ) , Nguyen & Hein ( 2017 ) , and Venturi et al . ( 2018 ) show that for sufficiently overparametrized networks , all local minima are global , but do not show how to find these minima via gradient descent . Du & Lee ( 2018 ) show that for two-layer networks with quadratic activations , all second-order stationary points are global minimizers . Arora et al . ( 2018a ) interpret over-parametrization as a means of implicit acceleration during optimization . Mei et al . ( 2018 ) , Chizat & Bach ( 2018 ) , and Sirignano & Spiliopoulos ( 2018 ) take a distributional view of over-parametrized networks . Chizat & Bach ( 2018 ) show that Wasserstein gradient flow converges to global optimizers under structural assumptions . We extend this to a polynomial-time result . 1.2 NOTATION . Let R denote the set of real numbers . We will use ‖·‖ to indicate a general norm , with ‖·‖1 , ‖·‖2 , ‖·‖∞ denoting the ` 1 , ` 2 , ` ∞ norms on finite dimensional vectors , respectively , and ‖ · ‖F denoting the Frobenius norm on a matrix . In general , we use ¯ on top of a symbol to denote a unit vector : when applicable , ū , u/‖u‖ , where the norm ‖ · ‖ will be clear from context . Let Sd−1 , { ū ∈ Rd : ‖ū‖2 = 1 } be the unit sphere in d dimensions . Let Lp ( Sd−1 ) be the space of functions on Sd−1 for which the p-th power of the absolute value is Lebesgue integrable . For α ∈ Lp ( Sd−1 ) , we overload notation and write ‖α‖p , ( ∫ Sd−1 |α ( ū ) | pdū ) 1/p . Additionally , for α1 ∈ L1 ( Sd−1 ) and α2 ∈ L∞ ( Sd−1 ) or α1 , α2 ∈ L2 ( Sd−1 ) , we can define 〈α1 , α2〉 , ∫ Sd−1 α1 ( ū ) α2 ( ū ) dū < ∞ . Furthermore , we will use Vol ( Sd−1 ) , ∫ Sd−1 1dū . Throughout this paper , we reserve the symbol X = [ x1 , . . . , xn ] to denote the collection of datapoints ( as a matrix ) , and Y = [ y1 , . . . , yn ] to denote labels . We use d to denote the dimension of our data . We often use Θ to denote the parameters of a prediction function f , and f ( Θ ; x ) to denote the prediction of f on datapoint x . We will use the notation . , & to mean less than or greater than up to a universal constant , respectively . Unless stated otherwise , O ( · ) , Ω ( · ) denote some universal constant in upper and lower bounds , respectively . The notation poly denotes a universal constant-degree polynomial in the arguments . 2 WEAK REGULARIZER GUARANTEES MAX MARGIN SOLUTIONS . In this section , we will show that when we add a weak regularizer to cross-entropy loss with a positive-homogeneous prediction function , the normalized margin of the optimum converges to some max-margin solution . As a concrete example , feedforward relu networks are positive-homogeneous . Let l be the number of labels , so the i-th example has label yi ∈ [ l ] . We work with a family F of prediction functions f ( Θ ; · ) : Rd → Rl that are a-positive-homogeneous in their parameters for some a > 0 : f ( cΘ ; x ) = caf ( Θ ; x ) , ∀c > 0 . We additionally require that f is continuous in Θ . For some general norm ‖ · ‖ , we study the λ-regularized cross-entropy loss Lλ , defined as Lλ ( Θ ) , n∑ i=1 − log exp ( fyi ( Θ ; xi ) ) ∑l j=1 exp ( fj ( Θ ; xi ) ) + λ‖Θ‖r ( 2.1 ) for fixed r > 0 . Let Θλ ∈ arg minLλ ( Θ ) .1 We define the normalized margin of Θλ as : γλ , min i ( fyi ( Θ̄λ ; xi ) −max j 6=yi fj ( Θ̄λ ; xi ) ) ( 2.2 ) Define the ‖ · ‖-max normalized margin as γ ? , max ‖Θ‖≤1 [ min i ( fyi ( Θ ; xi ) −max j 6=yi fj ( Θ ; xi ) ) ] and let Θ ? be a parameter achieving this maximum . We show that with sufficiently small regularization level λ , the normalized margin γλ approaches the maximum margin γ ? . Our theorem and proof are inspired by the result of Rosset et al . ( 2004a ; b ) , who analyze the special case when f is a linear predictor . In contrast , our result can be applied to non-linear f as long as f is homogeneous . Theorem 2.1 . Assume the training data is separable by a network f ( Θ ? ; · ) ∈ F with an optimal normalized margin γ ? > 0 . Then , the normalized margin of the global optimum of the weaklyregularized objective ( equation 2.1 ) converges to γ ? as the strength of the regularizer goes to zero . Mathematically , let γλ be defined in equation 2.2 . Then γλ → γ ? as λ→ 0 1We formally show that Lλ has a minimizer in Claim A.1 of Section A . An intuitive explanation for our result is as follows : because of the homogeneity , the loss L ( Θλ ) roughly satisfies the following ( for small λ , and ignoring problem parameters such as n ) : Lλ ( Θλ ) ≈ exp ( −‖Θλ‖aγλ ) + λ‖Θλ‖r Thus , the loss selects parameters with larger margin , while the regularization favors parameters with a smaller norm . The full proof of the theorem is deferred to Section A.1 . Theorem 2.1 applies to feedforward relu networks and states that global minimizers of the weaklyregularized loss will obtain a maximum margin among all networks of the given architecture . By considering global minimizers , Theorem 2.1 provides a framework for directly analyzing generalization properties of the solution without considering details of the optimization algorithm . In Section 3 we leverage this framework and existing generalization bounds ( Golowich et al. , 2017 ) to provide a clean argument that over-parameterization can improve generalization . We can also provide an analogue of Theorem 2.1 for the binary classification setting . For this setting , our prediction is now a single real output and we train using logistic loss . We provide formal definitions and results in Section A.2 . Our study of the generalization properties of the max-margin ( see Section 3 and Section 4 ) is based in this setting .

Summary: This paper studies the implicit bias of minimizers of a regularized cross entropy loss of a two-layer network with ReLU activations. By combining several results, the authors obtain a generalization upper bound which does not increase with the network size. Furthermore, they show that the maximum normalized margin is, up to a scaling factor, the l1 svm margin over the lifted feature space of an infinite-size network. Finally, in a setting of infinite-sized networks, it is proved that perturbed Wasserstein gradient flow finds a global minimum in polynomial time.