When Will Gradient Methods Converge to Max-margin Classifier under ReLU Models?

1 INTRODUCTION . It has been observed in various machine learning problems recently that the gradient descent ( GD ) algorithm and the stochastic gradient descent ( SGD ) algorithm converge to solutions with certain properties even without explicit regularization in the objective function . Correspondingly , theoretical analysis has been developed to explain such implicit regularization property . For example , it has been shown in Gunasekar et al . ( 2018 ; 2017 ) that GD converges to the solution with the minimum norm under certain initialization for regression problems , even without an explicit norm constraint . Another type of implicit regularization , where GD converges to the max-margin classifier , has been recently studied in Gunasekar et al . ( 2018 ) ; Ji & Telgarsky ( 2018 ) ; Nacson et al . ( 2018a ) ; Soudry et al . ( 2017 ; 2018 ) for classification problems as we describe below . Given a set of training samples zi = ( xi , yi ) for i = 1 , . . . , n , where xi denotes a feature vector and yi ∈ { −1 , +1 } denotes the corresponding label , the goal is to find a desirable linear model ( i.e. , a classifier ) by solving the following empirical risk minimization problem min w∈Rd L ( w ) : = 1 n n∑ i=1 ` ( yiw ᵀxi ) . ( 1 ) It has been shown in Nacson et al . ( 2018a ) ; Soudry et al . ( 2017 ; 2018 ) that if the loss function ` ( · ) is monotonically strictly decreasing and satisfies proper tail conditions ( e.g. , the exponential loss ) , and the data are linearly separable , then GD converges to the solution w with infinite norm and the maximum margin direction of the data , although there is no explicit regularization towards the maxmargin direction in the objective function . Such a phenomenon is referred to as the implicit bias of GD , and can help to explain some experimental results . For example , even when the training error achieves zero ( i.e. , the resulting model enters into the linearly separable region that correctly classifies the data ) , the testing error continues to decrease , because the direction of the model parameter continues to have an improved margin . Such a study has been further generalized to hold for various other types of gradient-based algorithms Gunasekar et al . ( 2018 ) . Moreover , Ji & Telgarsky ( 2018 ) analyzed the convergence of GD with no assumption on the data separability , and characterized the implicit regularization to be in a subspace-based form . The focus of this paper is on the following two fundamental issues , which have not been well addressed by existing studies . • Existing studies so far focused only on the linear classifier model . An important question one naturally asks is what happens for the more general nonlinear leaky ReLU and ReLU models . Will GD still converge , and if so will it converge to the max-margin direction ? Our study here provides new insights for the ReLU model that have not been observed for the linear model in the previous studies . • Existing studies mainly analyzed the convergence of GD with the only exceptions Ji & Telgarsky ( 2018 ) ; Nacson et al . ( 2018b ) on SGD . However , Ji & Telgarsky ( 2018 ) did not establish the convergence to the max-margin direction for SGD , and Nacson et al . ( 2018b ) established the convergence to the max-margin solution only epochwisely for cyclic SGD ( not iterationwise for SGD under random sampling with replacement ) . Moreover , both studies considered only the linear model . Here , our interest is to explore the iterationwise convergence of SGD under random sampling with replacement to the max-margin direction , and our result can shed insights for online SGD . Furthermore , our study provides new understanding for the nonlinear ReLU and leaky ReLU models . 1.1 MAIN CONTRIBUTIONS . We summarize our main contributions , where our focus is on the exponential loss function under ReLU model . We first characterize the landscape of the empirical risk function under the ReLU model , which is nonconvex and nonsmooth . We show that such a risk function has asymptotic global minima and asymptotic spurious local minima . Such a landscape is in sharp contrast to that under the linear model previously studied in Soudry et al . ( 2017 ) , where there exist only equivalent global minima . Based on the landscape property , we show that the implicit bias property in the course of the convergence of GD can fall into four cases : converges to the asymptotic global minimum along the max-margin direction , converges to an asymptotic local minimum along a local max-margin direction , stops at a finite spurious local minimum , or oscillates between the linearly separable and misclassified regions without convergence . Such a diverse behavior is also in sharp difference from that under the linear model Soudry et al . ( 2017 ) , where GD always converges to the max-margin direction . We then take a further step to study the implicit bias of SGD . We show that the expected averaged weight vector normalized by its expected l2 norm converges to the global max-margin direction or local max-margin direction , as long as SGD stays either in the linearly separable region or in a region of the local minima defined by a subset of data samples with positive label . The proof here requires considerable new technical developments , which are very different from the traditional analysis of SGD , e.g. , Bottou et al . ( 2016 ) ; Duchi & Singer ( 2009 ) ; Nemirovskii et al . ( 1983 ) ; Shalev-Shwartz et al . ( 2009 ) ; Xiao ( 2010 ) ; Bach & Moulines ( 2013 ) ; Bach ( 2014 ) . This is because our focus here is on the exponential loss function without attainable global/local minima , whereas traditional analysis typically assumed that the minimum of the loss function is attainable . Furthermore , our goal is to analyze the implicit bias property of SGD , which is also beyond traditional analysis of SGD . We further extend our analysis to the leaky ReLU model and multi-neuron networks . 1.2 RELATED WORK . Implicit bias of gradient descent : Gunasekar et al . ( 2018 ) studied the implicit bias of GD and SGD for minimizing the squared loss function under bounded global minimum , and showed that some of these algorithms converge to a global minimum that is closest to the initial point . Another collection of papers Gunasekar et al . ( 2018 ) ; Ji & Telgarsky ( 2018 ) ; Nacson et al . ( 2018a ) ; Soudry et al . ( 2017 ) ; Telgarsky ( 2013 ) ; Soudry et al . ( 2018 ) characterized the implicit bias of algorithms for the loss functions without attainable global minimum . Telgarsky ( 2013 ) showed that AdaBoost converges to an approximate max-margin classifier . Soudry et al . ( 2017 ; 2018 ) studied the convergence of GD in logistic regression with linearly separable data and showed that GD converges in direction to the solution of support vector machine at a rate of 1/ ln ( t ) . Nacson et al . ( 2018a ) improved this rate to ln ( t ) / √ t under the exponential loss via normalized gradient descent . Gunasekar et al . ( 2018 ) further showed that steepest descent can lead to margin maximization under generic norms . Ji & Telgarsky ( 2018 ) analyzed the convergence of GD on an arbitrary dataset , and provided the convergence rates along the strongly convex subspace and the separable subspace . Our work studies the convergence of GD and SGD under the nonlinear ReLU model with the exponential loss , as opposed to the linear model studied by all the above previous work on the same type of loss functions . Implicit bias of SGD : Ji & Telgarsky ( 2018 ) analyzed the average SGD ( under random sampling ) with fixed learning rate and proved the convergence of the population risk , but did not establish the parameter convergence of SGD in the max-margin direction . Nacson et al . ( 2018b ) established the convergence of cyclic SGD epochwisely in direction to the max-margin classifier at a rateO ( 1/ ln t ) . Our work differs from these two studies first in that we study the ReLU model , whereas both of these studies analyzed the linear model . Furthermore , we showed that under SGD with random sampling , the expectation of the averaged weight vector converges in direction to the max-margin classifier at a rate O ( 1/ √ ln t ) . Generalization of SGD : There have been extensive studies of the convergence and generalization performance of SGD under various models , of which we can not provide a comprehensive list due to the space limitations . In general , these type of studies either characterize the convergence rate of SGD or provide the generalization error bounds at the convergence of SGD , e.g. , Brutzkus et al . ( 2017 ) ; Wang et al . ( 2018 ) ; Li & Liang ( 2018 ) , but did not characterize the implicit regularization property of SGD , such as the convergence to the max-margin direction as provided in our paper . 2 RELU CLASSIFICATION MODEL . We consider the binary classification problem , in which we are given a set of training samples { z1 , . . . , zn } . Each training sample zi = ( xi , yi ) contains an input data xi and a corresponding binary label yi ∈ { −1 , +1 } . We denote I+ : = { i : yi = +1 } as the set of indices of samples with label +1 and denote I− : = { i : yi = −1 } in a similar way . Their cardinalities are denoted as n+ and n− , respectively , and are assumed to be non-zero . We consider all datasets that are linearly separable , i.e. , there exists a linear classifier w such that yiwᵀxi > 0 for all i = 1 , . . . , n. We are interested in training a ReLU model for the classification task . In specific , for a given input data x , the model outputs σ ( wᵀxi ) , where σ ( v ) = max { 0 , v } is the ReLU activation function and w denotes the weight parameters . The predicted label is set to be sgn ( wᵀx ) . Our goal is to learn a classifier by solving the following empirical risk minimization problem , where we adopt the exponential loss . min w∈Rd L ( w ) : = 1 n n∑ i=1 ` ( w , zi ) , where ` ( w , zi ) = exp ( −yiσ ( wᵀxi ) ) . ( P ) The ReLU activation causes the loss function in problem ( P ) to be nonconvex and nonsmooth . Therefore , it is important to first understand the landscape property of the loss function , which is critical for characterizing the implicit bias property of the GD and SGD algorithms . 3 IMPLICIT BIAS OF GD IN LEARNING RELU MODEL . 3.1 LANDSCAPE OF RELU MODEL . In order to understand the convergence of GD under the ReLU model , we first study the landscape of the loss function in problem ( P ) , which turns out to be very different from that under the linear activation model . As been shown in Soudry et al . ( 2017 ) ; Ji & Telgarsky ( 2018 ) , the loss function in problem ( P ) under linear activation is convex , and achieves asymptotic global minimum , i.e. , ∇L ( αw∗ ) α→ 0 and L ( αw∗ ) α→ 0 as the scaling constant α → +∞ , only if w∗ is in the linearly separable region . In contrast , under the ReLU model , the asymptotic critical points can be either global minimum or ( spurious ) local minimum depending on the training datasets , and hence the convergence property of GD can be very different in nature from that under the linear model . The following theorem characterizes the landscape properties of problem ( P ) . Throughout , we denote the infimum of the objective function in problem ( P ) as L∗ = n − n . Furthermore , we call a direction w∗ asymptotically critical if it satisfies∇L ( αw∗ ) →0 as α→ +∞ . Theorem 3.1 ( Asymptotic landscape property ) . For problem ( P ) under the ReLU model , any corresponding asymptotic critical direction w∗ fall into one of the following cases :

Summary: This paper studies ReLU model, or equivalently, one-layer-one-neuron model, for the classification problem. This paper shows if the data is linearly separable, gradient descent may converge to either a global minimum or a sub-optimal local minimum, or diverges. This paper further studies the implicit bias induced by GD and SGD and shows if they converge, they can have a maximum margin solution.