On the loss landscape of a class of deep neural networks with no bad local valleys

1 INTRODUCTION . It has been empirically observed in deep learning ( Dauphin et al. , 2014 ; Goodfellow et al. , 2015 ) that the training problem of over-parameterized1 deep CNNs ( LeCun et al. , 1990 ; Krizhevsky et al. , 2012 ) does not seem to have a problem with bad local minima . In many cases , local search algorithms like stochastic gradient descent ( SGD ) frequently converge to a solution with zero training error even though the training objective is known to be non-convex and potentially has many distinct local minima ( Auer et al. , 1996 ; Safran & Shamir , 2018 ) . This indicates that the problem of training practical over-parameterized neural networks is still far from the worst-case scenario where the problem is known to be NP-hard ( Blum & Rivest. , 1989 ; Sima , 2002 ; Livni et al. , 2014 ; ShalevShwartz et al. , 2017 ) . A possible hypothesis is that the loss landscape of these networks is “ wellbehaved ” so that it becomes amenable to local search algorithms like SGD and its variants . As not all neural networks have a well-behaved loss landscape , it is interesting to identify sufficient conditions on their architecture so that this is guaranteed . In this paper our motivation is to come up with such a class of networks in a practically relevant setting , that is we study multi-class problems with the usual empirical cross-entropy loss and deep ( convolutional ) networks and almost no assumptions on the training data , in particular no distributional assumptions . Thus our results directly apply to the networks which we use in the experiments . Our contributions . We identify a family of deep networks with skip connections to the output layer whose loss landscape has no bad local valleys ( see Figure 1 for an illustration ) . Our setting is 1These are the networks which have more parameters than necessary to fit the training data for the empirical loss and there are no distributional assumptions on the training data . Moreover , we study directly the standard cross-entropy loss for multi-class problems . There are little assumptions on the network structure which can be arbitrarily deep and can have convolutional layers ( weight sharing ) and skip-connections between hidden layers . From a practical perspective , one can generate an architecture which fulfills our conditions by taking an existing CNN architecture and then adding skip-connections from a random subset of N neurons ( N is the number of training samples ) , possibly from multiple hidden layers , to the output layer ( see Figure 2 for an illustration ) . For these networks we show that there always exists a continuous path from any point in parameter space on which the loss is non-increasing and gets arbitrarily close to zero . We note that this implies the loss landscape has no strict local minima , but theoretically non-strict local minima can still exist . Beside that , we show that the loss has also no local maxima . Beside the theoretical analysis , we show in experiments that despite achieving zero training error , the aforementioned class of neural networks generalize well in practice when trained with SGD whereas an alternative training procedure guaranteed to achieve zero training error has significantly worse generalization performance and is overfitting . Thus we think that the presented class of neural networks offer an interesting test bed for future work to study the implicit bias/regularization of SGD . 2 DESCRIPTION OF NETWORK ARCHITECTURE . We consider a family of deep neural networks which have d input units , H hidden units , m output units and satisfy the following conditions : 1 . Every hidden unit of the first layer can be connected to an arbitrary subset of input units . 2 . Every hidden unit at higher layers can take as input an arbitrary subset of hidden units from ( multiple ) lower hidden layers . 3 . Any subgroup of hidden units lying on the same layer can have non-shared or shared weights , in the later case their number of incoming units have to be equal . 4 . There exist N hidden units which are connected to the output nodes with independent weights ( N denotes the number of training samples ) . 5 . The output of every hidden unit j in the network , denoted as fj : Rd → R , is given as fj ( x ) = σj ( bj + ∑ k : k→j fk ( x ) uk→j ) where x ∈ Rd is an input vector of the network , σj : R→ R is the activation function of unit j , bj ∈ R is the bias of unit j , and uk→j ∈ R the weight from unit k to unit j . This definition covers a class of deep fully connected and convolutional neural networks with an additional condition on the number of connections to the output layer . In particular , while conventional architectures have just connections from the last hidden layer to the output , we require in our setting that there must exist at least N neurons , “ regardless ” of their hidden layer , that are connected to the output layer . Essentially , this means that if the last hidden layer of a traditional network has just L < N neurons then one can add connections from N − L neurons in the hidden layers below it to the output layer so that the network fulfills our conditions . Similar skip-connections have been used in DenseNet ( Huang et al. , 2017 ) which are different from identity skip-connections as used in ResNets ( He et al. , 2016 ) . In Figure 2 we illustrate a network with and without skip connections to the output layer which is analyzed in this paper . We note that several architectures like DenseNets Huang et al . ( 2017 ) already have skip-connections between hidden layers in their original architecture , whereas our special skip-connections go from hidden layers directly to the output layer . As our framework allow both kinds to exist in the same network ( see Figure 2 for an example ) , we would like to separate them from each other by making the convention that in the following skip-connections , if not stated otherwise , always refer to ones which connect hidden neurons to output neurons . We denote by d the dimension of the input and index all neurons in the network from the input layer to the output layer as 1 , 2 , . . . , d , d+ 1 , . . . , d+H , d+H + 1 , . . . , d+H +m which correspond to d input units , H hidden units and m output units respectively . As we only allow directed arcs from lower layers to upper layers , it follows that k < j for every k → j . Let N be the number of training samples . Suppose that there are M hidden neurons which are directly connected to the output with independent weights where it holds N ≤ M ≤ H. Let { p1 , . . . , pM } with pj ∈ { d+ 1 , . . . , d+H } be the set of hidden units which are directly connected to the output units . Let in ( j ) be the set of incoming nodes to unit j and uj = [ uk→j ] k∈in ( j ) the weight vector of the j-th unit . Let U = ( ud+1 , . . . , ud+H , bd+1 , . . . , bd+H ) denote the set of all weights and biases of all hidden units in the network . Let V ∈ RM×m be the weight matrix which connects the M hidden neurons to the m output units of the network . An important quantity in the following is the matrix Ψ ∈ RN×M defined as Ψ = fp1 ( x1 ) . . . fpM ( x1 ) ... ... fp1 ( xN ) . . . fpM ( xN ) ( 1 ) As Ψ depends on U , we write ΨU or Ψ ( U ) as a function of U . Let G ∈ RN×m be the output of the network for all training samples . In particular , Gij is the value of the j-th output neuron for training sample xi . It follows from our definition that Gij = 〈Ψi : , V : j〉 = M∑ k=1 fpk ( xi ) Vkj , ∀i ∈ [ N ] , j ∈ [ m ] Let ( xi , yi ) Ni=1 be the training set where yi denotes the target class for sample xi . In the following we analyze the commonly used cross-entropy loss given as Φ ( U , V ) = 1 N N∑ i=1 − log ( eGiyi∑m k=1 e Gik ) ( 2 ) We refer to Section C in the appendix for extension of our results to general convex losses . The cross-entropy loss is bounded from below by zero but this value is not attained . In fact the global minimum of the cross-entropy loss need not exist e.g . if a classifier achieves zero training error then by upscaling the function to infinity one can drive the loss arbitrarily close to zero . Due to this property , we do not study the global minima of the cross-entropy loss but the question if and how one can achieve zero training error . Moreover , we note that sufficiently small cross-entropy loss implies zero training error as shown in the following lemma . Lemma 2.1 If Φ ( U , V ) < log ( 2 ) N , then the training error is zero . Proof : We note that if Φ ( U , V ) < log ( 2 ) N , then it holds due to the positivity of the loss , maxi=1 , ... , N − log ( eGiyi∑m k=1 e Gik ) ≤ N∑ i=1 − log ( eGiyi∑m k=1 e Gik ) < log ( 2 ) . This implies that for all i = 1 , . . . , N , log ( 1 + ∑ k 6=yi eGik−Giyi ) < log ( 2 ) =⇒ ∑ k 6=yi eGik−Giyi < 1 . In particular : maxk 6=yi e Gik−Giyi < 1 and thus maxk 6=yi Gik − Giyi < 0 for all i = 1 , . . . , N which implies the result . 3 MAIN RESULT . The following conditions are required for the main result to hold . Assumption 3.1 1 . All activation functions { σd+1 , . . . , σd+H } are real analytic and strictly increasing 2 . Among M neurons { p1 , . . . , pM } which are connected to the output units , there exist N ≤M neurons , say w.l.o.g . { p1 , . . . , pN } , such that one of the following conditions hold : • For every 1 ≤ j ≤ N : σpj is bounded and limt→−∞ σpj ( t ) = 0 • For every 1 ≤ j ≤ N : σpj is the softplus activation ( 3 ) , and there exists a backward path from pj to the first hidden layer s.t . on this path there is no neuron which has skip-connections to the output or shared weights with other skip-connection neurons . 3 . The input patches of different training samples are distinct . In particular , let n1 be the number of units in the first hidden layer and denote by Si for i ∈ [ d+ 1 , d+ n1 ] their input support , then for all r 6= s ∈ [ N ] , and i ∈ [ d+ 1 , d+ n1 ] , it holds xr|Si 6= xs|Si . The first condition of Assumption 3.1 is satisfied for softplus , sigmoid , tanh , etc , whereas the second condition is fulfilled for sigmoid and softplus . For softplus activation function ( smooth approximation of ReLU ) , σγ ( t ) = 1 γ log ( 1 + eγt ) , for some γ > 0 , ( 3 ) we require an additional assumption on the network architecture . The third condition is always satisfied for fully connected networks if the training samples are distinct . For CNNs , this condition means that the corresponding input patches across different training samples are distinct . This could be potentially violated if the first convolutional layer has very small receptive fields . However , if this condition is violated for the given training set then after an arbitrarily small random perturbation of all training inputs it will be satisfied with probability 1 . Note that the M neurons which are directly connected to the output units can lie on different hidden layers in the network . Also there is no condition on the width of every individual hidden layer as long as the total number of hidden neurons in the network is larger than N so that our condition M ≥ N is feasible . Overall , we would like to stress that Assumption 3.1 covers a quite large class of interesting network architectures but nevertheless allows us to show quite strong results on their empirical loss landscape . The following key lemma shows that for almost all U , the matrix Ψ ( U ) has full rank . Lemma 3.2 Under Assumption 3.1 , the set of U such that Ψ ( U ) has not full rank N has Lebesgue measure zero . Proof : ( Proof sketch ) Due to space limitation , we can only present below a proof sketch . We refer the reader to the appendix for the detailed proof . The proof consists of two main steps . First , we show that there exists U s.t . the submatrix Ψ1 : N,1 : N has full rank . By Assumption 3.1 , all activation functions are real analytic , thus the determinant of Ψ1 : N,1 : N is a real analytic function of the network parameters which Ψ depends on . By the first result , this determinant function is not identically zero , thus Lemma A.1 shows that the set of U for which Ψ has not full rank has Lebesgue measure zero . Now we sketch the proof for the first step , that is to find a U s.t . Ψ has full rank . By Assumption 3.1.3 , one can always choose the weight vectors of the first hidden layer so that every neuron at this layer has distinct values for different training samples . For higher neurons , we set their initial weight vectors to be unit vectors with exactly one 1 and 0 elsewhere . Note that the above construction of weights can be easily done so that all the neurons from the same layer and with the same number of incoming units can have shared/unshared weights according to our description of network architecture in Section 2 . Let c ( j ) be the neuron below j s.t . uc ( j ) →j = 1 . To find U , we are going to scale up each weight vector uj by a positive scalar αj . The idea is to show that the determinant of Ψ1 : N,1 : N is non-zero for some positive value of { αj } . The biases can be chosen in such a way that the following holds for some β ∈ R , Ψij = fpj ( xi ) = σpj ( β + αpj ( fc ( pj ) ( xi ) − fc ( pj ) ( xj ) ) ) ∀j ∈ [ N ] ( 4 ) where fc ( pj ) ( x ) = σc ( pj ) ( αc ( pj ) σc ( c ( pj ) ) ( . . . fqj ( x ) . . . ) ) with qj being the index of some neuron in the first hidden layer . Note that by our construction the value of unit qj is distinct at different training samples , and thus it follows from the strict monotonic property of activation functions from Assumption 3.1 and the positivity of { αc ( pj ) , . . . } that fc ( pj ) ( xi ) 6= fc ( pj ) ( xj ) for every i 6= j . Next , we show that the set of training samples can be re-ordered in such a way that it holds fc ( pj ) ( xi ) < fc ( pj ) ( xj ) for every i > j . Note that this re-ordering does not affect the rank of Ψ . Now , the intuition is that if one let αpj go to infinity then Ψij converges to zero for i > j because it holds for all activations from Assumption 3.1 that limt→−∞ σpj ( t ) = 0 . Thus the determinant of Ψ1:1,1 : N converges to ∏N i=1 σpj ( β ) which can be chosen to be non-zero by a predefined value of β , in which case Ψ will have full rank . The detailed proof basically will show how to choose the specific values of { αj } so that all the above criteria are met . In particular , it is important to make sure that the weight vectors of two neurons j , j′ from the same layer will be scaled by the same factor αj = αj′ as we want to maintain any potential weight-sharing conditions . The choice of activation functions from the second condition of Assumption 3.1 basically determines how the values of α should be chosen . While we conjecture that the result of Lemma 3.2 holds for softplus activation function without the additional condition as mentioned in Assumption 3.1 , the proof of this is considerably harder for such a general class of neural networks since one has to control the output of neurons with skip connection from different layers which depend on each other . However , please note that the condition is also not too restrictive as it just might require more connections from lower layers to upper layers but it does not require that the network is wide . Before presenting our main result , we first need a formal definition of bad local valleys . Definition 3.3 The α-sublevel set of Φ is defined as Lα = { ( U , V ) | Φ ( U , V ) < α } . A local valley is defined as a connected component of some sublevel set Lα . A bad local valley is a local valley on which the loss function Φ can not be made “ arbitrarily small ” . Intuitively , a typical example of a bad local valley is a small neighborhood around a sub-optimal strict local minimum . We are now ready to state our main result . Theorem 3.4 The following holds under Assumption 3.1 : 1 . There exist uncountably many solutions with zero training error . 2 . The loss landscape of Φ does not have any bad local valley . 3 . There exists no suboptimal strict local minimum . 4 . There exists no local maximum . Proof : 1 . By Lemma 3.2 the set of U such that Ψ ( U ) has not full rank N has Lebesgue measure zero . Given U such that Ψ has full rank , the linear system Ψ ( U ) V = Y has for every possible target output matrix Y ∈ RN×m at least one solution V . As this is possible for almost all U , there exist uncountably many solutions achieving zero training error . 2 . Let C be a non-empty , connected component of some α-sublevel set Lα for α > 0 . Suppose by contradiction that the loss on C can not be made arbitrarily small , that is there exists an > 0 such that Φ ( U , V ) ≥ for all ( U , V ) ∈ C , where < α . By definition , Lα can be written as the pre-image of an open set under a continuous function , that is Lα = Φ −1 ( { a | a < α } ) , and thus Lα must be an open set ( see Proposition A.2 ) . Since C is a non-empty connected component of Lα , C must be an open set as well , and thus C has non-zero Lebesgue measure . By Lemma 3.2 the set of U where Ψ ( U ) has not full rank has measure zero and thus C must contain a point ( U , V ) such that Ψ ( U ) has full rank . Let Y be the usual zero-one one-hot encoding of the target network output . As Ψ ( U ) has full rank , there always exist V ∗ such that Ψ ( U ) V ∗ = Y t∗ , where t∗ = log ( m−1 e 2−1 ) Note that the loss of ( U , V ∗ ) is Φ ( U , V ∗ ) = − log ( et∗ et∗ + ( m− 1 ) ) = log ( 1 + ( m− 1 ) e−t ∗ ) = 2 . As the cross-entropy loss Φ ( U , V ) is convex in V and Φ ( U , V ) < α we have for the line segment V ( λ ) = λV + ( 1− λ ) V ∗ for λ ∈ [ 0 , 1 ] , Φ ( U , V ( λ ) ) ≤ λΦ ( U , V ) + ( 1− λ ) Φ ( U , V ∗ ) < λα+ ( 1− λ ) 2 < α . Thus the whole line segment is contained in Lα and as C is a connected component it has to be contained in C. However , this contradicts the assumption that for all ( U , V ) ∈ C it holds Φ ( U , V ) ≥ . Thus on every connected component C of Lα the training loss can be made arbitrarily close to zero and thus the loss landscape has no bad valleys . 3 . Let ( U0 , V0 ) be a strict suboptimal local minimum , then there exists r > 0 such that Φ ( U , V ) > Φ ( U0 , V0 ) > 0 for all ( U , V ) ∈ B ( ( U0 , V0 ) , r ) \ { ( U0 , V0 ) } where B ( · , r ) denotes a closed ball of radius r. Let α = min ( U , V ) ∈∂B ( ( U0 , V0 ) , r ) Φ ( U , V ) which exists as Φ is continuous and the boundary ∂B ( ( U0 , V0 ) , r ) of B ( ( U0 , V0 ) , r ) is compact . Note that α > Φ ( U0 , V0 ) as ( U0 , V0 ) is a strict local minimum . Consider the sub-level set D = Lα+Φ ( U0 , V0 ) 2 . As Φ ( U0 , V0 ) < α+Φ ( U0 , V0 ) 2 it holds ( U0 , V0 ) ∈ D. Let E be the connected component of D which contains ( U0 , V0 ) , that is , ( U0 , V0 ) ∈ E ⊆ D. It holds E ⊂ B ( ( U0 , V0 ) , r ) as Φ ( U , V ) < α+Φ ( U0 , V0 ) 2 < α for all ( U , V ) ∈ E. Moreover , Φ ( U , V ) ≥ Φ ( U0 , V0 ) > 0 for all ( U , V ) ∈ E and thus Φ can not be made arbitrarily small on a connected component of a sublevel set of Φ and thus E would be a bad local valley which contradicts 3.3.2 . 4 . Suppose by contradiction that ( U , V ) is a local maximum . Then the Hessian of Φ is negative semi-definite . However , as principal submatrices of negative semi-definite matrices are again negative semi-definite , then also the Hessian of Φ w.r.t V must be negative semidefinite . However , Φ is always convex in V and thus its Hessian restricted to V is positive semi-definite . The only matrix which is both p.s.d . and n.s.d . is the zero matrix . It follows that ∇2V Φ ( U , V ) = 0 . One can easily show that ∇2V : jΦ = N∑ i=1 eGij∑m k=1 e Gik ( 1− e Gij∑m k=1 e Gik ) Ψi : Ψ T i : From Assumption 3.1 it holds that there exists j ∈ [ N ] s.t . σpj is strictly positive , and thus some entries of Ψi : must be strictly positive . Moreover , one has e Gij∑m k=1 e Gik ∈ ( 0 , 1 ) . It follows that some entries of ∇2V : jΦ must be strictly positive . Thus ∇ 2 V : j Φ can not be identically zero , leading to a contradiction . Therefore Φ has no local maximum . Theorem 3.4 shows that there are infinitely many solutions which achieve zero training error , and the loss landscape is nice in the sense that from any point in the parameter space there exists a continuous path that drives the loss arbitrarily close to zero ( and thus a solution with zero training error ) on which the loss is non-increasing . While the networks are over-parameterized , we show in the next Section 4 that the modification of standard networks so that they fulfill our conditions leads nevertheless to good generalization performance , often even better than the original network . We would like to note that the proof of Theorem 3.4 also suggests a different algorithm to achieve zero training error : one initializes all weights , except the weights to the output layer , randomly ( e.g . Gaussian weights ) , denoted as U , and then just solves the linear system Ψ ( U ) V = Y to obtain the weights V to the output layer . Basically , this algorithm uses the network as a random feature generator and fits the last layer directly to achieve zero training error . The algorithm is successful with probability 1 due to Lemma 3.2 . Note that from a solution with zero training error one can drive the cross-entropy loss to zero by upscaling to infinity but this does not change the classifier . We will see , that this simple algorithm shows bad generalization performance and overfitting , whereas training the full network with SGD leads to good generalization performance . This might seem counter-intuitive as our networks have more parameters than the original networks but is inline with recent observations in Zhang et al . ( 2017 ) that state-of-the art networks , also heavily over-parameterized , can fit even random labels but still generalize well on the original problem . Due to this qualitative difference of SGD and the simple algorithm which both are able to find solutions with zero training error , we think that our class of networks is an ideal test bed to study the implicit regularization/bias of SGD , see e.g . Soudry et al . ( 2018 ) .

Summary: This paper presents a class of neural networks that does not have bad local valleys. The “no bad local valleys” implies that for any point on the loss surface there exists a continuous path starting from it, on which the loss doesn’t increase and gets arbitrarily smaller and close to zero. The key idea is to add direct skip connections from hidden nodes (from any hidden layer) to the output.