Piecewise linear activations substantially shape the loss surfaces of neural networks

1 INTRODUCTION . Neural networks have been successfully deployed in many real-world applications ( LeCun et al. , 2015 ; Witten et al. , 2016 ; Silver et al. , 2016 ; He et al. , 2016 ; Litjens et al. , 2017 ) . In spite of this , the theoretical foundations of neural networks are somewhat premature . To the many deficiencies in our knowledge of deep learning theory , the investigation into the loss surfaces of neural networks is of fundamental importance . Understanding the loss surface would be helpful in several relevant research areas , such as the ability to estimate data distributions , the optimization of neural networks , and the generalization to unseen data . This paper studies the role of the nonlinearities in activation functions in shaping the loss surfaces of neural networks . Our results demonstrate that the impact of nonlinearities is profound . First , we prove that the loss surfaces of nonlinear neural networks are substantially different to those of linear neural networks , in which local minima are created equal , and also , they are all global minima ( Kawaguchi , 2016 ; Baldi & Hornik , 1989 ; Lu & Kawaguchi , 2017 ; Freeman & Bruna , 2017 ; Zhou & Liang , 2018 ; Laurent & von Brecht , 2018 ; Yun et al. , 2018 ) . By contrast , Neural networks with arbitrary depth and arbitrary piecewise linear activations ( excluding linear functions ) have infinitely many spurious local minima under arbitrary continuously differentiable loss functions . ∗Both authors contributed equally . †Bohan Wang is also affiliated with University of Science and Technology of China . This work was completed when he was a summer intern at UBTECH Sydney AI Centre , School of Computer Science , Faculty of Engineering , the University of Sydney . This result only relies on four mild assumptions that cover most practical circumstances : ( 1 ) the training sample set is linearly inseparable ; ( 2 ) all training sample points are distinct ; ( 3 ) the output layer is narrower than the other hidden layers ; and ( 4 ) there exists some turning point in the piecewise linear activations that the sum of the slops on the two sides does not equal to 0 . Our result significantly extends the existing study on the existence of spurious local minimum . For example , Zhou & Liang ( 2018 ) prove that one-hidden-layer neural networks with two nodes in the hidden layer and two-piece linear ( ReLU-like ) activations have spurious local minima ; Swirszcz et al . ( 2016 ) prove that ReLU networks have spurious local minima under the squared loss when most of the neurons are not activated ; Safran & Shamir ( 2018 ) present a computer-assisted proof that two-layer ReLU networks have spurious local minima ; a recent work ( Yun et al. , 2019b ) have proven that neural networks with two-piece linear activations have infinite spurious local minima , but the results only apply to the networks with one hidden layer and one-dimensional outputs ; and a concurrent work ( Goldblum et al. , 2020 ) proves that for multi-layer perceptrons of any depth , the performance of every local minimum on the training data equals to a linear model , which is also verified by experiments . The proposed theorem is proved in three stages : ( 1 ) we prove that neural networks with one hidden layer and two-piece linear activations have spurious local minima ; ( 2 ) we extend the conditions to neural networks with arbitrary hidden layers and two-piece linear activations ; and ( 3 ) we further extend the conditions to neural networks with arbitrary depth and arbitrary piecewise linear activations . Since some parameters of the constructed spurious local minima are from continuous intervals , we have obtained infinitely many spurious local minima . At each stage , the proof follows a two-step strategy that : ( a ) constructs an infinite series of local minima ; and ( b ) constructs a point in the parameter space whose empirical risk is lower than the constructed local minimum in Step ( a ) . This strategy is inspired by Yun et al . ( 2019b ) but we have made significant and non-trivial development . Second , we draw a “ big picture ” for the loss surfaces of nonlinear neural networks . Soudry & Hoffer ( 2018 ) highlight a smooth and multilinear partition of the loss surfaces of neural networks . The nonlinearities in the piecewise linear activations partition the loss surface of any nonlinear neural network into multiple smooth and multilinear open cells . Specifically , every nonlinear point in the activation functions creates a group of the non-differentiable boundaries between the cells , while the linear parts of activations correspond to the smooth and multilinear interiors . Based on the partition , we discover a degenerate nature of the large amounts of local minima from the following aspects : • Every local minimum is globally minimal within a cell . This property demonstrates that the local geometry within every cell is similar to the global geometry of linear networks , although technically , they are substantially different . It applies to any one-hidden-layer neural network with two-piece linear activations for regression under convex loss . We rigorously prove this property in two stages : ( 1 ) we prove that within every cell , the empirical risk R̂ is convex with respect to a variable Ŵ mapped from the weights W by a mapping Q . Therefore , the local minima with respect to the variable Ŵ are also the global minima in the cell ; and then ( 2 ) we prove that the local optimality is maintained under the constructed mapping . Specifically , the local minima of the empirical risk R̂ with respect to the parameter W are also the local minima with respect to the variable Ŵ . We thereby prove this property by combining the convexity and the correspondence of the minima . This proof is technically novel and non-trivial , though the intuitions are natural . • Equivalence classes and quotient space of local minimum valleys . All local minima in a cell are concentrated as a local minimum valley : on a local minimum valley , all local minima are connected with each other by a continuous path , on which the empirical risk is invariant . Further , all these local minima constitute an equivalence class . This local minima valley may have several parallel valleys that are in the same equivalence class but do not appear because of the restraints from cell boundaries . If such constraints are ignored , all the equivalence classes constitute a quotient space . The constructed mapping Q is exactly the quotient map . This result coincides with the property of mode connectivity that the minima found by gradient-based methods are connected by a path in the parameter space with almost invariant empirical risk ( Garipov et al. , 2018 ; Draxler et al. , 2018 ; Kuditipudi et al. , 2019 ) . Additionally , this property suggests that we would need to study every local minimum valley as a whole . • Linear collapse . Linear neural networks are covered by our theories as a simplified case . When all activations are linear , the partitioned loss surface collapses to one single cell , in which all local minima are globally optimal , as suggested by the existing works on linear networks ( Kawaguchi , 2016 ; Baldi & Hornik , 1989 ; Lu & Kawaguchi , 2017 ; Freeman & Bruna , 2017 ; Zhou & Liang , 2018 ; Laurent & von Brecht , 2018 ; Yun et al. , 2018 ) . Notations . If M is a matrix , Mi , j denotes the ( i , j ) -th component of M . If M is a vector , Mi denotes the i-th component of M . Define Eij as a matrix in which the ( i , j ) -th component is 1 while all other components are 0 . Also , denote ei as a vector such that the i-th component is 1 while all others are 0 . Additionally , we define 1k ∈ Rk×1 is a vector whose components are all 1 , while those of 0n×m ∈ Rn×m ( or briefly , 0 ) are all 0 . For the brevity , [ i : j ] denotes { i , · · · , j } . 2 RELATED WORK . Some works suggest that linear neural networks have no spurious local minima . Kawaguchi ( 2016 ) proves that linear neural networks with squared loss do not have any spurious local minimum under three assumptions about the data matrix X and the label matrix Y : ( 1 ) both matrices XXT and XY T have full ranks ; and ( 2 ) the input layer is wider than the output layer ; and ( 3 ) the eigenvalues of matrix Y X > ( XXT ) −1 XY T are distinct with each other . Zhou & Liang ( 2018 ) give an analytic formulation of the critical points for the loss function of deep linear networks , and thereby obtain a group of equivalence conditions for that critical point is a global minimum . Lu & Kawaguchi ( 2017 ) prove the argument under one assumption that both matrices X and Y have full ranks , which is even more restrictive . However , in practice , the activations of most neural networks are not linear . The nonlinearities would make the loss surface extremely non-convex and even non-smooth and therefore far different from the linear case . The loss surfaces of over-parameterized neural networks have some special properties . Choromanska et al . ( 2015 ) empirically suggest that : ( 1 ) most local minima of over-parameterized networks are equivalent ; and ( 2 ) small-size networks have spurious local minima but the probability of finding one decreases rapidly with the network size . Li et al . ( 2018 ) prove that over-parameterized fullyconnected deep neural networks with continuous activation functions and convex , differentiable loss functions , have no bad strict local minimum . Nguyen et al . ( 2019 ) suggest that “ sufficiently overparameterized ” neural networks have no bad local valley under the cross-entropy loss . Nguyen ( 2019 ) further suggests that the global minima of sufficiently over-parameterized neural networks are connected within a unique valley . Many other works study the convergence , generalization , and other properties of stochastic gradient descent on the loss surfaces of over-parameterized networks ( Chizat & Bach ; Arora et al. , 2018 ; Brutzkus et al. , 2018 ; Du et al. , 2019 ; Soltanolkotabi et al. , 2018 ; Allen-Zhu et al. , 2019a ; b ; Oymak & Soltanolkotabi , 2019 ) . Many advances on the loss surfaces of neural networks are focused on other problems . Zhou & Feng ( 2018 ) and Mei et al . ( 2018 ) prove that the empirical risk surface and expected risk surface are linked . This correspondence highlights the value of investigating loss surfaces ( empirical risk surfaces ) to the study of generalization ( the gap between empirical risks to expected risks ) . Hanin & Rolnick ( 2019 ) demonstrate that the input space of neural networks with piecewise linear activations are partitioned by multiple regions , while our work focuses on the partition of the loss surface . Xie et al . ( 2017 ) proves that the training error and test error are upper bounded by the magnitude of the gradient , under the assumption that the geometry discrepancy of the parameterW is bounded . Sagun et al . ( 2016 ; 2018 ) present empirical results that the eigenvalues of the Hessian of the loss surface are two-fold : ( 1 ) a bulk centered closed to zero ; and ( 2 ) outliers away from the bulk . Kawaguchi & Kaelbling ( 2020 ) prove that we can eliminate the spurious local minima by adding one unit per output unit for almost any neural network in practice . Tian ( 2017 ) ; Andrychowicz et al . ( 2016 ) ; Soltanolkotabi ( 2017 ) ; Zhong et al . ( 2017 ) ; Brutzkus & Globerson ( 2017 ) ; Tian ( 2017 ) ; Li & Yuan ( 2017 ) ; Zou et al . ( 2019 ) ; Li & Liang ( 2018 ) ; Du et al . ( 2018a ; 2019 ) ; Zhang et al . ( 2019b ) ; Zhou et al . ( 2019 ) ; Wang et al . ( 2019 ) study the optimization methods for neural networks . Other relevant works include Sagun et al . ( 2016 ; 2018 ) ; Nguyen & Hein ( 2018 ) ; Du et al . ( 2018b ) ; Haeffele & Vidal ( 2017 ) ; Liang et al . ( 2018 ) ; Wu et al . ( 2018 ) ; Yun et al . ( 2019a ) ; Zhang et al . ( 2019a ) ; Kuditipudi et al . ( 2019 ) ; Garipov et al . ( 2018 ) ; Draxler et al . ( 2018 ) ; He et al . ( 2019 ) ; Kawaguchi & Kaelbling ( 2020 ) .

Summary: This paper studies the theoretical property of neural network's loss surface. The main contribution is to prove that the loss surface of every neural network (with arbitrary depth) with piecewise linear activations has infinite spurious local minima. Moreover, the paper further characterizes the partition of the local minima. More precisely, the loss surface is partitioned into multiple smooth and multilinear open cells and within each cell, the local minima are equally good. This result can also explain the linear neural network case where there is only one cell, implying that all local minima are global.