Weight-space symmetry in neural network loss landscapes revisited

Neural network training depends on the structure of the underlying loss landscape , i.e . local minima , saddle points , flat plateaus , and loss barriers . In relation to the structure of the landscape , we study the permutation symmetry of neurons in each layer of a deep neural network , which gives rise not only to multiple equivalent global minima of the loss function but also to critical points in between partner minima . In a network of d−1 hidden layers with nk neurons in layers k = 1 , . . . , d , we construct continuous paths between equivalent global minima that lead through a ‘ permutation point ’ where the input and output weight vectors of two neurons in the same hidden layer k collide and interchange . We show that such permutation points are critical points which lie inside high-dimensional subspaces of equal loss , contributing to the global flatness of the landscape . We also find that a permutation point for the exchange of neurons i and j transits into a flat high-dimensional plateau that enables all nk ! permutations of neurons in a given layer k at the same loss value . Moreover , we introduce higher-order permutation points by exploiting the hierarchical structure in the loss landscapes of neural networks , and find that the number of K-th order permutation points is much larger than the ( already huge ) number of equivalent global minima – at least by a polynomial factor of order K. In two tasks , we demonstrate numerically with our path finding method that continuous paths between partner minima exist : first , in a toy network with a single hidden layer on a function approximation task and , second , in a multilayer network on the MNIST task . Our geometric approach yields a lower bound on the number of critical points generated by weight-space symmetries and provides a simple intuitive link between previous theoretical results and numerical observations . 1 INTRODUCTION . The structure of the loss landscape plays an important role in the optimization of neural network parameters . A large number of numerical ( Dauphin et al. , 2014 ; Goodfellow et al. , 2014 ; Li et al. , 2018 ; Sagun et al. , 2014 ; 2016 ; Ballard et al. , 2017 ; Garipov et al. , 2018 ; Draxler et al. , 2018 ; Sagun et al. , 2017 ; Baity-Jesi et al. , 2018 ) and theoretical ( Choromanska et al. , 2015 ; Rasmussen , 2003 ; Freeman and Bruna , 2016 ; Soudry and Carmon , 2016 ; Nguyen and Hein , 2017 ) studies have explored the properties of the loss landscape . In particular , in a multilayer network of d − 1 hidden layers with n neurons each , there are ( n ! ) d−1 equivalent configurations corresponding to the permutation of neuron indices in each layer of the network ( Goodfellow et al. , 2016 ; Bishop , 1995 ) . The permutation symmetries give rise to a loss landscape where any given global minimum in the weight space must have ( n ! ) d−1 − 1 completely equivalent partner minima . This property of neural network landscapes is called weight-space symmetry . Several ( Saad and Solla , 1995 ; Amari et al. , 2006 ; Wei et al. , 2008 ) works explored the implications of weight-space symmetry for training dynamics in two-layer networks and found that training dynamics slow down near the singular regions caused by weight-space symmetry . Dauphin et al . ( 2014 ) ; Orhan and Pitkow ( 2017 ) argue that optimization paths may get close to the singular regions induced by weight-space symmetry and this , in turn , slows down training for deep neural networks . Exploiting weight-space symmetries , we give insights into and partial explanations of three observations on neural network landscapes . Observation 1 . Training dynamics are slow near singular regions caused by weight-space symmetry and stochastic gradient descent might travel near these regions throughout training ( Saad and Solla , 1995 ; Wei et al. , 2008 ; Amari et al. , 2006 ; Dauphin et al. , 2014 ; Orhan and Pitkow , 2017 ) . Observation 2 . The Hessian of the loss function has numerous almost-zero eigenvalues throughout training , thus the landscape is flat in many directions ( Sagun et al. , 2017 ; Papyan , 2018 ; Ghorbani et al. , 2019 ) . Related to observation 1 and 2 , we prove the existence of numerous connected high-dimensional plateaus extending across the landscape due to weight-space symmetries . Observation 3 . The number of saddles can grow exponentially in neural network landscapes ( Auer et al. , 1996 ; Dauphin et al. , 2014 ; Choromanska et al. , 2015 ) . Related to observation 3 , we prove that there are at least polynomially many more saddles than the global minima due to weight-space symmetries in neural networks , without any further assumptions . In addition , we propose a novel low-loss path finding algorithm to find barriers between partner minima . We start from the known permutation symmetries and consider continuous low-loss paths that connect two equivalent global minima by merging the weight vectors of two neurons in a specific way . At a so-called permutation point , where the distance between the input and output weight vectors of the two neurons vanishes , the indices of the two neurons can be interchanged at no extra cost . After the change , the system returns on the ‘ mirrored ’ path back to the original configuration — except for the permutation of one pair of indices . Surprisingly , we find that we can permute all neuron indices in the same layer at the same cost as the loss at a permutation point reached by moving along the path that merges a single pair of neurons . These constant-loss permutations are possible because each permutation point lies in a high-dimensional plateau of critical points . Our theory can be extended to higher-order saddles and provides explicit lower bounds for the number of first- and higher-order permutation points . Numerically , we confirm the existence of first-order permutation saddles . In particular , the specific contributions of our work are : • A simple low-loss path-finding algorithm linking partner global minima via a permutation point , implemented by minimization under a single scalar constraint ( distance of weight vectors ) . • The theoretical characterization of permutation points , for example that these are critical points and several permutation points are connected via paths at equal loss . • A lower bound for the number of first- and higher-order permutation points and their corresponding plateaus . • Numerical demonstrations of the path finding method in multilayer neural networks trained on MNIST . 1.1 RELATED WORK . Structure of the landscape . For linear networks , it was shown that all the critical points – except for the global minimum – are saddles in the case of two-layer ( Baldi and Hornik , 1989 ) or multilayer networks ( Freeman and Bruna , 2016 ; Kawaguchi , 2016 ; Lu and Kawaguchi , 2017 ) . Interestingly , deep linear networks are reported to exhibit sharp transitions at the edges of extended plateaus ( Saxe et al. , 2013 ) , similar to the plateaus observed in deep nonlinear networks ( Goodfellow et al. , 2014 ) . For nonlinear multilayer networks , Choromanska et al . ( 2015 ) argue that all local minima lie below a certain loss value by drawing connections to the spherical spin-glass model . Improving upon this result , Soudry and Carmon ( 2016 ) ; Nguyen and Hein ( 2017 ) prove that almost all local minima are global minima for multilayer networks under mild over-parametrization assumptions . Bottom of the landscape . Another line of research studies the bottom of the landscape containing global minima and low-loss barriers between them . Freeman and Bruna ( 2016 ) prove the existence of low-loss paths connecting global minima for wide two-layer networks by upper-bounding the loss along the path with a parameter that depends on the number of parameters and data smoothness . Draxler et al . ( 2018 ) use Nudged Elastic Band method introduced in Jónsson et al . ( 1998 ) to connect independent minima and numerically find that the barrier vanishes consistently for increasing width and depth in DenseNet , ConvNet and ResNet architectures trained on CIFAR datasets . In a simultaneous work , Garipov et al . ( 2018 ) confirm that there is no significant barrier by connecting independent minima with polygonal chains . Training dynamics in the landscape . For general loss functions , Lee et al . ( 2016 ) show that gradient descent with sufficiently small step-size converges to local minima if all the saddles have at least one negative eigenvalue . For overparametrized neural networks , gradient descent converges to global minima without moving far from initialization ( Jacot et al. , 2018 ; Du et al. , 2018a ; b ) , thus suggesting convex-like behavior around random initialization . For the finite size networks , how training dynamics converge to a minima and in particular how fast they converge remain an open question . For soft-committee machines , it turns out that the initial learning dynamics are slowed down by correlation of hidden neurons ( Saad and Solla , 1995 ; Engel and Van den Broeck , 2001 ; Inoue et al. , 2003 ) . Amari et al . ( 2006 ) ; Wei et al . ( 2008 ) show that training dynamics slow down near singular regions due to weight-space symmetry . Dauphin et al . ( 2014 ) empirically argue that the large number of saddle points in the landscape makes training slow . Orhan and Pitkow ( 2017 ) numerically find that stochastic gradient descent may slow down near plateaus due to weight-space symmetry for deep ( 30 layers ) feedforward networks trained on CIFAR100 . In this paper we show that there is an impressively large number of permutation points . Each permutation point is a critical point ( either a local minimum or a saddle ) with a large number of flat directions , potentially linked to the empirically observed plateaus . In contrast to an earlier study by Fukumizu and Amari ( 2000 ) with a scalar output for two-layered networks where a line of critical points around the permutation point was reported , we study a deep network with d− 1 hidden layers and find multi-dimensional equal-loss plateaus . Moreover , we give a novel lower bound on the number of permutation points and construct sample paths between global minima using an algorithm that is different from previously used methods ( Garipov et al. , 2018 ; Draxler et al. , 2018 ) , since it exploits the symmetries at the permutation point . 1.2 PRELIMINARIES . We study multilayer neural networks f ( x ; θ ) with input x ∈ Rn0 , d layers of n1 , . . . , nd neurons per layer , parameters θ = { W ( k ) ∈ Rnk×nk−1 and b ( k ) ∈ Rnk : k ∈ { 1 , . . . , d } } ∈ Θ and nd-dimensional output f ( x ; θ ) = W ( d ) g ( · · · g ( W ( 2 ) g ( W ( 1 ) x+ b ( 1 ) ) + b ( 2 ) ) · · · ) + b ( d ) , ( 1 ) where g is a nonlinear activation function that operates component-wise on any vector . Definition 1 & 2 . ( Parameter vector ) We define the parameter vector ϑ ( k ) m of neuron m in layer k as the incoming weights to a neuron m in layer k concatenated with its bias term : ϑ ( k ) m = [ W ( k ) m,1 , . . . , W ( k ) m , nk−1 , b ( k ) m ] . ( Output weight vector ) We define the output weight vec- tor of neuron m in layer k as its outgoing weights from neuron m in layer k to the next layer : [ W ( k+1 ) 1 , m , . . . , W ( k+1 ) nk+1 , m ] . Since one can permute the neurons within each layer without changing the network function f ( x ; θ ) , any point θ induces a ‘ permutation set ’ . Definition 3 . ( Permutation set ) P ( θ ) = { θ′ ∈ Θ : W ′ ( k ) σ ( k ) ( i ) , σ ( k−1 ) ( j ) = W ( k ) i , j and b ′ ( k ) σ ( k ) ( i ) = b ( k ) i , k ∈ { 1 , . . . , d } } , ( 2 ) of points θ′ with f ( x ; θ ) = f ( x ; θ′ ) , where σ ( k ) are permutations of the neuron indices { 1 , . . . , nk } in ( hidden ) layer k where σ ( 0 ) and σ ( d ) are fixed trivial permutations , since we want to permute neither the indices of the input nor that of the output . We will use the notation θ′ = σ ( k ) l⇔m ( θ ) to indicate a point θ′ that differs from θ only by swapping neurons l and m in layer k. Note that the cardinality of a permutation set is maximal with |P ( θ ) | = ∏d−1 j=1 nj ! only if all parameter vectors ϑ ( k ) l 6= ϑ ( k ) m are distinct for every l 6= m and layer k ∈ { 1 , . . . , d − 1 } . In the following , we will assume that , at global minima , all parameter vectors are distinct at every layer k. Definition 4 . ( Permutation point ) Consider a minimum of a multilayer network with ( n1 , . . . , nk− 1 , . . . , nd ) neurons per layer . We can map this minimum to a configuration in the landscape of a multilayer network with ( n1 , . . . , nk , . . . , nd ) neurons per layer by duplicating one neuron m ∈ { 1 , . . . , nk − 1 } in layer k as follows : ( i ) substitute the parameter vector of the new neuron with a copy of the parameter vector ϑ ( k ) m of neuron m , ( ii ) replace the output weight vector of the new neuron and the duplicated neuron m with the initial output weight of neuron m rescaled by 12 , and ( iii ) keep all the other parameters the same . This new configuration where the parameter vectors and the output weight vectors of the new neuron and the duplicated neuron m are the same will be called a permutation point , denoted by θ ( k ) l⇔m , i.e . θ ( k ) l⇔m = σ ( k ) l⇔m ( θ ( k ) l⇔m ) . For training data D = { ( xµ , yµ ) : µ ∈ { 1 , . . . , T } } with targets yµ ∈ Y , we define a loss function L ( θ ; D ) = 1T ∑T µ=1 ` ( yµ , f ( xµ ; θ ) ) , where ` : Y × Rnd → R is some single-sample loss function . To simplify notation we will usually omit the explicit mentioning of the data in the loss function , i.e . L ( θ ) ≡ L ( θ ; D ) . A B

Summary: The paper presented a method for studying the landscape of the loss function w.r.t. parameters in a neural network from the perspective of weight-space symmetry. The detailed method includes constructing/optimising a low-loss path in between two parameter vectors (incoming connections from the previous layer) of two neurons respectively, and then set the output weight vectors (outgoing connections to the next layer) to be the same without changing the output of the current layer.