Weight-space symmetry in neural network loss landscapes revisited

Neural network training depends on the structure of the underlying loss landscape , i.e . local minima , saddle points , flat plateaus , and loss barriers . In relation to the structure of the landscape , we study the permutation symmetry of neurons in each layer of a deep neural network , which gives rise not only to multiple equivalent global minima of the loss function but also to critical points in between partner minima . In a network of d−1 hidden layers with nk neurons in layers k = 1 , . . . , d , we construct continuous paths between equivalent global minima that lead through a ‘ permutation point ’ where the input and output weight vectors of two neurons in the same hidden layer k collide and interchange . We show that such permutation points are critical points which lie inside high-dimensional subspaces of equal loss , contributing to the global flatness of the landscape . We also find that a permutation point for the exchange of neurons i and j transits into a flat high-dimensional plateau that enables all nk ! permutations of neurons in a given layer k at the same loss value . Moreover , we introduce higher-order permutation points by exploiting the hierarchical structure in the loss landscapes of neural networks , and find that the number of K-th order permutation points is much larger than the ( already huge ) number of equivalent global minima – at least by a polynomial factor of order K. In two tasks , we demonstrate numerically with our path finding method that continuous paths between partner minima exist : first , in a toy network with a single hidden layer on a function approximation task and , second , in a multilayer network on the MNIST task . Our geometric approach yields a lower bound on the number of critical points generated by weight-space symmetries and provides a simple intuitive link between previous theoretical results and numerical observations . 1 INTRODUCTION . The structure of the loss landscape plays an important role in the optimization of neural network parameters . A large number of numerical ( Dauphin et al. , 2014 ; Goodfellow et al. , 2014 ; Li et al. , 2018 ; Sagun et al. , 2014 ; 2016 ; Ballard et al. , 2017 ; Garipov et al. , 2018 ; Draxler et al. , 2018 ; Sagun et al. , 2017 ; Baity-Jesi et al. , 2018 ) and theoretical ( Choromanska et al. , 2015 ; Rasmussen , 2003 ; Freeman and Bruna , 2016 ; Soudry and Carmon , 2016 ; Nguyen and Hein , 2017 ) studies have explored the properties of the loss landscape . In particular , in a multilayer network of d − 1 hidden layers with n neurons each , there are ( n ! ) d−1 equivalent configurations corresponding to the permutation of neuron indices in each layer of the network ( Goodfellow et al. , 2016 ; Bishop , 1995 ) . The permutation symmetries give rise to a loss landscape where any given global minimum in the weight space must have ( n ! ) d−1 − 1 completely equivalent partner minima . This property of neural network landscapes is called weight-space symmetry . Several ( Saad and Solla , 1995 ; Amari et al. , 2006 ; Wei et al. , 2008 ) works explored the implications of weight-space symmetry for training dynamics in two-layer networks and found that training dynamics slow down near the singular regions caused by weight-space symmetry . Dauphin et al . ( 2014 ) ; Orhan and Pitkow ( 2017 ) argue that optimization paths may get close to the singular regions induced by weight-space symmetry and this , in turn , slows down training for deep neural networks . Exploiting weight-space symmetries , we give insights into and partial explanations of three observations on neural network landscapes . Observation 1 . Training dynamics are slow near singular regions caused by weight-space symmetry and stochastic gradient descent might travel near these regions throughout training ( Saad and Solla , 1995 ; Wei et al. , 2008 ; Amari et al. , 2006 ; Dauphin et al. , 2014 ; Orhan and Pitkow , 2017 ) . Observation 2 . The Hessian of the loss function has numerous almost-zero eigenvalues throughout training , thus the landscape is flat in many directions ( Sagun et al. , 2017 ; Papyan , 2018 ; Ghorbani et al. , 2019 ) . Related to observation 1 and 2 , we prove the existence of numerous connected high-dimensional plateaus extending across the landscape due to weight-space symmetries . Observation 3 . The number of saddles can grow exponentially in neural network landscapes ( Auer et al. , 1996 ; Dauphin et al. , 2014 ; Choromanska et al. , 2015 ) . Related to observation 3 , we prove that there are at least polynomially many more saddles than the global minima due to weight-space symmetries in neural networks , without any further assumptions . In addition , we propose a novel low-loss path finding algorithm to find barriers between partner minima . We start from the known permutation symmetries and consider continuous low-loss paths that connect two equivalent global minima by merging the weight vectors of two neurons in a specific way . At a so-called permutation point , where the distance between the input and output weight vectors of the two neurons vanishes , the indices of the two neurons can be interchanged at no extra cost . After the change , the system returns on the ‘ mirrored ’ path back to the original configuration — except for the permutation of one pair of indices . Surprisingly , we find that we can permute all neuron indices in the same layer at the same cost as the loss at a permutation point reached by moving along the path that merges a single pair of neurons . These constant-loss permutations are possible because each permutation point lies in a high-dimensional plateau of critical points . Our theory can be extended to higher-order saddles and provides explicit lower bounds for the number of first- and higher-order permutation points . Numerically , we confirm the existence of first-order permutation saddles . In particular , the specific contributions of our work are : • A simple low-loss path-finding algorithm linking partner global minima via a permutation point , implemented by minimization under a single scalar constraint ( distance of weight vectors ) . • The theoretical characterization of permutation points , for example that these are critical points and several permutation points are connected via paths at equal loss . • A lower bound for the number of first- and higher-order permutation points and their corresponding plateaus . • Numerical demonstrations of the path finding method in multilayer neural networks trained on MNIST . 1.1 RELATED WORK . Structure of the landscape . For linear networks , it was shown that all the critical points – except for the global minimum – are saddles in the case of two-layer ( Baldi and Hornik , 1989 ) or multilayer networks ( Freeman and Bruna , 2016 ; Kawaguchi , 2016 ; Lu and Kawaguchi , 2017 ) . Interestingly , deep linear networks are reported to exhibit sharp transitions at the edges of extended plateaus ( Saxe et al. , 2013 ) , similar to the plateaus observed in deep nonlinear networks ( Goodfellow et al. , 2014 ) . For nonlinear multilayer networks , Choromanska et al . ( 2015 ) argue that all local minima lie below a certain loss value by drawing connections to the spherical spin-glass model . Improving upon this result , Soudry and Carmon ( 2016 ) ; Nguyen and Hein ( 2017 ) prove that almost all local minima are global minima for multilayer networks under mild over-parametrization assumptions . Bottom of the landscape . Another line of research studies the bottom of the landscape containing global minima and low-loss barriers between them . Freeman and Bruna ( 2016 ) prove the existence of low-loss paths connecting global minima for wide two-layer networks by upper-bounding the loss along the path with a parameter that depends on the number of parameters and data smoothness . Draxler et al . ( 2018 ) use Nudged Elastic Band method introduced in Jónsson et al . ( 1998 ) to connect independent minima and numerically find that the barrier vanishes consistently for increasing width and depth in DenseNet , ConvNet and ResNet architectures trained on CIFAR datasets . In a simultaneous work , Garipov et al . ( 2018 ) confirm that there is no significant barrier by connecting independent minima with polygonal chains . Training dynamics in the landscape . For general loss functions , Lee et al . ( 2016 ) show that gradient descent with sufficiently small step-size converges to local minima if all the saddles have at least one negative eigenvalue . For overparametrized neural networks , gradient descent converges to global minima without moving far from initialization ( Jacot et al. , 2018 ; Du et al. , 2018a ; b ) , thus suggesting convex-like behavior around random initialization . For the finite size networks , how training dynamics converge to a minima and in particular how fast they converge remain an open question . For soft-committee machines , it turns out that the initial learning dynamics are slowed down by correlation of hidden neurons ( Saad and Solla , 1995 ; Engel and Van den Broeck , 2001 ; Inoue et al. , 2003 ) . Amari et al . ( 2006 ) ; Wei et al . ( 2008 ) show that training dynamics slow down near singular regions due to weight-space symmetry . Dauphin et al . ( 2014 ) empirically argue that the large number of saddle points in the landscape makes training slow . Orhan and Pitkow ( 2017 ) numerically find that stochastic gradient descent may slow down near plateaus due to weight-space symmetry for deep ( 30 layers ) feedforward networks trained on CIFAR100 . In this paper we show that there is an impressively large number of permutation points . Each permutation point is a critical point ( either a local minimum or a saddle ) with a large number of flat directions , potentially linked to the empirically observed plateaus . In contrast to an earlier study by Fukumizu and Amari ( 2000 ) with a scalar output for two-layered networks where a line of critical points around the permutation point was reported , we study a deep network with d− 1 hidden layers and find multi-dimensional equal-loss plateaus . Moreover , we give a novel lower bound on the number of permutation points and construct sample paths between global minima using an algorithm that is different from previously used methods ( Garipov et al. , 2018 ; Draxler et al. , 2018 ) , since it exploits the symmetries at the permutation point . 1.2 PRELIMINARIES . We study multilayer neural networks f ( x ; θ ) with input x ∈ Rn0 , d layers of n1 , . . . , nd neurons per layer , parameters θ = { W ( k ) ∈ Rnk×nk−1 and b ( k ) ∈ Rnk : k ∈ { 1 , . . . , d } } ∈ Θ and nd-dimensional output f ( x ; θ ) = W ( d ) g ( · · · g ( W ( 2 ) g ( W ( 1 ) x+ b ( 1 ) ) + b ( 2 ) ) · · · ) + b ( d ) , ( 1 ) where g is a nonlinear activation function that operates component-wise on any vector . Definition 1 & 2 . ( Parameter vector ) We define the parameter vector ϑ ( k ) m of neuron m in layer k as the incoming weights to a neuron m in layer k concatenated with its bias term : ϑ ( k ) m = [ W ( k ) m,1 , . . . , W ( k ) m , nk−1 , b ( k ) m ] . ( Output weight vector ) We define the output weight vec- tor of neuron m in layer k as its outgoing weights from neuron m in layer k to the next layer : [ W ( k+1 ) 1 , m , . . . , W ( k+1 ) nk+1 , m ] . Since one can permute the neurons within each layer without changing the network function f ( x ; θ ) , any point θ induces a ‘ permutation set ’ . Definition 3 . ( Permutation set ) P ( θ ) = { θ′ ∈ Θ : W ′ ( k ) σ ( k ) ( i ) , σ ( k−1 ) ( j ) = W ( k ) i , j and b ′ ( k ) σ ( k ) ( i ) = b ( k ) i , k ∈ { 1 , . . . , d } } , ( 2 ) of points θ′ with f ( x ; θ ) = f ( x ; θ′ ) , where σ ( k ) are permutations of the neuron indices { 1 , . . . , nk } in ( hidden ) layer k where σ ( 0 ) and σ ( d ) are fixed trivial permutations , since we want to permute neither the indices of the input nor that of the output . We will use the notation θ′ = σ ( k ) l⇔m ( θ ) to indicate a point θ′ that differs from θ only by swapping neurons l and m in layer k. Note that the cardinality of a permutation set is maximal with |P ( θ ) | = ∏d−1 j=1 nj ! only if all parameter vectors ϑ ( k ) l 6= ϑ ( k ) m are distinct for every l 6= m and layer k ∈ { 1 , . . . , d − 1 } . In the following , we will assume that , at global minima , all parameter vectors are distinct at every layer k. Definition 4 . ( Permutation point ) Consider a minimum of a multilayer network with ( n1 , . . . , nk− 1 , . . . , nd ) neurons per layer . We can map this minimum to a configuration in the landscape of a multilayer network with ( n1 , . . . , nk , . . . , nd ) neurons per layer by duplicating one neuron m ∈ { 1 , . . . , nk − 1 } in layer k as follows : ( i ) substitute the parameter vector of the new neuron with a copy of the parameter vector ϑ ( k ) m of neuron m , ( ii ) replace the output weight vector of the new neuron and the duplicated neuron m with the initial output weight of neuron m rescaled by 12 , and ( iii ) keep all the other parameters the same . This new configuration where the parameter vectors and the output weight vectors of the new neuron and the duplicated neuron m are the same will be called a permutation point , denoted by θ ( k ) l⇔m , i.e . θ ( k ) l⇔m = σ ( k ) l⇔m ( θ ( k ) l⇔m ) . For training data D = { ( xµ , yµ ) : µ ∈ { 1 , . . . , T } } with targets yµ ∈ Y , we define a loss function L ( θ ; D ) = 1T ∑T µ=1 ` ( yµ , f ( xµ ; θ ) ) , where ` : Y × Rnd → R is some single-sample loss function . To simplify notation we will usually omit the explicit mentioning of the data in the loss function , i.e . L ( θ ) ≡ L ( θ ; D ) . A B

Summary: This paper studies the permutation symmetry of deep neural networks. It was known that by reordering neurons and their connections in each layer, the input -> output map the neural network represents can be preserved. This corresponded to a set of unconnected equivalent points in the weight space. The authors study the weight-space connectivity of these points through midpoints they call “permutation points”. They demonstrate that such points are members of high-dimensional manifolds of equivalent points. After that, they look at empirical experiments and explicitly construct a path between two equivalent weight-space points on a toy task and MNIST.