Escaping Flat Areas via Function-Preserving Structural Network Modifications

1 INTRODUCTION . Training deep neural networks involves the minimization of a non-convex , ( piecewise ) smooth error function defined over a high-dimensional space of parameters . Such objectives are most often optimized by stochastic gradient descent or variants thereof Duchi et al . ( 2011 ) ; Zeiler ( 2012 ) ; Kingma & Ba ( 2014 ) . One of the main difficulties occurring in this minimization problem is caused by the proliferation of saddle points , often surrounded by flat areas , which may substantially slow down learning Dauphin et al . ( 2014 ) . Escaping plateaus around saddle points is of critical importance not only to accelerate learning , but also because they can be mistaken for local minima , leading to poor solutions if learning were to be stopped too early . Recent approaches to analyze and deal with this challenge are often based on adding noise , e.g . Jin et al . ( 2017 ) . Here , we pursue a different philosophy . Inspired by the seminal paper of Fukumizu & Amari ( 2000 ) , we propose to expand a network by creating an additional hidden unit or filter , whenever optimization is slowing down too much . This unit is obtained by a simple cloning process of a feature or feature map , combined with a fast to execute optimization that scales outbound weights . The latter explicitly aims at opening-up an escape direction by maximizing the gradient norm in the expanded network . In particular , we exploit this ability to devise a method to escape saddle points , bad local minima and general flat areas in neural network optimization , which we call Tunnels . After taking a small number of update steps in the expanded network , we can decide to project back to the original network architecture . The hope is that the flat area or bad local minimum will have been left behind . Our contributions are as follows : • We extend the network embedding idea of Fukumizu & Amari ( 2000 ) to deep networks and CNNs . • We show that in contrast to the case of a three-layer perceptron , the newly constructed network has a non-zero gradient , even if the original network did not . • We derive a closed-form solution for cloning a unit or feature map to maximize the norm of this gradient . • We develop a practical version of our algorithm that requires no network structure modification and can be deployed as plug-and-play into any current deep learning framework . • We show experimentally that our method escapes effectively from flat areas during training . In Section 2 , we provide background , motivation , and explain in detail how to open the tunnel by solving an interposed optimization problem . We discuss the fully connected case as well as the practically relevant case of convolutional layers . In Section 3 , we describe how to close the tunnel , i.e . how to project back the expanded network onto one of the original dimensionality . We also propose a practical version of our idea that does not require an explicit modification of the network structure , based on a well-chosen reorganization of the weights . Finally , experiments and related work are presented in Sections 4 and 5 respectively . 2 TUNNEL OPENING . 2.1 REDUCIBLE NETWORKS . Feedforward neural networks can be defined in terms of a directed acylic graph ( U , E ) of computational units . Concretely , we assume that each unit u ∈ U is equiped with an activiation function σu and a bias parameter bu ∈ R. Moreover , each edge ( u , v ) ∈ E has an associated weight wuv ∈ R , where we implicitly set wuv = 0 for ( u , v ) 6∈ E. We can interpret a network computationally by assigning values xu to source or input units and propagating these values through the DAG using local computations with ridge function xv = σv ( bv + ∑ u wuvxu ) all the way to the sink or output units . Ignoring values at intermediate , hidden nodes , a network thus realizes a transfer function F ( · ; { w , b } ) : Rn → Rm , where n and m denote the number of input and output nodes , respectively . We call two networks input-output equivalent , if they define the same transfer function . It is a fundamental question whether the parameterization { w , b } 7→ F ( · ; { w , b } ) is one-to-one and , if not , what the sources of non-uniqueness are . As investigated in the classical work of Sussmann ( 1992 ) , one source of non-uniqueness can be tied to specific properties of the parameters , which make the network reducible to a smaller network . There are precisely three such conditions on hidden units , which do not depend on special properties of the activation function1 : Inconsequential unit u′ ∀v : wu′v = 0 ( 1a ) Constant unit u′ xu′ = const ( 1b ) Redundant units u′ , u xu′ = ±xu ( 1c ) ( 1a ) says that xu′ is ignored downstream as all outbound weights are zero . Note that this means , that the inbound weights can be chosen arbitrarily , in particular we may transform wu′v 7→ 0 ( ∀v ) , thus converting u′ into an isolated node , without altering the network ’ s transfer function . ( 1b ) implies that each child v of u′ can effectively be disconnected from u′ via the parameter transformation ( wu′v , bv ) 7→ ( 0 , bv + wu′vxu′ ) , rendering u′ inconsequential . Finally , ( 1c ) permits the invariant parameter transformation ( wu′v , wuv ) 7→ ( 0 , wuv ± wu′v ) , which reconnects all children of u′ with u instead , again rendering u′ inconsequential . A network is reducible , if at least one of its hidden nodes is reducible , otherwise it is irreducible . As shown rigorously in Sussmann ( 1992 ) for three-layer perceptrons with a single output , irreducible networks are minimal , i.e . they are not input-output equivalent to a network with fewer hidden units , and the remaining unidentifiability can be tied to a simple symmetry group ( e.g . permutations and sign changes of hidden units ) . 2.2 NETWORK EXPANSION . We are interested in embedding smaller networks into larger ones by adding a single hidden node , such that the transfer functions remains invariant . If the smaller network is irreducible , then by the above result , the new unit will have to be reducible . This tells us that such extensions can only be obtained through reversing the operations in ( 1 ) . 1Further redundancies may be introduced by specifics of σu . Table 1 : Parameter transformations for network extension . CASE SMALL EXTENDED NET ( A ) ∀p wpu′ = 0 wpu′ ∈ R ∀v wu′v = 0 wu′v = 0 ( B ) ∀p wpu′ = 0 wpu′ = 0 ∀v wu′v = 0 wu′v ∈ R ∀p wpu ∈ R wpu 7→ wpu ( C ) wpu′ = 0 wpu′ 7→ wpu ∀v wuv ∈ R wuv 7→ λvwuv wu′v = 0 wu′v 7→ ( 1− λv ) wuv Table 2 : Consequences of network extension on gradients . CASE SMALL EXTENDED NET ( I ) xu′ = 0 xu′ ∈ R , ∂E ∂wu′v ∈ R ( ∀v ) ( II ) δu′ = 0 δu′ ∈ R , ∂E ∂wpu′ ∈ R ( ∀p ) Assume that we have an error function E : Rm → R , inducing a risk R ( F ) = E [ E ( F ( x ) ) ] , defined empirically in practice . We consider the set of critical points of a network via the condition ∇ { w , b } R ( F ( · ; { w , b } ) ) = 0 . Fukumizu & Amari ( 2000 ) have investigated how the critical points of a network change after adding a reducible unit , one of the main results being that a global minimum of the smaller network can induce affine subspaces of local minima and saddle points in the extended network . This analysis only holds for the special case of a single downstream ( i.e . output ) unit . In general , embedding a smaller network in an input-output equivalent larger one may introduce directions of non-vanishing error derivatives , even at critical points . Critical points of a smaller network are not guaranteed to remain critical points after extension . This is the motivation of why one can potentially create escape tunnels from regions near poor critical points by such operations . Specifically , we have the options to add a node u′ : ( A ) with non-zero outbound weights , or ( B ) with non-zero inbound weights , or ( C ) by cloning it from an existing unit u and re-weighting its outbound weights . Formally , this can be realized by parameter transformations , as described2 in Table 1 . Let us first state what can be said about the partial derivatives , after a new unit has been added . In a DAG one can compute ∂E/∂wu′v for fixed inputs in three steps : ( i ) computing all activations xp via forward propagation ( ii ) computing deltas δp ≡ ∂E/∂xp via backpropagation from sink nodes to source nodes and ( iii ) in a local application of the chain rule ∂E ∂wu′v = δv x ′ v xu′ . ( 2 ) This results in the situation described in Table 2 for the activation of inbound ( I ) or outbound ( II ) weights . Intuitively speaking , by activating the outbound weights , the unit is no longer inconsequential and the inbound weights matter , whereas by activating the input weights , what was a constant ( and thus useless ) node before , turns into a selective and potentially useful feature for downstream computations . It is not clear how to best initialize the new weights . One option is to proceed in the spirit of methods such as Breiman ( 1993 ) and more broadly boosting algorithms Friedman et al . ( 2000 ) ; Buehlmann ( 2006 ) , which iteratively fit a residual loss , clamping all or some of the existing parameters . However , this seems ( i ) computationally unattractive on present-day computers , and ( ii ) pursues the philosophy of incremental learning , whereas our objective is not to greedily grow the network unit by unit , but rather to temporarily open an escape tunnel . 2.3 NODE CLONING . First , we look at a motivational example of an existing node u with two children nodes v1 , v2 . The backpropagation formula yields δu = δ1u + δ 2 u , where δ i u = δviwuvix ′ vi , i ∈ { 1 , 2 } . This splits the effect that u has on the error via the computational paths leading through v1 and v2 , respectively . The partial derivative of an inbound weight from some parent p of u in the original network is then simply ∂E ∂wpu = ( δ1u + δ 2 u ) x′u xp . ( 3 ) 2 α ∈ R expresses the fact that a parameter is unconstrained . Let us define the shortcuts api : = E [ δiux ′ u xp ] . At a critical point it holds that ap1 + ap2 = 0 , i.e . ap : = ap1 = −ap2 . If we clone u into u′ ( case ( C ) in Table 1 ) , then E [ ∂E ∂wpu ] = ( λ1 − λ2 ) ap = −E [ ∂E ∂wpu′ ] ( 4 ) In particular , we can see that by choosing λ1 = λ2 we remain at a critical point and that the contributions to the squared gradient norm are maximal for λ1 = 1 − λ2 ∈ { 0 , 1 } . This holds irrespective of the value of aw and hence irrespective of the chosen upstream unit p. Note that we could consider increasing weights even further by choosing λ1 > 1 and λ2 < 0 ( or vice versa ) . However this would artificially increase the gradient norm by making outbound weights of u and u′ arbitrarily large . In the general case , where u has in-degree K and out-degree L , we can use the same quantities api as above , which we can summarize in a matrix A = ( api ) ∈ RK×L . We can now define an objective quantifying the contribution of the two modified units u and u′ to the squared gradient norm obtained after cloning : H ( λ ) = ∑ p E [ ∂E ∂wpu ] 2 +E [ ∂E ∂wpu′ ] 2 ( 5 ) which can also be written as H ( λ ) = λTATAλ+ ( 1− λ ) TATA ( 1− λ ) . ( 6 ) Note that at a critical point , we have that the columns of A add up to the zero vector ∑ i a·i = 0 , in which case the above quantity simplifies asH ( λ ) = 2λTATAλ . At a critical point , this is a convex maximization problem which achieves its minimal value of 0 at any λ ∈ R1 . Maximizing H over the unit hyper-cube will in many cases degenerate to an integer problem , which may not be fast to solve . Also , we generally prefer to not make too drastic changes to the network , when opening the tunnel . Hence , we pursue the strategy of taking some λ0 ∈ R1 as a natural starting point and considering λt = λ0 + tλ∗ , where λ∗ is a unit length principal eigenvector of A > A . This is equivalent to maximizingH over the sphere of radius t , centered at λ0 . We choose λ0 = 121 as it minimizes H over R1 in the general case , i.e . at a non-critical point . Note that H ( λt ) = ( tρ ) 2 , where ρ is the largest eigenvalue . The tunnel opening we propose in order to escape saddles therefore consists in cloning an existing unit u into a new unit u′ , while assigning them the outbound weights of u respectively rescaled by λt and 1− λt , where t > 0 is a hyperparameter controlling the norm of the newly obtained gradient .

Summary: This paper presents a new strategy for escaping saddle points by adding and removing hidden units during training. The method essentially finds conditions where adding a hidden unit does not change the overall input-output map, and uses these as constraints to add a hidden unit that maximally increases the gradient norm (thus potentially getting learning unstuck). Experiments show that the method can improve training speed relative to the same network with randomly added new units.