G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space

1 INTRODUCTION . Over the past ten years , neural networks with rectified linear hidden units ( ReLU ) ( Hahnloser et al. , 2000 ) as activation functions have demonstrated the power in many important applications , such as information system ( Cheng et al. , 2016 ; Wang et al. , 2017 ) , image classification ( He et al. , 2016a ; Huang et al. , 2017 ) , text understanding ( Vaswani et al. , 2017 ) , etc . These networks are usually trained with Stochastic Gradient Descent ( SGD ) , where the gradient of loss function with respect to the weights can be efficiently computed via back propagation method ( Rumelhart et al. , 1986 ) . Recent studies ( Neyshabur et al. , 2015 ; LeCun et al. , 2015 ) show that ReLU networks have positively scale-invariant property , i.e. , if the incoming weights of a hidden node with ReLU activation are multiplied by a positive constant c and the outgoing weights are divided by c , the neural network with the new weights will generate exactly the same output as the old one for an arbitrary input . Conventional SGD optimizes ReLU neural networks in weight space . However , it is clear that weight vector is not positively scale-invariant . This mismatch may lead to problems during the optimization process ( Neyshabur et al. , 2015 ) . Then , a natural question is : can we construct a new vector space that is positively scale-invariant and sufficient to represent ReLU neural networks so as to better facilitate the optimization process ? In this paper , we provide positive answer to this question . ∗The notation ∗ denotes equal contribution . †This work was done when the author was visiting Microsoft Research Asia . We investigate the positively scale-invariant space to sufficiently represent ReLU neural networks by the following four steps . Firstly , we define the positive scaling operators and show that they form a transformation group ( denoted as G ) . The transformation group G will induce an equivalence relationship called positive scaling equivalence . Then , We found that the values of the paths are invariant to positive scaling operators . Furthermore , we prove that two weight vectors are positively scale-equivalent if and only if the values of the paths in one neural network equal to those in the other neural network , given the signs of some weights unchanged . That is to say , the values of all the paths can sufficiently represent a ReLU neural network . After that , we show that the path vectors are linearly dependent.1 We define the maximal group of paths which are linearly independent as basis path , which corresponds to the basis of the structure matrix constituted by the path vectors . Thus , the values of the basis paths are also positively scale-invariant and can sufficiently to represent the ReLU neural networks . We denote the vector whose coordinations are composed by values of basis paths as basis path value vector and call the vector space composed by basis path value vector as G-space . In addition , we prove that the dimension of G-space is `` H '' smaller comparing to the weight space , where H is the total number of hidden units in a multi-layer perceptron ( MLP ) or feature maps in a convolutional networks ( CNN ) . To sum up , we find G-space constituted by the values of the basis paths , which is positively scaleinvariant , can sufficiently represent the ReLU neural networks , and has a smaller dimension than the vector space of weights . Therefore , we propose to optimize the ReLU neural networks in its positively scale-invariant space , i.e. , G-space . We design a novel stochastic gradient descent algorithm in G-space ( abbreviated as G-SGD ) to optimize the ReLU neural networks utilizing the gradient with respect to the values of the basis paths . First , we design skeleton method to construct one group of the basis paths . Then , we develop inverse-chain rule and weight allocation to efficiently compute the gradient of the values of the basis paths by leveraging the back-propagation method . Please note that by using these techniques , there is very little additional computation overhead for G-SGD in comparison with the conventional SGD . We conduct experiments to show the effectiveness of G-SGD . First , we evaluate G-SGD of training deep convolutional networks on benchmark datasets and demonstrate that G-SGD achieves clearly better performance than baseline optimization algorithms . Second , we empirically test the performance of G-SGD with different degrees of positive scale-invariance . The experimental results show that the higher the positive scale-invariance is , the larger the performance improvement of G-SGD over SGD . This is consistent with that , the positive scale-invariance in weight space will negatively influence the optimization and our proposed G-SGD algorithm can effectively solve this problem . 2 BACKGROUNDS . 2.1 RELATED WORKS . There have been some prior works that study the positively scale-invariant property of ReLU networks and design algorithms that are positively scale-invariant . For example , Badrinarayanan et al . ( 2015 ) notice the positive scale-invariance in ReLU netowrks , and inspired by this , they design algorithms to normalize gradients by layer-wise weight norm . Du et al . ( 2018 ) study the gradient flow in MLP or CNN models with linear , ReLU or Leaky ReLU activation , and prove the squared norms of gradient across different layers are automatically balanced and remained invariant in gradient descent with infinitesimal step size . In our work , we do not care whether the models are balanced or not . Besides , many other optimization algorithms also have positively scale-invariant property such as Newton ’ s method and natural gradient descent . The most related work is Path-SGD ( Neyshabur et al. , 2015 ) , which also considers the geometry inspired by path norm . This work is different from ours : 1 ) they regularize the gradient in weight space by path norm while we optimize the loss function directly in a positively scale-invariant space ; 2 ) they do not consider the dependency between paths and it ’ s hard for them to compute the exactly path-regularized gradients . Different from the previous works , we propose to directly optimize the ReLU networks in its positively scale-invariant space , instead of 1A path vector is represented by one element in { 0 , 1 } m , where m is the number of weights . Please check the details in Section 2.2. optimizing in the weight space which is not positive scale-invariant . To the best of our knowledge , at the first time , we solve this mismatch by theoretical analysis and an effective and efficient algorithm . 2.2 RELU NEURAL NETWORKS . Let Nw ( x ) : X → Y denote a L-layer multi-layer perceptron ( MLP ) with weight w ∈ W ⊂ Rm , the input space X ⊂ Rd and the output space Y ⊂ RK . In the l-th layer ( l = 0 , · · · , L ) , there are hl nodes . It is clear that , h0 = d , hL = K. We denote the il-th node and its value as Olil and o l il , respectively . We usewl to denote the weight matrix between layer l−1 and layer l , and use wl ( il−1 , il ) to denote the weight connecting nodes Ol−1il−1 and O l il . The values of the nodes are propagated as ol = σ ( ( wl ) T ol−1 ) , where σ ( · ) = max ( · , 0 ) is the ReLU activation function . We use ( i0 , · · · , iL ) to denote the path starting from input feature node O0i0 to output node O L iL passing though hidden nodes O1i1 , · · · , O L−1 iL−1 . We can also regard the network structure as a directed graph ( O , E ) , whereO = { O1 , · · · , OH+d+K } is the set of nodes where H denotes the number of hidden nodes and E = { eij } denote the set of edges in a network where eij denotes the edge pointing to Oj from nodes Oi . We use we , e ∈ E to denote the weight on edge e. If |E| = m , the weights compose a vector w = ( w1 , · · · , wm ) T . We define a path as a vector p = ( p1 , · · · , pm ) T and if the edge e is contained in path p , pe = 1 ; otherwise pe = 0 . Because a path crosses L edges for an L-layer MLP , there are L elements with value 1 and others elements with value 0 . Using these notations , the value of path p can be calculated as vp ( w ) = ∏m i=1 w pi i and the activation status of path p can be calculated as ap ( x ; w ) =∏ j : peij=1 I ( oj ( x ; w ) > 0 ) . We denote the set composed by all paths as P and the set composed by paths which contain edge connecting the i0-th input node and the k-th output node as Pi0 , k . Thus , the output can be computed as follows : Nkw ( x ) = d∑ i0=1 ∑ p∈Pi0 , k vp ( w ) · ap ( x ; w ) · xi0 . ( 1 ) 3 POSITIVELY SCALE-INVARIANT SPACE OF RELU NETWORKS . In this section , we first define positive scaling transformation group and the equivalence class induced by this group . Then we study the invariant space under positive scaling transformation group of ReLU networks and study its dimension . 3.1 POSITIVE SCALING TRANSFORMATION GROUP . We formally define the positive scaling operator . We first define a node positive scaling operator gc , O ( w ) : W →W with constant c > 0 and one hidden node O as w̃ = gc , Olil ( w ) , where w̃l ( il−1 , il ) = c · wl ( il−1 , il ) for il−1 = 1 , · · · , hl−1 ; w̃l+1 ( il , il+1 ) = 1c · w l+1 ( il , il+1 ) for il+1 = 1 , · · · , hl+1 ; and values of other elements of w̃ are the same with w. Definition 3.1 ( positive scaling operator ) Suppose that { O1 , · · · , OH } is the set of all the hidden nodes in the network where H denotes the number of hidden nodes . A positive scaling operator g ( c1 , ··· , cH ) ( · ) : W →W with c1 , · · · , cH ∈ R + is defined as g ( c1 , ··· , cH ) ( · ) : = gc1 , O1 ◦ gc2 , O2 ◦ · · · ◦ gcH , OH ( · ) , where ◦ denotes function composition . We then collect all the g ( c1 , ··· , cH ) ( · ) together to form a set G : = { g ( c1 , ··· , cH ) ( · ) : c1 , · · · , cH ∈ R + } . It is easy to check that G together with the operation `` ◦ '' is a group which is called positive scaling transformation group , and we call the group action of G onW as G-action . ( Please refer to Section 8 in Appendix . ) Clearly , if there exists an operator g ∈ G to make w = g ( w′ ) , ReLU networks Nw and Nw′ will generate the same output for any fixed input x . We define the positive scaling equivalence induced by G-action . Definition 3.2 Consider two ReLU networks with weights w , w′ ∈ W and the positive scaling transformation group G. We say w and w′ are positively scale-equivalent if ∃g ∈ G such that w = g ( w′ ) , denote as w ∼G w′ . Given G-action onW , the equivalence relation `` ∼G '' partitionsW into G-equivalent classes . The following theorem shows that the sufficient and necessary condition for ReLU networks in the same equivalent class is that they have the same values and activation status of paths . Theorem 3.3 Consider two ReLU neural networks with weights w , w′ ∈ W . We have that w ∼G w′ iff for ∀ path p ∈ P and any fixed input x ∈ X , we have vp ( w ) = vp ( w′ ) and ap ( x ; w ) = ap ( x ; w ) . Invariant variables for a group action are important and widely studied in group theory and geometry . We say a function f : W → R is invariant variable of G-action if f ( w ) = f ( g ( w ) ) , ∀g ∈ G. Based on Theorem 3.3 , a direct corollary is that values and activation status of paths are invariant variables under G-action . Considering that 1 ) values of paths are G-invariant variables while the weights aren ’ t ; 2 ) values of paths together with the activation status determines an positive scale-equivalent class , and are sufficient to determine the loss , we propose to optimize the values of paths instead of weights .

Summary: In prior deep learning papers it has been observed that ReLU networks are positively scale invariant due to the non-negative homogeneity property of the max(0, x) function. The present paper proposes to change the optimization procedure so that it is done in a space where this invariance is preserved (in contrast to the weight space, where it is not). To do so, the authors define the group G of positive scaling operators, and note that the "value of path" (product of weights along the path) is G-invariant and together with "activation status of paths" allows for the definition of an equivalence class. They then build a G-space which has fewer dimensions than the weight space, and proposed g-SGD to optimize the network in this space. In g-SGD gradients are computed normally, then projected to G-space via a sparse matrix in order to update the values of paths. The weights are then updated based on a "weight allocation method" that involves the inverse projection.