Path Space for Recurrent Neural Networks with ReLU Activations

1 INTRODUCTION . Over the past ten years , ReLU activations have become increasingly popular in various types of neural networks such as Multilayer Perceptron ( MLP ) ( Nair & Hinton , 2010 ; Glorot et al. , 2011 ; Neyshabur et al. , 2015a ) , Convolutional Neural Networks ( CNN ) ( ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) ) and Recurrent Neural Networks ( RNN ) ( Le et al. , 2015 ; Neyshabur et al. , 2016 ) . Theoretical studies on the MLP and CNN with ReLU activations , particularly on its positively scale-invariant ( PSI ) property have also been conducted ( Neyshabur et al. , 2016 ; 2015a ; Dinh et al. , 2017 ; Meng et al. , 2018 ) . Specifically , PSI property means if the incoming weights of a hidden node ( or a feature map for CNN ) are multiplied by a positive scalar c , and the outgoing weights of this hidden node ( or the feature map ) are divided by c , the output for arbitrary input will keep unchanged . However , conventional algorithms like stochastic gradient descent optimize the neural networks in the vector space of weights , which are not positively scale-invariant . This mismatch may lead to problems during the optimization process ( Neyshabur et al. , 2015a ; Meng et al. , 2018 ) , e.g. , in an unbalanced network ( i.e. , the norm of incoming weights to different units have relatively large differences ) , the training process of vanilla stochastic gradient descent will be significantly slowed down . To solve the above mismatch , some recent studies have been conducted on paths of ReLU neural networks ( Neyshabur et al. , 2015a ; Meng et al. , 2018 ) . The value of a path is defined as the multiplication of weights along the path which starts from an input node , successively crosses a hidden node at every layer , and finally ends at an output node . Obviously , the value of a path is positively scale-invariant . Thus , a new parameter space , called path space , for MLP and CNN have been proposed . Path space is a vector space constituted by the values of given paths called basis paths , whose values are sufficient to calculate the output of the MLP and CNN with ReLU activations . In a word , path space is a PSI parameter space and can sufficiently represent MLP and CNN with ReLU activation . In the work ( Meng et al. , 2018 ) , an algorithm is designed to identify the basis paths for MLP and CNN . Efficient optimization algorithms in path space have also been proposed , which iteratively update the values of basis paths according to the gradient of loss with respect to basis paths . Optimizing MLP or CNN with ReLU activations in this way can achieve better performances compared with traditional optimization methods in the weight space . However , the studies of path space especially the identification of basis paths depend on the network structure , so it can not be naturally extended to more complex network structures , like recurrent neural network ( RNN ) . RNN is a powerful neural network model to process sequential data and show excellent performance in various domains such as language modeling ( Mikolov et al. , 2010 ; Jozefowicz et al. , 2016 ) , machine translation ( Bahdanau et al. , 2014 ) and speech recognition ( Graves et al. , 2013 ; Miao et al. , 2015 ) . RNN with ReLU activations ( abbrev . ReLU RNN ) is also positively scale-invariant ( Neyshabur et al. , 2016 ) . Considering the superiority of optimizing MLP or CNN with ReLU activations in their path space , the path space for RNN is worth studying . In this work , we aim to construct path space for ReLU RNN so that we can employ the optimization algorithm in path space . Due to the recurrent structure , the paths in RNN depend on the time-steps . That means , if the sequence length of the input is not pre-given , the number of paths in RNN can not be determined . It makes the identification of basis paths from the undetermined number of paths difficult . To handle this , we propose leveraging the reduction graph of RNN to remove the influence of time-steps and theoretically develop path space for RNN in the following three steps . First , we define paths on reduction graph , whose number is fixed and not influenced by the sequence length . Second , we prove that the paths on reduction graph can serve as a sufficient representation of the RNN with ReLU activations . Third , we define path space for RNN which is composed by the values of basis paths in the reduction graph . Next , we design Skeleton Method to identify basis paths in reduction graph of RNN efficiently . Compared with MLP and CNN , the skeleton method for RNN needs extra steps to handle the recurrent weights . Specifically , the skeleton method for RNN contains two steps : it first identifies the basis paths in the reduction graph of RNN without recurrent weights ; then it constructs the basis paths that contain recurrent weight . We prove that paths selected by Skeleton Method are basis paths , and the number of basis paths which is also the dimension of the path space is # ( weights ) − # ( hidden nodes ) in the reduction graph of RNN . Finally , with the identified basis paths , we employ optimization algorithm in path space for RNN models , which operates update on basis paths according to the gradient of loss w.r.t basis paths . Our experiments on several benchmark datasets show that we can obtain significantly more effective RNN models in this way than using optimization methods in the weight space . It indicates that optimizing ReLU RNN in path space can solve the optimization problem brought by positively scale-invariant property and can indeed help optimization . To the best of our knowledge , our work is the first to construct the new parameter space , path space , for the RNN model and the first to optimize RNN in its path space . The paper is organized as follows : in Section 2 , we introduce the background about the path representation of ReLU RNN and related works ; in Section 3 , we formally define the basis path for ReLU RNN and propose using reduction graph as a tool to handle the difficulty when identifying the basis paths in ReLU RNN ; in Section 4 , we present our the basis path identification algorithm and the optimization algorithm in the path space ; in Section 5 , we show the experiment results to verify the effectiveness of our proposed algorithm ; finally , we conclude this paper in Section 6 . 2 BACKGROUND . In this section , we briefly review the path representation of recurrent neural networks with ReLU activations and the related works in this direction . 2.1 PATH REPRESENTATION OF RELU RNN . As shown in Figure 1 , the network structure of RNN can be regarded as a directed graph G ( N , E ) , where N = ∪tN t denotes the set of nodes and E denotes the set of edges , where N t = { NI , ( t ) 1 , N I , ( t ) 2 · · · , N H , ( t ) d+1 , N H , ( t ) d+2 , · · · , N O , ( t ) d+h+1 , N O , ( t ) d+h+2 · · ·N O , ( t ) d+h+K } denotes the set of nodes at the t-th time-step , the superscripts I , H , O means the input , hidden and output node , respectively . Here , d , h , K denote the dimension of input , hidden nodes and output , respectively , and the superscript t denotes the time-step . In the graph G ( N , E ) , there must be an edge between an input node and a hidden node that are at the same time step , a hidden node and an output node that are at the same time-step , and two hidden nodes whose time-steps are adjacent . A Path on the G ( N , E ) is defined in Definition 1 . Definition 1 ( Paths in RNN ) . A path in the recurrent neural network is defined as a list of nodes and the corresponding edges between adjacent nodes , which starts from an input node NI , ( t ) i0 to one output node NO , ( t+s ) is+1 with s ≥ 0 by successively crossing hidden nodes . Specifically , a path can be expressed as NI , ( t ) i0 → N H , ( t ) i1 → · · · → NH , ( t+s ) is+1 → N O , ( t+s ) is+2 , and there must be an edge between the adjacent nodes , where i. is the index of node along the path . For example , the red line in Figure 1 shows an example of the paths . For simplicity , we omit the explicit expression of paths and use p to denote the index of paths in the following context . The value of a path is defined as the multiplication of the values of weights on edges contained in the path . Specifically , for an edge ek ∈ E , we use wk to denote the weight on edge ek , then the value of path p is vp = ∏ ek∈p wk . Using W i , W o and W r to denote three weight matrices , use x ( t ) to denote the input and use nH , ( t ) , nO , ( t ) to denote the corresponding hidden and output values at nodes NH , ( t ) and NO , ( t ) , recurrent neural network computes the output vector nO , ( t ) at t-th time-step by a recurrent formulation as : nH , ( t ) = σ ( W ix ( t ) +W rnH , ( t−1 ) ) , ( 1 ) nO , ( t ) =W onH , ( t ) . ( 2 ) If the activation σ ( · ) is ReLU , i.e. , σ ( z ) = max ( z , 0 ) , we can get nO , ( t ) = W o · ( W ix ( t ) + W rnH , ( t−1 ) ) · diag ( I ( nH , ( t ) 1 > 0 ) , · · · , I ( n H , ( t ) h > 0 ) ) by combining Eq . ( 1 ) and Eq. ( 2 ) . 1 According to the definition of value of a path , the output of ReLU RNN can be calculated using the values of paths and its activation status as follows , n O , ( T ) k ( x ) = d∑ i=1 T∑ t=1 ∑ p∈PT , k , i , t vp · ap ( w ; x ) · x ( t ) i , ( 3 ) where the set PT , k , i , t consists of the paths that connects the k-th output at time-step T and the i-th input at time-step t with length T + 1 − t , vp is the value of path p , and ap ( w ; x ) =∏ N H , ( t ) i ∈p I ( nH , ( t ) i ( w ; x ) > 0 ) is the activation status of path p .

Summary: This paper aims to improve the training of recurrent neural networks (RNN) with ReLU activations by optimizing in the path space instead of the weight space. Studies on multi-layered perceptrons (MLP) and convolutional neural networks (CNN) have shown that these architectures are positively scale invariant (PSI), however traditional SGD optimizes in the weight space that does not have this property. It has been shown in prior work that this mismatch can slow down the optimization process for SGD and it has been demonstrated that optimizing in the so called path space, which has the PSI property, can be faster and more efficient.  The authors of this paper aim to extend an existing path-space framework (G-SGD) that is used for ReLU networks to facilitate RNNs.  To tackle the challenge posed by the time-dependency of RNNs, they use a static representation, called reduction graph, to define the path space. First, they prove that any path in the original RNN graph can be easily obtained from paths in its reduction graph by simple operations. Then, they define the basis for the path space in the reduction graph, and show that basis paths are sufficient to represent the output of the RNN. They propose a method (Skeleton method) to generate a basis path in the reduction graph and specify an equivalent of the G-SGD algorithm for RNNs.  Through numerical simulations they demonstrate that (1) G-SGD for RNNs has a fairly low additional computational cost compared to SGD and (2) G-SGD for RNNs achieves better test accuracy compared with SGD and an additional path space based approach.