Path Space for Recurrent Neural Networks with ReLU Activations

1 INTRODUCTION . Over the past ten years , ReLU activations have become increasingly popular in various types of neural networks such as Multilayer Perceptron ( MLP ) ( Nair & Hinton , 2010 ; Glorot et al. , 2011 ; Neyshabur et al. , 2015a ) , Convolutional Neural Networks ( CNN ) ( ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) ) and Recurrent Neural Networks ( RNN ) ( Le et al. , 2015 ; Neyshabur et al. , 2016 ) . Theoretical studies on the MLP and CNN with ReLU activations , particularly on its positively scale-invariant ( PSI ) property have also been conducted ( Neyshabur et al. , 2016 ; 2015a ; Dinh et al. , 2017 ; Meng et al. , 2018 ) . Specifically , PSI property means if the incoming weights of a hidden node ( or a feature map for CNN ) are multiplied by a positive scalar c , and the outgoing weights of this hidden node ( or the feature map ) are divided by c , the output for arbitrary input will keep unchanged . However , conventional algorithms like stochastic gradient descent optimize the neural networks in the vector space of weights , which are not positively scale-invariant . This mismatch may lead to problems during the optimization process ( Neyshabur et al. , 2015a ; Meng et al. , 2018 ) , e.g. , in an unbalanced network ( i.e. , the norm of incoming weights to different units have relatively large differences ) , the training process of vanilla stochastic gradient descent will be significantly slowed down . To solve the above mismatch , some recent studies have been conducted on paths of ReLU neural networks ( Neyshabur et al. , 2015a ; Meng et al. , 2018 ) . The value of a path is defined as the multiplication of weights along the path which starts from an input node , successively crosses a hidden node at every layer , and finally ends at an output node . Obviously , the value of a path is positively scale-invariant . Thus , a new parameter space , called path space , for MLP and CNN have been proposed . Path space is a vector space constituted by the values of given paths called basis paths , whose values are sufficient to calculate the output of the MLP and CNN with ReLU activations . In a word , path space is a PSI parameter space and can sufficiently represent MLP and CNN with ReLU activation . In the work ( Meng et al. , 2018 ) , an algorithm is designed to identify the basis paths for MLP and CNN . Efficient optimization algorithms in path space have also been proposed , which iteratively update the values of basis paths according to the gradient of loss with respect to basis paths . Optimizing MLP or CNN with ReLU activations in this way can achieve better performances compared with traditional optimization methods in the weight space . However , the studies of path space especially the identification of basis paths depend on the network structure , so it can not be naturally extended to more complex network structures , like recurrent neural network ( RNN ) . RNN is a powerful neural network model to process sequential data and show excellent performance in various domains such as language modeling ( Mikolov et al. , 2010 ; Jozefowicz et al. , 2016 ) , machine translation ( Bahdanau et al. , 2014 ) and speech recognition ( Graves et al. , 2013 ; Miao et al. , 2015 ) . RNN with ReLU activations ( abbrev . ReLU RNN ) is also positively scale-invariant ( Neyshabur et al. , 2016 ) . Considering the superiority of optimizing MLP or CNN with ReLU activations in their path space , the path space for RNN is worth studying . In this work , we aim to construct path space for ReLU RNN so that we can employ the optimization algorithm in path space . Due to the recurrent structure , the paths in RNN depend on the time-steps . That means , if the sequence length of the input is not pre-given , the number of paths in RNN can not be determined . It makes the identification of basis paths from the undetermined number of paths difficult . To handle this , we propose leveraging the reduction graph of RNN to remove the influence of time-steps and theoretically develop path space for RNN in the following three steps . First , we define paths on reduction graph , whose number is fixed and not influenced by the sequence length . Second , we prove that the paths on reduction graph can serve as a sufficient representation of the RNN with ReLU activations . Third , we define path space for RNN which is composed by the values of basis paths in the reduction graph . Next , we design Skeleton Method to identify basis paths in reduction graph of RNN efficiently . Compared with MLP and CNN , the skeleton method for RNN needs extra steps to handle the recurrent weights . Specifically , the skeleton method for RNN contains two steps : it first identifies the basis paths in the reduction graph of RNN without recurrent weights ; then it constructs the basis paths that contain recurrent weight . We prove that paths selected by Skeleton Method are basis paths , and the number of basis paths which is also the dimension of the path space is # ( weights ) − # ( hidden nodes ) in the reduction graph of RNN . Finally , with the identified basis paths , we employ optimization algorithm in path space for RNN models , which operates update on basis paths according to the gradient of loss w.r.t basis paths . Our experiments on several benchmark datasets show that we can obtain significantly more effective RNN models in this way than using optimization methods in the weight space . It indicates that optimizing ReLU RNN in path space can solve the optimization problem brought by positively scale-invariant property and can indeed help optimization . To the best of our knowledge , our work is the first to construct the new parameter space , path space , for the RNN model and the first to optimize RNN in its path space . The paper is organized as follows : in Section 2 , we introduce the background about the path representation of ReLU RNN and related works ; in Section 3 , we formally define the basis path for ReLU RNN and propose using reduction graph as a tool to handle the difficulty when identifying the basis paths in ReLU RNN ; in Section 4 , we present our the basis path identification algorithm and the optimization algorithm in the path space ; in Section 5 , we show the experiment results to verify the effectiveness of our proposed algorithm ; finally , we conclude this paper in Section 6 . 2 BACKGROUND . In this section , we briefly review the path representation of recurrent neural networks with ReLU activations and the related works in this direction . 2.1 PATH REPRESENTATION OF RELU RNN . As shown in Figure 1 , the network structure of RNN can be regarded as a directed graph G ( N , E ) , where N = ∪tN t denotes the set of nodes and E denotes the set of edges , where N t = { NI , ( t ) 1 , N I , ( t ) 2 · · · , N H , ( t ) d+1 , N H , ( t ) d+2 , · · · , N O , ( t ) d+h+1 , N O , ( t ) d+h+2 · · ·N O , ( t ) d+h+K } denotes the set of nodes at the t-th time-step , the superscripts I , H , O means the input , hidden and output node , respectively . Here , d , h , K denote the dimension of input , hidden nodes and output , respectively , and the superscript t denotes the time-step . In the graph G ( N , E ) , there must be an edge between an input node and a hidden node that are at the same time step , a hidden node and an output node that are at the same time-step , and two hidden nodes whose time-steps are adjacent . A Path on the G ( N , E ) is defined in Definition 1 . Definition 1 ( Paths in RNN ) . A path in the recurrent neural network is defined as a list of nodes and the corresponding edges between adjacent nodes , which starts from an input node NI , ( t ) i0 to one output node NO , ( t+s ) is+1 with s ≥ 0 by successively crossing hidden nodes . Specifically , a path can be expressed as NI , ( t ) i0 → N H , ( t ) i1 → · · · → NH , ( t+s ) is+1 → N O , ( t+s ) is+2 , and there must be an edge between the adjacent nodes , where i. is the index of node along the path . For example , the red line in Figure 1 shows an example of the paths . For simplicity , we omit the explicit expression of paths and use p to denote the index of paths in the following context . The value of a path is defined as the multiplication of the values of weights on edges contained in the path . Specifically , for an edge ek ∈ E , we use wk to denote the weight on edge ek , then the value of path p is vp = ∏ ek∈p wk . Using W i , W o and W r to denote three weight matrices , use x ( t ) to denote the input and use nH , ( t ) , nO , ( t ) to denote the corresponding hidden and output values at nodes NH , ( t ) and NO , ( t ) , recurrent neural network computes the output vector nO , ( t ) at t-th time-step by a recurrent formulation as : nH , ( t ) = σ ( W ix ( t ) +W rnH , ( t−1 ) ) , ( 1 ) nO , ( t ) =W onH , ( t ) . ( 2 ) If the activation σ ( · ) is ReLU , i.e. , σ ( z ) = max ( z , 0 ) , we can get nO , ( t ) = W o · ( W ix ( t ) + W rnH , ( t−1 ) ) · diag ( I ( nH , ( t ) 1 > 0 ) , · · · , I ( n H , ( t ) h > 0 ) ) by combining Eq . ( 1 ) and Eq. ( 2 ) . 1 According to the definition of value of a path , the output of ReLU RNN can be calculated using the values of paths and its activation status as follows , n O , ( T ) k ( x ) = d∑ i=1 T∑ t=1 ∑ p∈PT , k , i , t vp · ap ( w ; x ) · x ( t ) i , ( 3 ) where the set PT , k , i , t consists of the paths that connects the k-th output at time-step T and the i-th input at time-step t with length T + 1 − t , vp is the value of path p , and ap ( w ; x ) =∏ N H , ( t ) i ∈p I ( nH , ( t ) i ( w ; x ) > 0 ) is the activation status of path p .

Summary: This paper proposes a parameter space, called path space for RNNs with ReLU activation. For the construction of the path space, this paper utilises a reduction graph approach to minimise the difficulty brought by the parameter-sharing scheme in RNNs. Furthermore, the authors propose a Skeleton method for the efficient identification of basis path for RNNs in reduction graph.