Decoupling Hierarchical Recurrent Neural Networks With Locally Computable Losses

1 INTRODUCTION . Recurrent neural networks ( RNNs ) model sequential data by observing one sequence element at a time and updating their internal ( hidden ) state towards being useful for making future predictions . RNNs are theoretically appealing due to their Turing-completeness Siegelmann and Sontag ( 1995 ) , and , crucially , have been tremendously successful in complex real-world tasks , including machine translation Cho et al . ( 2014 ) ; Sutskever et al . ( 2014 ) , language modelling Mikolov et al . ( 2010 ) , and reinforcement learning Mnih et al . ( 2016 ) . Still , training RNNs in practice is one of the main open problems in deep learning , as the following issues prevail . ( 1 ) Learning long-term dependencies is extremely difficult because it requires that the gradients ( i.e . the error signal ) have to be propagated over many steps , which easily causes them to vanish or explode Hochreiter ( 1991 ) ; Bengio et al . ( 1994 ) ; Hochreiter ( 1998 ) . ( 2 ) Truncated Backpropagation Through Time ( TBPTT ) Williams and Peng ( 1990 ) , the standard training algorithm for RNNs , requires memory that grows linearly in the length of the sequences on which the network is trained . This is because all past hidden states must be stored . Therefore , the memory requirements of training RNNs with large hidden states on long sequences become prohibitively large . ( 3 ) In TBPTT , parameters can not be updated until the full forward and backward passes have been completed . This phenomenon is known as the parameter update lock Jaderberg et al . ( 2017 ) . As a consequence , the frequency at which parameters can be updated is inversely proportional to the length of the time-dependencies that can be learned , which makes learning exceedingly slow for long sequences . The problem of vanishing/exploding gradients has been alleviated by a plethora of approaches ranging from specific RNN architectures Hochreiter and Schmidhuber ( 1997 ) ; Cho et al . ( 2014 ) to optimization techniques aiming at easing gradient flow Martens and Sutskever ( 2011 ) ; Pascanu et al . ( 2013 ) . A candidate for effectively resolving the vanishing/exploding gradient problem is hierarchical RNNs ( HRNNs ) Schmidhuber ( 1992 ) ; El Hihi and Bengio ( 1996 ) ; Koutnik et al . ( 2014 ) ; Sordoni et al . ( 2015 ) ; Chung et al . ( 2016 ) . In HRNNs , the network itself is split into a hierarchy of levels , which are updated at decreasing frequencies . As higher levels of the hierarchy are updated less frequently , these architectures have short ( potentially logarithmic ) gradient paths that greatly reduce the vanishing/exploding gradients issue . In this paper , we show that in HRNNs , the lower levels of the hierarchy can be decoupled from the higher levels , in the sense that the gradient flow from higher to lower levels can effectively be replaced by locally computable losses . Also , we demonstrate that in consequence , the decoupled HRNNs admit training with memory decreased by a factor exponentially in the depth of the hierarchy compared to HRNNs with standard TBPTT . The local losses stem from decoder networks which are trained to decode past inputs to each level from the hidden state that is sent up the hierarchy , thereby forcing this hidden state to contain all relevant information . We experimentally show that in a diverse set of tasks which rely on long-term dependencies and include deep hierarchies , the performance of the decoupled HRNN with local losses is indistinguishable from the standard HRNN . In summary , we introduce a RNN architecture with short gradient paths that can be trained memoryefficiently , thereby addressing issues ( 1 ) and ( 2 ) . In the bigger picture , we believe that our approach of replacing gradient flow in HRNNs by locally computable losses may eventually help to attempt solving issue ( 3 ) as well . 2 RELATED WORK . Several techniques have been proposed to deal with the memory issues of TBPTT Chen et al . ( 2016 ) ; Gruslys et al . ( 2016 ) . Specifically , they trade memory for computation by storing only certain hidden states and recomputing the missing ones on demand . This is orthogonal to our ideas and thus , both can potentially be combined . The memory problem and the update lock have been tackled by online optimization algorithms such as Real Time Recurrent Learning ( RTRL ) Williams and Zipser ( 1989 ) and its recent approximations Tallec and Ollivier ( 2017 ) ; Mujika et al . ( 2018 ) ; Benzing et al . ( 2019 ) ; Cooijmans and Martens ( 2019 ) . Online algorithms are promising , as the parameters can be updated in every step while the memory requirements do not grow with the sequence length . Yet , large computation costs and noise in the approximations make these algorithms impractical so far . Another way to deal with the parameter update lock and the memory requirements are Decoupled Neural Interfaces , introduced by by Jaderberg et al . in Jaderberg et al . ( 2017 ) . Here , a neural network is trained to predict incoming gradients , which are then used instead of the true gradients . HRNNs have been widely studied over the last decades . One early example by Schmidhuber in Schmidhuber ( 1992 ) proposes updating the upper hierarchy only if the lower one makes a prediction error . El Hihi and Bengio showed that HRNNs with fixed but different frequencies per level are superior in learning long-term dependencies El Hihi and Bengio ( 1996 ) . More recently , Koutník et al . introduced the Clockwork RNN Koutnik et al . ( 2014 ) , here the RNN is divided into several modules that are updated at exponentially decreasing frequencies . Many approaches have also been proposed that are explicitly provided with the hierarchical structure of the data Sordoni et al . ( 2015 ) ; Ling et al . ( 2015 ) ; Tai et al . ( 2015 ) , for example character-level language models with word boundary information . Interestingly , Chung et al . Chung et al . ( 2016 ) present an architecture where this hierarchical structure is extracted by the network itself . As our model utilizes fixed or given hierarchical structure , yet does not learn it , models that can learn useful hierarchies may improve the performance of our approach . Auxiliary losses in RNNs have been used to improve generalization or the length of the sequences that can be learnt . In Schmidhuber ( 1992 ) , Schmidhuber presented an approach , where a RNN should not only predict a target output to solve the task , but also its next input as an auxiliary task . More recently , Goyal et al . showed that in a variational inference setting , training of the latent variables can be eased by an auxiliary loss which forces the latent variables to reconstruct the state of a recurrent network running backwards Goyal et al . ( 2017 ) . Subsequently , Trinh et al . demonstrated that introducing auxiliary losses at random anchors in time which aim to reconstruct or predict previous or subsequent input subsequences significantly improve optimization and generalization of LSTMs Hochreiter and Schmidhuber ( 1997 ) and helps to learn time dependencies beyond the truncation horizon of TBPTT . The main difference to our approach is that they use auxiliary losses to incentivize an RNN to keep information in memory . This extends the sequences that can be solved by an additive factor through an echo-state network-like effect . On the contrary , we use auxiliary losses to replace gradient paths in hierarchical models . This , together with the right memorization scheme , allows us to discard all hidden states of the lower hierarchical levels when doing TBPTT , which reduces the memory requirements by a multiplicative factor . 3 METHODOLOGY . In this section we describe our approach in detail . We start by defining a standard HRNN , then explain how cutting its gradient flow saves memory during training , and conclude by introducing an auxiliary loss , that can be computed locally , which prevents a performance drop despite the restricted gradient flow ( as shown in Section 4 below ) . 3.1 HIERARCHICAL RECURRENT NEURAL NETWORKS . The basis of our architecture is a simple HRNN with two hierarchical levels ( deeper HRNNs are considered below ) . We describe it in terms of general RNNs . Such a RNNX is simply a parameterized and differentiable function fX which maps a hidden state h and an input x to a new hidden state h′ = fX ( h , x ) . The HRNN consists of the lower RNN ( indicated by superscript L ) and the upper RNN ( superscript U ) . The lower RNN receives the input xt and generates the output yt in addition to the next hidden state ht at every time-step . Every k steps , the upper RNN receives the hidden state of the lower RNN , updates its own hidden state and sends its hidden state to the lower RNN . Then , the hidden state of the lower RNN is reset to all zeros . The update rules are summarized in Equation ( 1 ) for the lower and Equation ( 2 ) for the upper RNN . The unrolled computation graph is depicted in Figure 1. hLt = { fL ( 0 , [ xt , h U t ] ) if t mod k = 0 , fL ( hLt−1 , [ xt , 0 ] ) else . ( 1 ) hUt = { fU ( hUt−1 , h L t−1 ) if t mod k = 0 , hUt−1 else . ( 2 ) While we have outlined the case for two hierarchical levels , this model naturally extends to deeper hierarchies by applying it recursively ( see Section 4.4 ) . This leads to levels that are updated exponentially less frequently . Moreover , the update frequencies can also be flexible in case the hierarchical structure of the input is explicitly provided ( see Section 4.3 ) . 3.2 RESTRICTED GRADIENT FLOW . In TBPTT , the network is first unrolled for T steps in the forward pass , see Figure 1 , while the hidden states are stored in memory . Thereafter , the gradients of the loss with respect to the parameters are computed using the stored hidden states in the backward pass , see Figure 2 ( a ) . To save memory during gradient computation , we simply ignore the edges from higher to lower levels in the computation graph of the backward pass , see Figure 2 . Importantly , the resulting gradients are not the true gradients and we term them restricted gradients as they are computed under restricted gradient flow . We will refer to the restricted gradients with the ∂̃ symbol.Thus , ∂̃h U t ∂̃hLt−1 = 0 if t is a time-step when the upper RNN ticks ( i.e . t mod k = 0 ) and it is equal to the true partial derivatives everywhere else . We call the HRNN that is trained using these restricted gradients the gradient restricted HRNN ( gr-HRNN ) . Before we address the issue of compensating for the restricted gradients in the next section , we first analyze how much memory is needed to compute the restricted gradients . In the following , let t be a time-step when the upper RNN ticks ( i.e . t mod k = 0 ) . A direct consequence of the restricted gradients is that ∑T i=t ∂̃Li ∂̃hLt−1 = 0 , where Li is the loss at time-step i and we consider the network unrolled for time-steps 1 , . . . , T . Therefore , right before the upper RNN updates , ∂̃Li ∂̃θL , where θ are the parameters of the network , can be computed for the previous k time-steps using just the previous k hidden states , hLt−k , . . . , h L t−1 , of the lower RNN . We further need these hidden states to compute ∑k i=1 ∂̃Lt−i ∂̃hUt−k . However , we will not need the hidden states hLt−k , . . . , h L t−1 for any other step of the backward pass and thus , we do not need to keep them in memory . Therefore , the restricted gradients can be computed using memory proportional to k for the hidden states hLt−k , . . . , h L t−1 of the lower RNN and proportional to 2T/k for both the T/k hidden states of the upper RNN and the T/k accumulated restricted gradients of the loss with respect to the hidden states of the upper RNN . Standard TBPTT requires memory proportional to T in order to compute the true gradients . Here , we showed that the restricted gradients can be computed using memory proportional to k + 2T/k , thus improving by a factor of 2/k . For ( deep ) HRNNs with l levels , the above argument can be applied recursively , which yields memory requirements of ( l − 1 ) k for the hidden states of the lower RNNs and 2T/kl−1 for the uppermost RNN .

Summary: The proposed work investigates the problem of learning hierarchy in RNNs. Authors note that different layers of the hierarchy are trained in "sync". The proposed paper suggests to decouple the different layers of hierarchy using auxiliary losses.  The form of auxiliary losses used in the paper are of the form of local losses, where there is a decoder, which is used to decode past inputs to each level from the hidden state that is sent up the hierarchy, therebyforcing this hidden state to contain all relevant information.