Decoupling Hierarchical Recurrent Neural Networks With Locally Computable Losses
Authors: Asier Mujika
Felix Weissenberger
Angelika Steger
Venue:  (2020)

Abstract: Learning long-term dependencies is a key long-standing challenge of recurrent neural networks (RNNs). Hierarchical recurrent neural networks (HRNNs) have been considered a promising approach as long-term dependencies are resolved through shortcuts up and down the hierarchy. Yet, the memory requirements of Truncated Backpropagation Through Time (TBPTT) still prevent training them on very long sequences. In this paper, we empirically show that in (deep) HRNNs, propagating gradients back from higher to lower levels can be replaced by locally computable losses, without harming the learning capability of the network, over a wide range of tasks. This decoupling by local losses reduces the memory requirements of training by a factor exponential in the depth of the hierarchy in comparison to standard TBPTT.

Under review as a conference paper at ICLR 2020
DECOUPLING HIERARCHICAL RECURRENT NEURAL NETWORKS WITH LOCALLY COMPUTABLE LOSSES
Anonymous authors Paper under double-blind review
ABSTRACT
Learning long-term dependencies is key long-standing challenge of recurrent neu- ral networks (RNNs). Hierarchical recurrent neural networks (HRNNs) have been considered promising approach long-term dependencies resolved through shortcuts up and down the hierarchy. Yet, the memory requirements of Truncated Backpropagation Through Time (TBPTT) still prevent training them on very long sequences. In this paper, we empirically show that in (deep) HRNNs, propagating gradients back from higher to lower levels can be replaced by locally computable losses, without harming the learning capability of the network, over a wide range of tasks. This decoupling by local losses reduces the memory requirements of training by a factor exponential in the depth of the hierarchy in comparison to standard TBPTT.
INTRODUCTION
Recurrent neural networks (RNNs) model sequential data by observing one sequence element at time and updating their internal (hidden) state towards being useful for making future predictions. RNNs are theoretically appealing due to their "uring-completeneer (Siegelmann and Sontag ( (1995), and, crucially, have been tremendously successful complex real-world tasks, including machine translation Cho et al. (2014) Sutskever et al. (2014), language modelling Mikolov et al. (2010), and reinforcement learning Mnih et al. (2016).
Still, training RNNs in practice is one of the main open problems in deep learning, as the following issues prevail.
(1) Learning long-tern dependencies is extremely difficult because it requires that the gradients (i.c. the error signal) have to be propagated over many steps, which casily causes them to vanish or explode Hochreiter (1991) Bengio et al. (1994): Hochreiter (1998).
(2) Truncated Backpropagation Through Time (TBPTT) Williams and Peng 1990), the standard training algorithm for RNNs, requires memory that grows linearly in the length of the sequences on which the network is trained. This is because all past hidden states must be stored. Therefore, the memory requirements of training RNNs with large hidden states long sequences become prohibitively large.
(3) In TBPTT, parameters cannot be updated until the full forward and backward passes have been completed. This phenomenon is known as the parameter update lock Jaderberg et al.) (2017). As consequence, the frequency at which parameters can be updated is inversely proportional to the length of the dimeedependencies that can be learned, which makes learning exceedingly slow for long sequences.
The problem of vanishing/exploding gradients has been alleviated by a plethora of approaches ranging from specific RNN architectures Hochreiter and Schmidhuber (1997 ) Cho al. (2014) to
optimization techniques aiming at easing gradient flow Martens and Sutskever (201) 1): Pascanu et al.
gradient problem is hierarchical RNNs (HRNNs) |Schmidhuber (1992); El Hihi and Bengio (996);; Koutnik al. (2014); Sordoni et al. 22015;; Chung et al. (2016). In HRNNs, the network itself is split into a hierarchy of levels, which updated at decreasing frequencies. As higher levels of the hierarchy are updated less frequently, these architectures have short (potentially logarithmic) gradient paths that greatly reduce the vanishing/explodingg gradients issue.
(2033). A candidate for effectively resolving the vanishing/explodiin

---

Under review as a conference paper at ICLR 2020
In this paper, we show that in HRNNs, the lower levels of the hierarchy can be decoupled from the higher levels, in the sense that the gradient flow from higher to lower levels can effectively be replaced by locally computable losses. Also, demonstrate that in consequence, the decoupled HRNNs admit training with memory decreased by a factor exponentially in the depth of the hierarchy compared to HRNNS with standard TBPTT. The local losses stem from decoder networks which are trained to decode past inputs to each level from the hidden state that is sent up the hierarchy, thereby forcing this hidden state to contain all relevant information. We experimentally show that in a diverse set of tasks which rely on long-tern oong-term dependencies and include deep hierarchies, the performance of the decoupled HRNN with local losses is indistinguishable from the standard HRNN.
In summary, we introduce RNN architecture with short gradient paths that can be trained memory- efficiently, thereby addressing issues (1) and (2). In the bigger picture, we believe that our approach of replacing gradient flow in HRNNs by locally computable losses may eventually help to attempt solving issue (3) as well.
2 RELATED WORK
Several techniques have been proposed to deal with the memory issues of TBPTT [hen al. (2016); Gruslys et al. (2016). Specifically, they trade memory for computation by storing only certain hidden states and recomputing the missing ones on demand. This is orthogonal to our ideas and thus, both can potentially be combined.
The memory problem and the update lock have been tackled by online optimization algorithms such as Real Time Recurrent Learning (RTRL) Williams and Zipser| (1989) and its recent approxima- tions Tallec and Ollivier (2017): Mujika et al. [Beziig
[Benzing et al. (2019), 9;; Cooijmans and Martens the parameters can be updated in every step while
(2019). Online algorithms are promising, the memory requirements do not grow with the sequence length. Yet, large computation costs and noise in the approximations make these algorithms impractical far. Another way to deal with the parameter update lock and the memory requirements are Decoupled Neural Interfaces, introduced by by Jaderberg et al. in Daderberg et al. (017). Here, neural network is trained to predict incoming gradients, which are then used instead of the true gradients.
HRNNS have been widely studied over the last decades. One early example by Schmidhuber in SSchiddhuberr (1992) proposes updating the upper hierarchy only if the lower one makes a prediction error. El Hihi and Bengio showed that HRNNs with fixed but different frequencies per level are superior in learning long-term dependencies El Hihi and Bengio (199)). More recently. Koutník et al. introduced the Clockwork RNN Koutnik et al. 20044) here the RNN divided into several modules that are updated at exponentially decreasing frequencies. Many approaches have also been proposed that are explicitly provided with the hierarchical structure of the data (Sordoni et al.) (2015); Ling et al. (2015); Tai et al. (2055), for example character-level language models with word boundary information. Interestingly, Chung et al. [hung et al. (2016) present an architecture where this hierarchical structure is extracted by the network itself. As model utilizes fixed or given hicrarchical structure, yet does not learn it, models that can learn useful hicrarchies may improve the performance of our approach.
Auxiliary losses in RNNs have been used to improve generalization or the length of the sequences that can be learnt. In Schmidhuber (1992), Schmidhuber presented approach, where a RNN should not only predict a target output to solve the task, but also its next input as an auxiliary task. More recently,
Goyal et al. showed that in variational inference setting, training of the latent variables can be eased by an auxiliary loss which forces the latent variables to reconstruct the state of a recurrent network running backwards Goyal al. (2017). Subsequently, Trinh al. demonstrated that introducing auxiliary losses at random anchors in time which aim to reconstruct or predict previous or subsequent input subsequences significantly improve optimization and generalization of LSTMss Hochreiter and Schmidhuberr ( (1997) and helps to learn time dependencies beyond the truncation horizon of TBPIT. The main difference to our approach is that they use auxiliary losses to incentivize an RNN to keep information in memory. This extends the sequences that can be solved by an additive factor through an echo-state network-like effect. On the contrary, we use auxiliary losses to replace gradient paths in hierarchical models. This, together with the right memorization scheme, allows us to discard all hidden states of the lower hierarchical levels when doing TBPTT, which reduces the memory requirements by a multiplicative factor.
2

---

Under review as a conference paper at ICLR 2020
3 METHODOLOGY
In this section we describe our approach in detail. We start by defining a standard HRNN, then explain how cutting its gradient flow saves memory during training, and conclude by introducing an auxiliary loss, that can be computed locally, which prevents performance drop despite the restricted gradient flow (as shown in Section (eloow)
3.1 HIERARCHICAL RECURRENT NEURAL NETWORKS
The basis of our architecture is simple HRNN with two hierarchical levels (deeper HRNNS considered below). We describe it in terms of general RNNs. Such RNN X is simply parameterized and differentiable function fx which maps hidden state and an input h' f (h.
new hidden state
The HRNN consists of the lower RNN (indicated by superscript L) and the upper RNN (superscript U). The lower RNN receives the input and generates the output Ye in addition to the next hidden state h2 at every time-step. Every steps, the upper RNN receives the hidden state of the lower RNN, updates its own hidden state and sends its hidden state to the lower RNN. Then, the hidden state of lower RNN is reset to zeros.
The update rules summarized in Equation (T) for the lower and Equation (2 for the upper RNN.
The unrolled computation graph is depicted in Figure
h [54 (xr,,1))))))))
if mod k = 0,
1' 0)
else.
(1)
1i h 1
mod h: = O,
(2)
else.
While we have outlined the case for two hierarchical levels, this model naturally extends to deeper hierarchies by applying it recursively (see Section 4.4). This leads to levels that are updated exponen- tially less frequently. Moreover, the update frequencies can also be flexible in case the hierarchical structure of the input is explicitly provided (see Section 4.3)
hy
hV
h
h
h
I
20
Ut
F+4k-1
De4k-1
T2+h
1/++
T242k-1
1/4422
Figure Unrolled computation graph HRNN with two hierarchical levels.
3.2 RESTRICTED GRADIENT FLOW
TBPTT, the network first unrolled for I' steps in the forward pass, see Figure [ while the hidden states stored in memory. Thereafter, the gradients of the loss with respect the parameters computed using the stored hidden states in the backward pass, see Figure (a).
To save memory during gradient computation, simply ignore the edges from higher to lower levels in the computation graph the backward pass, see Figure [ Importantly, the resulting gradients are not the true gradients and term them restricted gradients as they computed under restricted

---

Under review as a conference paper at ICLR 2020
gradient flow. We will refer to the restricted gradients with the symbol.Thus, anf time-step when the upper RNN ticks (i.e. t mod k = 0) and it is equal to the true partial derivatives everywhere else. We call the HRNN that is trained using these restricted gradients the gradient restricted HRNN (gr-HRNN),
if
by
hg
1+
b
1 11
(a) Gradient flow in the unrolled HRNN
hy
hy
h
1i+k
M1+22-1
(b) Gradient flow in the unrolled gr-HRNN
Figure Gradient flow the unrestricted and gradient restricted HRNN.
Before we address the issue of compensating for the restricted gradients in the next section, we first analyze how much memory is needed to compute the restricted gradients.
In the following, let t be time-step when the upper RNN ticks (i.e. t mod k 0). A direct consequence of the restricted gradients is that Σ âc, O, where L, is the loss at time-step and we consider the network unrolled for time-steps I..... T. Therefore, right before the upper åc; where 8 are the parameters of the network, can be computed for the previous
anf
RNN updates, 00 time-steps using just the previous i hidden states, h h....... M of the lower RNN. We further need these hidden states to compute Σ dc-i ong
However, we will not need the hidden states
h/ k h/11 for any other step of the backward pass and thus, do not need to keep them in memory.
Therefore, the restricted gradients be computed using memory proportional A for the hidden states h , of the lower RNN and proportional to 2T/K for both the 7/k hidden states of the upper RNN and the T/k accumulated restricted gradients of the loss with respect to the hidden states of the upper RNN. Standard TBPTT requires memory proportional to T in order to compute the true gradients. Here, we showed that the restricted gradients can be computed using memory proportional to k 27) /k; thus improving above argument can be applied recursively. which yields memory requirements of (1 1)k for the hidden states of the lower RNNs and 2T [-1 for the uppermost RNN.
factor of 2/k. For (deep) HRNNs with / levels, the
3.3 LOCALLY COMPUTABLE AUXILIARY LOSSES
In the gr-HRNN, the lower RNN is not incentivized to provide useful representation as input for the upper RNN. However,
clear that if this hidden state which sent upwards contains all information about the last & inputs, then the upper RNN has access to all information about the input, and thus the gr-HRNN should be able learn as effectively as the HRNN without gradient restrictions.
Hence, we introduce an auxiliary loss term to force this hidden state to contain all information about the last & inputs. This auxiliary loss comes from simple feed-forward decoder network, which given this hidden state and a randomly sampled index E {1...., k} (in one-hot encoding) must output the i-th previous input. Then, the combined loss of the HRNN the sum of the loss defined by the task
4

---

Under review as a conference paper at ICLR 2020
and the auxiliary decoder loss multiplied by hyper- B. For hierarchies with / levels, add - decoder networks with respectively weighted losses.
Crucially, the auxiliary loss be computed locally in time. Thus, the memory requirements for training increase only by an additive term in the size of the decoder network.
EXPERIMENTS
In this section, we experimentally test replacing the gradient flow from higher to lower levels by adding the local auxiliary losses. Therefore, we evaluate the gr-HRNN with auxiliary loss (simply termed 'ours' from here on) on four different tasks: Copy task, pixel MNIST classification, permuted pixel MNIST classification, and character- language modeling. All these tasks require that the model effectively learns long-term dependencies to achieve good performance. Additionally, to our to fully understand the importance and
model, we also evaluate several ablationn/augmmenttinns effects of its individual components. These other models are:
This model is identical to ours, except that all gradients from higher to lower levels are propagated. The weight of the auxiliary loss is hyper- aypee--aramettr which is using grid-search (as the auxiliary loss may improve the performance, we keep it here to make fair comparison with our model). While this model has much larger memory cost for training, provides an upper bound on the performance we can expect from our model.
HRNN:
model, except that the weight of the auxiliary loss is set to zero. Thereby, it permits studying if using the auxiliary loss function (and the corresponding decoder network) really necessary when the gradients are stopped.
gr-HRNN:
This model is equivalent
mr-HRRNN
This model is identical to the HRNN, except that it is trained using only as much memory as our model requires for training. That is, is unrolled for much smaller number of steps than the other models. In consequence, it admits fair performance comparison (in terms of memory) our model.
If not mentioned otherwise below, the parameters are follows. All RNNs are LSTMs with hidden state of 256 units. The network for the auxiliary loss is a two layer network with 256 units, a ReLU Glorot et al. (201T) non-linearity and uses either the cross-entropy loss for discrete values or the mean squared loss for continuous ones. For each model we pick the optimal | from
[10- 3, 10 1011 0, 10 10 , except for the gr-HRNN for which / 0. The models
trained using a batch size of 100 and the Adam optimizer Kingma and Ba (014) with the default
Tensorflow Abadi al. (2015) parameters: learning rate of 0.001, B1 0.9 and .32 0.999.
4.1 COPY TASK
Table 1: Results for copy task (3 repetitions). Memory refers the number vectors of the same size as the hidden state need be stored for each algorithm and batch element.
Model
Gradients
Aux. loss
Memory
Iynax (longest solved sequences)
ours
X
50
108 + 0
gr-HRNN
X
50
21 + 13
HRNN
200
107.7 0.47
mr-HRNN
50
48.3 + 0.94
In the copy task, the network is presented with a binary string and should afterwards output the presented string in the same order (an example for a sequence of length 5 is: input 01101* with target output ***** 0 1110)). We refer to the maximum sequence length that model can solve (i.e. achieves loss less than 0.15 bits/charr as Lmux- Table summarizes the results for a sequence of length 100.
5

---

Under review as a conference paper at ICLR 2020
The copy task requires exact storage of the input sequence many steps. Since the length of the sequence and therefore the dependencies are a controllable parameter in this task, permits to explicitly assess how long dependencies model can capture.
Both our model and the model using all gradients (HRNN) achieve very similar performance, which is limited by the number of steps for which the network is unrolled in training. Moreover, the auxiliary loss is necessary, as the model without it (gr-HRNN) performs poorly. Moreover, our model drastically outperforms the model with all gradients, when given the same memory budget, as the memory restricted HRNN (mr-HRNN also performs poorly. Notably, our model is remarkably robust the choice of the auxiliary loss weight 3. for all values except it learns sequences of length at least 100. Finally, the HRNN actually uses the capacity of the decoder network, as the grid-search yields a nonzero B.
All models, except the mr-HRNN, are unrolled for T = 200 steps and the upper RNN ticks every 10 steps of the lower one. To equate memory budgets, the mr-HRNN unrolled for 27/k k = 50 steps. Hence, for sequence lengths at most 100, gradients are propagated through the whole sequence. The model parameters are only updated once per batch.
4.2 (PERMUTED) PIXEL MNIST CLASSIFICATION
Table 2: Results for the pixel MNIST tasks (3 repetitions). Memory refers to the number of vectors of the same size as the hidden state that need to be stored for each algorithm and batch element.
Model
Gradients
Aux. loss
Memory
Accuracy
Accuracy (permuted)
ours
X
166
0.9886
0.0002
0.9680 + 0.0008
gr-HRNN
X
166
0.2109 + 0.0743
0.9368 + 0.0049
HRNN
V
784
0.9885
0.0003
0.9681 + 0.0008
mr-HRNN
V
166
0.8939 = 0.0041
0.9553 = 0.0015
In the pixel MNIST classification task, the pixels of MNIST digits presented sequentially. After observing the whole sequence the network must predict the class label of the presented digit. The permuted pixel MNIST task is exactly the same, but the pixels are presented in fixed random order. the accuracy of the different models on the two tasks.
Both pixel MNIST classification tasks require learning long-term dependencies, especially when using the default permutation, as the most informative pixels are around the center of the image (i.c. in the middle of the input sequence) and the class prediction is only made the end. Additionally, unlike in the copy task, the input has to be processed in more complex manner in order to make class prediction.
The results on both tasks are in line with the copy task. Our model performs par with the model using the true gradient (HRNN), Again, the auxiliary loss is necessary. the accuracy of the gr- HRNN is significantly worse. Given the same memory budget, our model outperforms the HRNN with all gradients.
All models are unrolled for T = 784 steps (the number of pixels of an MNIST image) except for the memory restricted model which unrolled for 166 steps. The upper RNN ticks every k = 10 steps of the lower one. The same permutation is used for all runs of the permuted task.
4.3 CHARACTER- LANGUAGE MODELING
In the Character-leved language modeling task, text is presented character by character and each step, the network must predict the next character. The results for this task on the Penn TreeBank corpus Marcus and Marcinkiewicz (1993) are summarized in Table 3
In contrast to the previous tasks, character language modeling contains a complex mix of both short- and long-term dependencies, where short-term dependencies typically dominate long-term

---

Under review as a conference paper at ICLR 2020
Table 3: Results for the character-level language modeling task on the Penn TreeBank corpus (3 repetitions).
Model
Gradients
Aux. loss
Validation bits/char
Test bits/char
ours
X
v
1.4519 + 0.0014
1.4027 = 0.0003
gr-HRNN
X
X
1.4545 + 0.0016
1.4074 0.0011
HRNN
V
V
1.4480 + 0.0011
1.3974 + 0.0021
dependencies. Thus, one may expect that replacing the true gradients by the local loss is the most harmful here.
The performance of all 3 models is very close in this task, even for the gr-HRNN, which has neither gradients nor the auxiliary loss to capture long-term dependencies. We believe that this is due to the dominance short-term dependencies in character-level language modeling. It has been widely reported that the length of the unrolling, and thus long-term dependencies, have minimal effect on the final performance of character-leve language models Daderberg et al., (2017). particularly for small RNNs, as in experiments. Still, we observe the best performance when using the true gradients (HRNN), a slight degradation when replacing gradients by the auxiliary loss (our), which slightly improves not using the auxiliary loss (gr-HRNN).
Here, we explicitly provide the hierarchical structure the data to the model by updating the upper RNN once per word, while the lower RNN ticks once per character. As discussed in Section 2 there are models that can extract the hierarchical structure, which is a separate goal from ours.
We refrain from comparing with memory restricted model and displaying memory budgets because the dynamic memory requirements depending on the input sequence prohibit a fair comparison. However, stated earlier, the unroll length usually has minimal effect on the performance of character-level language modeling. We unroll the models for 50 characters and use an upper RNN with 512 units to deal with the extra complexity of the task.
4.4 DEEPER HIERARCHIES
In deeper hierarchies, the output modalities of the individual decoder networks are different: whereas the decoder network the lowest level has to predict elements of the input sequence (e.g. bits in the copy task), the decoder networks in higher levels have predict hidden states of the lower level's RNNs. Here, we confirm that our approach generalizes to deeper hierarchies.
In particular, use the copy task with sequence length 500 (and truncation horizon 1000) test this. We consider a HRNN with 3 levels, where the lowest RNN is updated in every step, the middle RNN every 5 steps and the upper RNN every 25 steps, where no gradients from higher lower RNNs are propagated. We compare using the auxiliary loss only on the lowest level with using it in the lowest and the middle layer. Thereby, we check whether applying the auxiliary loss in the middle RNN (i.e. over hidden states rather than raw inputs) is necessary to solving the task.
The RNNs have 64, 256 and 1024 units (from lower to higher levels). The for the lowest level is 0.1 and 1 for the middle level. All other experimental details are kept as in Section 4.1 We run several repetitions which are shown in Figure Without the auxiliary loss in the middle layer, the network cannot solve the task (in fact, not even close as performance is close to chance level). However, when the auxiliary loss added also to the middle layer, the model can solve the task perfectly on every run.
5 CONCLUSION
In this paper, we have shown that in hierarchical RNNs the gradient flow from higher to lower levels can be effectively replaced by locally computable losses. This allows memory savings up to an exponential factor in the depth of the hierarchy. In particular, first explained how propagating gradients from higher to lower levels permits these memory savings. Then, we introduced auxiliary losses that encourage information flow up the hierarchy. Finally, demonstrated experimentally
7

---

Under review as a conference paper at ICLR 2020
6.8
0.0
6.0
0.2
20000
40000
60000
80000
100000
Learning steps
Figure 3: Results copy task with sequence length of 500. Three different runs of deep HRNNs with the auxiliary loss in both levels (red) and with the auxiliary loss only the lower level (green).
that the memory-efficien HRNNs with our auxiliary loss perform on par with the memory-heavy HRNNs and strongly outperform HRNNs given the same memory budget on wide range of tasks, including deeper hierarchies.
High capacity RNNs, like Differentiable Plasticity Miconi et al. 2018). or Neural Turing Ma- chines Graves al. (2014) have been shown be useful and even achieve tfate-of-the-an in many tasks. However, due to the memory cost of TBPTT, training such models is often impractical for long sequences. We think that combining these models with our techniques in future work could open the possibility for using high capacity RNNs for tasks involving long-term dependencies that have been out of reach so far.
Still, the problem of the parameter update lock remains. While this is the most under-explored of the three big problems when training RNNs (vanishing/explodin being the other two), resolving it is just important in order to be able learn long-term depen- dencies. We believe that the techniques laid out in this work (i.c. replacing gradients in HRNNs by locally computable losses) can be stepping stone towards solving the parameter update lock. We leave this for future work.
gradients and memory requirements
REFERENCES
M. Abadi, A. Agarwal, Barham, E. Brevdo, Chen, C. Citro, G. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL htttps:/wwwwtenssttt Software available from tensorflow.org.
org/
Y. Bengio, P. Simard, P. Frasconi, et al. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Benzing, M. M. Gauy, Mujika, A. Martinsson, and Steger. Optimal kronecker-sune sum approxi- mation of real time recurrent learning. arXiv preprint arXiv: 2019.
T. Chen, B. Xu, C. Zhang, and C. Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:I604.06111 2016.
K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.
Chung, S. Ahn, and Y. Bengio. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv: 1609. 01 00404, 2016.
8

---

Under review as a conference paper at ICLR 2020
T. Cooijmans and J. Martens. On the variance of unbiased online recurrent optimization. arXiv preprint (eX.v:1902.024400 2019.
S. EI Hihi and Y. Bengio. Hierarchical recurrent neural networks for long-term dependencies. In Advances in neural information processing systems, pages 493-499, 1996.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statisties, pages 315- 113-,23, 2011.
A. G. A. P. Goyal, A. Sordoni, M.-A. Côté, N. R. Ke, and Bengio. Z-forcing: Training stochastic recurrent networks. In Advances in neural information processing systems, pages 6713 6723, 2017.
A. Graves, G. Wayne, and I. Danihelka. Neural turing machines, arXiv preprint arXiv: 11X0::441055401 2014.
A. Gruslys, R. Munos, I. Danihelka, M. Lanctot, and A. Graves. Memory-efficient backpropagation through time. Advances in Neural Information Processing Systems, pages 4125-4133, 2016.
S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universität Münnhen, 91(1), 1991.
S. Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02): 107-116, 1998.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 985):1335-1780 1997.
M. Jaderberg, W. M. Czarnecki, S. Osindero, O. Vinyals, A. Graves, D. Silver, and K. Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1627-1635. JMLR. org, 2017.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 142.6690, 2014.
J. Koutnik, K. Greff, F. Gomez, and J. Schmidhuber. A clockwork mn. In International Conference on Machine Learning, pages 1863-1871, 2014.
W. Ling, I. Trancoso, C. Dyer, and A. W. Black. Character-based neural machine translation. arXiv preprint arXiv:151 1.04566, 2015.
M. P. Marcus and M. A. Marcinkiewicz. Building large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2), 1993.
Martens and I. Sutskever. Learning recurrent neural networks with hessian-free optimization. In Proceedings of the 28th International Conference Machine Learning (ICML-II), pages 1033-1040. Citeseer, 2011.
T. Miconi, J. Clune, and K. O. Stanley. Differentiable plasticity: training plastic neural networks with backpropagation. arXiv preprint arXiv: 1004.88444..444 2018.
Mikolov, M. Karafíát, L. Burget, Čemookký, and Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech commmication association, 2010.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937, 2016.
A. Mujika, F. Meier, and A. Steger. Approximating real-time recurrent learning with random kronecker factors. In Advances in Neural Information Processing Systems, pages 6594-6603. 2018.

---

Under review as a conference paper at ICLR 2020
R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318, 2013.
J. Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242, 1992.
H. Siegelmann and E. Sontag. the computational power of neural nets, Journal of computer and system sciences,
1995.
A. Sordoni, Y. Bengio, H. Vahabi, C. Lioma, J. Grue Simonsen, and J.-Y. Nic. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the 24th ACM International on Conference Information and Knowledge Management, pages 553-562. ACM, 2015.
I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112, 2014.
K. Tai, R. Socher, and C. D. Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprini arXiv: urXiv:/503..0075. 2015.
C. Tallec and Y. Ollivier. Unbiased online recurrent optimization. arXiv preprint arXiv:1702.050043 2017.
R. J. Williams and J. Peng. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural computation, 2(4):490-501, 1990.
R. Williams and D. Zipser. learning algorithm for continually running fully recurrent neural networks. Neural computation, (2):27702800, 1989.
10


Reviews:
[{"review_id": "SJxfjMU9YB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed work investigates the problem of learning hierarchy in RNNs. Authors note that different layers of the hierarchy are trained in \"sync\". The proposed paper suggests to decouple the different layers of hierarchy using auxiliary losses.  The form of auxiliary losses used in the paper are of the form of local losses, where there is a decoder, which is used to decode past inputs to each level from the hidden state that is sent up the hierarchy, therebyforcing this hidden state to contain all relevant information. \n\nClarity of the paper: The paper is clearly written.\n\nMethod: The proposed method  ignores the gradients from higher to lower levelsin the backward pass,  (because of this, the authors can also save some memory). In order to compensate for the lost gradients, authors propose to use local losses, and we introduce an auxiliary loss term to force this hidden state to contain all information aboutthe last k inputs. The authors note that the hidden state from the lower level (to the higher level) should contain the summary of the past, and hence use a decoder network (which is simply parameterized) as a feedforward network which is used to decoder a \"past\" hidden state. \n\nRelated Work Section: The related work section is nicely written. The authors have covered mostly everything. These 3 papers may still be relevant. (a), (b), (c).   (b) could be relevant for mitigating the parameter update lock problem as mentioned by authors in the introduction of the paper. (c) is also relevant as authors in (c) also consider using auxiliary losses for learning long term dependencies. \n(a) SkipRNN: https://arxiv.org/abs/1708.06834(b) Sparse Attentive Backtracking: http://papers.nips.cc/paper/7991-sparse-attentive-backtracking-temporal-credit-assignment-through-reminding\n(c)  Learning long term dependencies in RNNs using auxiliary losses https://arxiv.org/abs/1803.00144\nExperiment Section: In order to validate the proposed method, authors evaluate it on copying task, pixel MNIST classification, permutedpixel MNIST classification, and character-level language modeling. \na) Copying results show that the decoder network are essential to achieve decent results. This task though does not show the strength of the proposed method though as baseline also solves the problem completely. It might be interesting to actually scale the \"gap\" time in copying time step to something larger like T = 1000 or something.\nb) PIXEL MNIST classification: Authors use the pixel by pixel classification task to test the proposed method. Here, the proposed method performs comparable to the hierarchical RNN (but without using too much memory). \nc) Character level modelling: Authors demonstrate the performance of the proposed method on language modelling task (PTB). These results are particularly not interesting, as the performance gain is very marginal. Also, may be using other language modelling datasets like Wikitest103 or Text8 might be more useful.  As for the results, even unregularized LSTM performs better than the baseline in this paper. (For reference, see https://arxiv.org/abs/1606.01305) \n\nWhat authors can do to improve paper:\n- The problem considered in the proposed paper is very interesting to me. Though, the results are not (yet) convincing. It might be interesting to think about a task, where there are really long term dependencies like reading CIFAR10 digit pixel by pixel and then doing classification, where the authors can actually show the promise of the proposed method. \n- It might also be interesting to know how are the original training cost objective is weighed against the auxiliary loss. Have authors tried any search over what kind of auxiliary loss performs well ? ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HyeYxYAs_H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper introduces a hierarchical RNN architecture that could be trained more (memory) efficiently. The difference in the architecture seems to be an auxiliary loss that decodes k step inputs and some perturbation of TBPTT.\n\nComments on the paper\n\n1. The paper seems to be have been written in a rush. The language could be improved, the format is not always consistent and in general the paper could be much better written. There are quite some typos as well in the paper, for example , Trinh et al.  is not a proper citation.\n\n2. The authors mentioned that TBPTT is not memory efficient, this is not very clear to me, as it only needs to keep the number of truncation steps that it backprops through and hence much more memory efficient compared to full BPTT.\n\n3. It is not clear to me what is the benefit of gr-HMRNN. It is not clear why cutting of the gradients from the higher level to the lower level would help.\n\n4. It is surprising to me that HMRNN could only solve the copy task upto a length of 108. \n\n5. I would also suggest another copy task from Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997.\n\nIn general, the paper seems to have been written in a rush. I would recommend the papers to be revised.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJge242NjH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Claim: Backpropagation of gradients from a higher to lower level in a HRNN can be removed and replaced with auxiliary losses predicting input tokens at the lower level without affecting performance. \n\nSignificance: The significance of the claim hinges on whether HRNNs are more effective than other methods designed to help RNNs capture long-term dependencies (e.g. stacking RNNs or using different architectures). I think the authors could make a more substantive argument why this would be the case in the introduction, but they do a nice job of situating their work in the context of the present literature.\n\nNovelty: The proposed method is not very original, since augmenting RNNs with auxiliary losses in order to better capture long-term dependencies has been used in many previous papers. The authors mention some of these papers in the related work section.\n\nClarity: The paper's description of the proposed method is well-written. Some parts of the experiment section could be made clearer. \n--  I encourage the authors to invent a new acronym to refer to \"our model\" (perhaps aux-HRNN?). In the description of the mr-HRNN (pg. 5), I find the sentence \"trained using only as much memory as our model requires for training\" confusing.  I initially thought our model referred to the mr-HRNN in the setence.\n-- Training settings (e.g. the number of ticks of the upper RNN) should be described at the beginning of each section. \n-- A seeming contradiction is made when discussing the results in 4.3. First, it said that because short term dependencies dominate long term dependencies it is expected that the proposed method will suffer greatly (pg. 6, bottom). In the next paragraph, it is claimed that all three models perform similarly due to the same reason. Which is it?\n\nSupporting evidence: The claim is empirical and the supporting evidence is experimental. As such, I find the comprehensiveness of the experiments wanting. There are several ways the experiments could be improved. \n-- Results for each \\beta value should be included, to see how placing increasing significance on the auxiliary loss impacts the results.\n-- Include all relevant details necessary to reproduce the results, such as the length of training or stopping criterion used. \n-- Additional results when varying the number of ticks.\n-- More results with deeper hierarchies, since the ability to capture salient information at different levels of coarseness is a key selling point of HRNNs. \n-- Results on larger scale tasks besides character level language modelling on Penn TreeBank.\n\nOther comments:\n-- In the intro, I think some mention of parallel architectures such as transformers or convolutional architectures is warranted here, since parallelizability of training is a significant reason why these architectures are becoming preferred over RNNs.\n-- Citations are mishandled throughout the paper. Citations should be enclosed in parentheses unless used as a subject in the sentence (e.g. \"Sordoni et al. make the case that...\"). There is no need to refer to a citation twice in a sentence, like you do in \"More recently, Koutnik et al. introduced the Clockwork RNN Koutnik et al. (2014)...\"\n-- I don't understand why the permuted accuracy of the gr-HRNN is so much higher than the non-permuted accuracy. One possible explanation is that the important pixels ended up at the end in each of the three trials, hence the gr-HRNN did not have to remember much information from the past. This should be addressed in the paper. \n-- I would welcome some theoretical analysis as to why replacing the gradient path with this particular auxiliary loss does not impact results. I also think some discussion of what this means HRNNs are actually doing might be nice as well.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers gave this paper a score of 1.
The AC recommends rejection.
Decision: Reject