Title: Hierarchical Multiscale Recurrent Neural Networks
Authors: Junyoung Chung
Sungjin Ahn
Yoshua Bengio
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: Learning both hierarchical and temporal representation has been among the long- standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural network, that can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that the proposed model can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence generation.

TL;DR: Propose a recurrent neural network architecture that can discover the underlying hierarchical structure in the temporal data.

Review discussion:
How much computational savings did the described architecture yield, in particular as the number of LSTM layers increased?
By design, our architecture ensures that a layer performs less computations than its previous (lower) layer.
-More specifically, in our model, each time step consists of two sub-phases of computation: (1) operation selection based on the hidden states of its previous layer at the same time or the previous time of the same layer and (2) the execution of the chosen operation (e.g., UPDATE, COPY, FLUSH).
-Our architecture performs (2) only when (1) says it is necessary.
For example, we do not go to (2) if (1) decides it is time to do COPY. Computation of (1) is usually much lighter than execution of the UPDATE or FLUSH operation.
-Traditional RNN architectures, however, perform the UPDATE operation at every time step.
-Also, by design of our model, an UPDATE at a layer can happen only after *at least* one UPDATE is performed at its previous layer.
Thus, the number of UPDATE of a higher layer is always lower than a lower layer.
-This computational saving can easily be obtained when batch size = 1 (e.g., at test time).
However, implementing this in mini-batch training may require more engineering because each sequence in the batch may have different sequences of operations.
Thus, we did not implemented this in our mini-batch training. Note that obtaining speed-up is a secondary advantage of the proposed method, the primary contribution being obtaining the hierarchical latent structure.
It's better to mention how many characters in Penn Treebank similar to the descriptions of Text8 and Hutter Prize Wikipedia tasks.
What kind of toolkit are you using to implement it?
Some toolkit does not allow us to touch a time-directional forloop easily, and it would be better information to describe how you implement things to which toolkit.
-Penn Treebank dataset consists of 5.8M characters.
-We used Theano to implement our models.
This paper proposes a novel variant of recurrent networks that is able to learn the hierarchy of information in sequential data (e.g., character->word). Their approach does not require boundary information to segment the sequence in meaningful groups (like in Chung et al., 2016).
Their model is organized as a set of layers that aim at capturing the information form different “level of abstraction”. The lowest level activate the upper one and decide when to update it based on a controller (or state cell, called c). A key feature of their model is that c is a discrete variable, allowing potentially fast inference time.
However, this makes their model more challenging to learn, leading to the use of the straight-through estimator by Hinton, 2012.
The experiment section is thorough and their model obtain competitive performance on several challenging tasks. The qualitative results show also that their model can capture natural boundaries.
Overall this paper presents a strong and novel model with promising experimental results.
On a minor note, I have few remarks/complaints about the writing and the related work:
- In the introduction:
“One of the key principles of learning in deep neural networks as well as in the human brain” : please provide evidence for the “human brain” part of this claim.
“For modelling temporal data, the recent resurgence of recurrent neural networks (RNN) has led to remarkable advances” I believe you re missing Mikolov et al. 2010 in the references.
“in spite of the fact that hierarchical multiscale structures naturally exist in many temporal data”: missing reference to Lin et al., 1996
- in the related work:
“A more recent model, the clockwork RNN (CW-RNN) (Koutník et al., 2014) extends the hierarchicalRNN (El Hihi & Bengio, 1995)” : It extends the NARX model of Lin et al. 1996, not the El Hihi & Bengio, 1995.
While the above models focus on online prediction problems, where a prediction needs to be made…”: I believe there is a lot of missing references, in particular to Socher’s work or older recursive networks.
“The norm of the gradient is clipped with a threshold of 1 (Pascanu et al., 2012)”: this is not the first work using gradient clipping. I believe it was introduced in Mikolov et al., 2010.
Missing references:
“Recurrent neural network based language model.”, Mikolov et al. 2010
“Learning long-term dependencies in NARX recurrent neural networks”, Lin et al. 1996
“Sequence labelling in structured domains with hierarchical recurrent neural networks“, Fernandez et al. 2007
“Learning sequential tasks by incrementally adding  higher  orders”, Ring, 1993
Thank you for pointing out the missing references, I will fix the citation accordingly. Apologies to the authors of related work that we've missed.
The paper proposes a modified RNN architecture with multiple layers, where higher layers are only passed lower layer states if a FLUSH operation is predicted, consisting of passing up the state and reseting the lower layer's state. In order to select one of three operations at each time step, the authors propose using the straight-through estimator with a slope-annealing trick during training. Empirical results and visualizations illustrate that the modified architecture performs well at boundary detection.
Pros:
- Paper is well-motivated, exceptionally well-composed
- Provides promising initial results on learning hierarchical representations through visualizations and thorough experiments on language modeling and handwriting generation
- The annealing trick with the straight-through estimator also seems potentially useful for other tasks containing discrete variables, and the trade-off in the flush operation is innovative.
Cons:
- In a couple cases the paper does not fully deliver. Empirical results on computational savings are not given, and hierarchy beyond a single level (where the data contains separators such as spaces and pen up/down) does not seem to be demonstrated.
- It's unclear whether better downstream performance is due to use of hierarchical information or due to the architecture changes acting as regularization, something which could hopefully be addressed.
Thank you for a constructive review.
>>Empirical results on computational savings are not given
It is not entirely clear how we should implement the conditional computation in a mini-batch setting, however it is very straight forward when the batch_size is equal to 1 (e.g., at test time). We are planning to include this experiment in a near future.
>>It's unclear whether better downstream performance is due to use of hierarchical information or due to the architecture changes acting as regularization, something which could hopefully be addressed.
A good generalization performance is a result that we obtain by having the structural inductive bias (i.e., the hierarchical multiscale) incorporated in the RNN architecture. This is similar to the receptive field structure implemented in the CNNs. If we see the CNN architecture as an architectural regularizer, our model also implements an architectural regularizer. It depends on how broadly we define the term 'regularizer'. It is a regularizer in the sense that it restricts the model search space to a specific class (hierarchical multiscale).
This paper proposes a new multiscale recurrent neural network, where each layer has different time scale, and the scale is not fixed but variable and determined by a neural network. The method is elegantly formulated within a recurrent neural network framework, and shows the state-of-the-art performance on several benchmarks. The paper is well written.
Question) Can you extend it to bidirectional RNN?
We can think of two types of simple approach that exploit a backward RNN.
1) building representation based on intrinsic hierarchical structure of the data.
We can think of two HM-LSTM, forward RNN and backward RNN, let these RNNs to not interfere into inference of each other.
Then, we concatenate the hidden representations of these two HM-LSTMs and provide as input to the output module.
A downside of this approach is that, the model cannot take the full advantage of using bidirectional RNN - building representation of the next level based on the combined representation (forward RNN and backward RNN) of the previous level.
2) use future information to yield better segmentation.
We can run the backward RNN (can be standard RNN, e.g., LSTM-RNN) first and use the hidden states as auxiliary input to the forward RNN, which is HM-LSTM. The downside here is that, the backward RNN is not a HM-LSTM, it cannot take any advantage such as conditional computation and etc.

Meta-review:
role: PC
data: comment: This extension to RNNs is clearly motivated, and the details of the proposed method are sensible. The paper would have benefitted from more experiments such as those in Figure 5 teasing out the representations learned by this model.

Decision: Accept (Poster)