How to Construct Deep Recurrent Neural Networks
Authors: Razvan Pascanu, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio
ICLR 2014

Abstract: In this paper, we propose a novel way to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of the depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, we define three points which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. This can be considered in addition to stacking multiple recurrent layers proposed earlier by Schmidhuber (1992). Based on this observation, we propose two novel architectures of a deep RNN and provide an alternative interpretation of these deep RNN's using a novel framework based on neural operators. The proposed deep RNN's are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNN's benefit from the depth and outperform the conventional, shallow RNN.


Peer review:
[Review]:
The paper explores a variety of ways of making RNNs 'deep'. Some of these techniques, such as stacking multiple RNN hidden layers on top of each other, and adding additional feedforward layers between the input and hidden layers, or between hidden and output, have been explored in the literature before. Others, such as adding extra layers between the hidden to hidden transitions, are more unusual. In its simplest form this would be more or less equivalent to inserting a 'null input' at every second step in the input sequences. But as the authors point out, this would exacerbate vanishing gradients, so they add skip connections too.

As with all neural networks, the space of RNN topologies is vast. This paper doesn't cover a huge amount of new territory, but it does do a good job of exploring a few promising variants. 

The experimental results don't suggest a massive benefit from deeper architectures, or a consistent advantage of one over the other, but they do show that adding depth can make a difference. Interestingly, the effect of using dropout, maxout and other generic neural network tricks seems to be much greater than the effect of changing the architecture.

The results on the Penn Treebank data are very good. But it isn't entirely clear why they're so good. In particular, the standard shallow RNN with a small hidden layer appears to perform much better than similar networks elsewhere in the literature, e.g. by Tomas Mikolov. Is this a consequence of the training method, the regularisation, or something else? A few more experimental details would help to clarify this.

Small comments:

abstract: - the paper doesn't really propose 'a novel way' to extend RNNs so much as explore several possible ways.
- 'modelling variable-length sequences' is a bit too broad to be a 'task'... maybe 'evaluated on two sequence modelling tasks'?
- The 'In Sec. 2' paragraph is more confusing than helpful
- Eq. (3): do the denominators (1/N and 1/T_n) really belong in the loss? I'd say not... otherwise training would be biased towards shorter sequences and smaller datasets. The dataset factor can be incorporated into the learning rate, but I don't think you want to normalise by sequence length.
- I found the repetition of the h_t and y_t terms in eqs (4) and (5) confusing
- How many hidden layers did the 'sRNN' have? I couldn't find this.
- 'Practical Variational Inference for Neural Networks' would be a better reference for adaptive/fixed weight noise.