How to Construct Deep Recurrent Neural Networks
Authors: Razvan Pascanu, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio
ICLR 2014

Abstract: In this paper, we propose a novel way to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of the depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, we define three points which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. This can be considered in addition to stacking multiple recurrent layers proposed earlier by Schmidhuber (1992). Based on this observation, we propose two novel architectures of a deep RNN and provide an alternative interpretation of these deep RNN's using a novel framework based on neural operators. The proposed deep RNN's are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNN's benefit from the depth and outperform the conventional, shallow RNN.


Peer review:
[Review]:
This paper proposes three techniques to make recurrent neural networks (RNN) 'deeper'. In addition to an inherent deepness over time, the authors propose to consider additional hidden layers for all the transition matrices, i.e. input-to-hidden, hidden-to-hidden and hidden-to-output.

To implement those ideas, neural operators are introduced:
 - predicting the next hidden state in function of the previous one and the input
 - predicting the output given the current hidden state

those operators are implemented with multi-layer neural networks, in principle of any arbitrary depth.  However, only one-hidden layer operator networks are considered.  I think that it would be really interesting to consider even deeper operators. Do we achieve additional improvements ?  Is it possible to train these networks with back-prop (through time) ?

In my opinion, you do not consider deep input-to-hidden RNN. In Figure 2b you propagate h_t-1 and x_t through the SAME hidden layer to h_t. I propose to use separate hidden layers for these tasks, i.e. one deep operator to map the input to the hidden state, and an INDEPENDENT deep operator to map from one hidden to the next hidden state. The input-to-hidden operator could be quite deep since you won't have the problem of vanishing gradients over time.

You write that you initialize the sRNN and DOTS-RNN with the weights of a normal RNN to simulate layer-wise pretraining. How do you initialize the weights of the additional hidden layers ?  Is this really comparable to layer-wise pre-training of deep MLPs ? Layer-wise pre-training is unsupervised while you need targets. Maybe we should say that you initialize the weights so that the network is easier to train ? Can you elaborate how important this initialization is ?
Do the deep RNN still converge when starting from random weights ?

Language modeling: 
 - please add the results of Graves and Mikolov to Table 3
 - please explain 'dynamic evaluation'

In general, it would be helpful that the authors provide more details on the architectures which do NOT work very well.  This would allow the reader to assess the importance of the different tricks, e.g. short cut connection, initialization scheme, etc.  By these means, one could more easily reproduce the experiments and apply the same ideas to other tasks.

The authors should discuss the relation to other works in more detail, in particular Pinheiro and Collobert (2013)

Overall, this paper presents interesting ideas and opens directions for further research.