How to Construct Deep Recurrent Neural Networks
Authors: Razvan Pascanu, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio
ICLR 2014

Abstract: In this paper, we propose a novel way to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of the depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, we define three points which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. This can be considered in addition to stacking multiple recurrent layers proposed earlier by Schmidhuber (1992). Based on this observation, we propose two novel architectures of a deep RNN and provide an alternative interpretation of these deep RNN's using a novel framework based on neural operators. The proposed deep RNN's are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNN's benefit from the depth and outperform the conventional, shallow RNN.


Peer review:
[Review]:
A new version of the paper is available on arXiv.org, which incorporates most of the comments made by the reviewers. In particular, we have made the following changes:

* (69d2) In the abstract we say that we 'explore' instead of 'propose a novel way'.

* (69d2) Changed from 'modeling variable-length sequences' to 'evaluated on two sequence modeling tasks'.

* (69d2) Removed the normalization by the length of a sequence in Eq. (3)
  
* (69d2) Added information on how many levels of recurrent layers we used with sRNN.
  
* (bfb0) Added the number of parameters for each model in Table 1. However, note that the model sizes were selected based on validation errors, meaning larger models also have been tried.

* (Bayer) Emphasized the difference between sRNN and DT-RNN. Also, strengthend the intuition behind the reason why the deep
transition is a useful extension to RNN which is orthogonal to the other variants of deep RNNs.

* (1f07) Added more details about Pinheiro and Collobert (2013).

* (1f07) Added the previous results by Mikolov and Tomas in Table 3.

* (1f07) Added a footnote briefly explaining the method of dynamic evaluation.

We would like to thank all the three reviewers for their comments. We are still aiming to improve the paper, particularly by having started more experiments as suggested by the reviewers.

We believe that our paper provides a novel perspective on the depth of RNNs and the empirical evidence showing that this direction of research is worth pursuing. We hope the reviewers find this work up to the quality standard of the conference.