How to Construct Deep Recurrent Neural Networks
Authors: Razvan Pascanu, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio
ICLR 2014

Abstract: In this paper, we propose a novel way to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of the depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, we define three points which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. This can be considered in addition to stacking multiple recurrent layers proposed earlier by Schmidhuber (1992). Based on this observation, we propose two novel architectures of a deep RNN and provide an alternative interpretation of these deep RNN's using a novel framework based on neural operators. The proposed deep RNN's are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNN's benefit from the depth and outperform the conventional, shallow RNN.


Peer review:
[Review]:
The paper proposes ways of extending ordinary RNNs to deep RNNs, where deep operations are used in favor of shallow (=affine with an elementwise nonlinearity) at various stages.

I think this paper is important especially because of the new deep transitions. It will help practitioners as well as researchers with designing models for sequential data.

During my first read I was not sure what the actual benefit of a DOT-RNN is compared to an sRNN with one additional hidden layer--but if one looks closely, the amount of layers information travels through scales faster for a DOT-RNN than for an sRNN. Maybe the authors would like to explicitly state this (maybe they did and I missed it).

I also think it is valuable that the optimisation of these models has been identified as one of the major problems, such that the community can focus on trying to improve on this. I applaud the availability of the source code.

The experimental evaluation is solid and the results are excellent.