How to Construct Deep Recurrent Neural Networks
Authors: Razvan Pascanu, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio
ICLR 2014

Abstract: In this paper, we propose a novel way to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of the depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, we define three points which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. This can be considered in addition to stacking multiple recurrent layers proposed earlier by Schmidhuber (1992). Based on this observation, we propose two novel architectures of a deep RNN and provide an alternative interpretation of these deep RNN's using a novel framework based on neural operators. The proposed deep RNN's are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNN's benefit from the depth and outperform the conventional, shallow RNN.


Peer review:
[Review]:
Cool paper!

some minor edits:

2. 1st sentence: discrete -> discrete-time

3.1 2nd sentence: result -> results.  I would rephrase as: 'A number of recent theoretical results support this hypothesis'

3.2
Furthermore, there IS a wealth...

'in this paper we only consider feedforward, intermediate layers' - If I'm reading this right, there should be no comma.

3.3.1 'approximate f_h instead' In your notation isn't f_h the function that the network performs, not the 'true' transition function you hope to approximate?

'We call this RNN with a multilayered transition function a deep transition RNN (DT-RNN).' - this was already defined in 3.2

3.3.3 I'd cite Schmidhuber again in the first sentence

4. I didn't understand this sentence: 'Note that this is different from (Mikolov et al., 2013a) where the learned embeddings of words were suitable for algebraic operators'

5. 'on three different tasks' -> 'on two...' same problem next sentence