How to Construct Deep Recurrent Neural Networks
Authors: Razvan Pascanu, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio
ICLR 2014

Abstract: In this paper, we propose a novel way to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of the depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, we define three points which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. This can be considered in addition to stacking multiple recurrent layers proposed earlier by Schmidhuber (1992). Based on this observation, we propose two novel architectures of a deep RNN and provide an alternative interpretation of these deep RNN's using a novel framework based on neural operators. The proposed deep RNN's are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNN's benefit from the depth and outperform the conventional, shallow RNN.


Peer review:
[Review]:
Solid paper describing a variety of 'deep' RNN architectures constructed by adding additional layers between input and hidden, hidden and hidden and hidden and output. The authors test their models on two types of data (polyphonic music prediction and word and character-based language modeling) and obtain good results (at least on the LM task). While the arguments they make seem compelling, we feel that the experimental results are not very convincing. In particular, on the music task there are a few problems:
- There is no best architecture so there's not much one can learn except to try them all which is not very informative. 
- To compete with an RNN with fast dropout, the authors add a lot of bells and whistles to their model (L_p units, maxout units, dropout). What happens if you train a standard RNN with the same bells and whistles ? Conversely, which of the 3 techniques made the deep models better than the RNN? A breakdown of the improvements would be helpful.
- The authors describe a deep input to hidden function model (3.2.1) however no experimental evidence of its utility is shown.
- Adding columns with the number of parameters to Table 1 would help the reader compare the sizes of the different models.
- What happens if you train a DO-RNN?