How to Construct Deep Recurrent Neural Networks
Authors: Razvan Pascanu, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio
ICLR 2014

Abstract: In this paper, we propose a novel way to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of the depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, we define three points which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. This can be considered in addition to stacking multiple recurrent layers proposed earlier by Schmidhuber (1992). Based on this observation, we propose two novel architectures of a deep RNN and provide an alternative interpretation of these deep RNN's using a novel framework based on neural operators. The proposed deep RNN's are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNN's benefit from the depth and outperform the conventional, shallow RNN.


Peer review:
[Review]:
Nice paper. Good to see that the deep RNN thread is being continued. A few thoughts, questions and ideas (I’m afraid it became a bit more lengthy than initially anticipated):

- I only have one true criticism on the content of the paper. The biggest issue I have with the results on the character-wise prediction task (the only task I have a little experience in myself) is that the performance seems to correlate rather strongly with the number of trainable parameters in each network. I fear this might in the end play a more important role than the actual network architecture. What I found when I was working on a highly similar task is that the number of parameters is the single biggest determining factor in model performance, simply because of (much needed) raw storage power. Network architecture itself only provides a modest improvement. I would find the results in this part far more convincing if all architectures are given the same number of trainable parameters.

- Concerning deep transitions: the authors rightfully state that deep RNNs can be deep both in their transition and in a more spatial sense. When I was working with deep (stacked) RNNs, in the end the reason I didn’t try deep transitions (by feeding top-layer input back into the bottom layer at the next time step) was the fact that it slowed down the simulations too much (harder to exploit parallel computing architectures). I wonder if the authors could give some comments on how fast the simulations are for each model.

- The discussion at the end of 3.2.4. is interesting. Stacked RNNs could be stated to have a `deep’ transition, but only from lower to higher layers. This means that the higher layers can represent complex features of a summarised history of the input. In this sense, they should also be able to model complicated state transitions, as each time step new information needs to travel through all the layers. What they are *not* able to do is have deep, hierarchical transitions between the high-level representations themselves. The question then arises whether this is *necessary*, as a high level representation should already consist of a conceptualised representation of the input information, in which properties of the input, including complicated transitions, should already be easier to perform.
After reading your paper, I believe the best way to systematically investigate deep recurrent topologies is as follows: represent each time step as a vertical hierarchy with a fixed number of layers, and only study different way the layers are connected in time (over a state transition). In this way all topologies discussed in the paper, and more, can be represented and studied in a systematic manner. By keeping the number of layers fixed you also eliminate any performance difference caused by the data running through more or less layers. I hope this is more or less clear, as I cannot post a visual representation of what I mean.