Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 

This paper proposes a fully connected dense RNN architecture that has connections to every layer and the preceding connections of each layer. The connections are also gated by using a simple gating mechanism. The authors very briefly discusses about the effect of these on the dynamics of the learning. They report results on PTB character-level language modelling task.


Questions:
What is the computational complexity of this approach compared to a vanilla RNN architecture?
What is the implications of these skip connections in terms of memory consumption during BPTT?
Did you use gradient clipping and have you used any specific type of initialization for the parameters?
How would this approach would compare against the Clockwork RNNs which has a block-diagonal weight matrices? [1]
How would dense-RNNs compare against to the MANNs [2]?
How would you implement this model efficiently?

Pros:
Interesting idea.
Cons:
Lack of experiments and empirical results supporting the arguments.
Hand-wavy theory.
Lack of references to the relevant literature. 

General Comments:
In general the paper is relatively well written despite having some minor typos. The idea is interesting, however the experiments in this paper is seriously lacking. The only results presented in this paper is on PTB. The results are quite behind the SOTA and PTB is a really tiny, toyish language modeling task. The theory is very hand-wavy, the connections to the previous attempts to come up with related properties of the recurrent models should be cited. The Figure 2 is very related to the Gersgorin circle theorem in [3]. The discussion about the skip-connections is very related to the results in [2]. 

Overall, I think this paper is rushed and not ready for the publication.

[1] Koutnik, J., Greff, K., Gomez, F., & Schmidhuber, J. (2014, January). A clockwork rnn. In International Conference on Machine Learning (pp. 1863-1871).
[2] Gulcehre, Caglar, Sarath Chandar, and Yoshua Bengio. "Memory Augmented Neural Networks with Wormhole Connections." arXiv preprint arXiv:1701.08718 (2017).
[3] Zilly, Julian Georg, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. "Recurrent highway networks." arXiv preprint arXiv:1607.03474 (2016).

review_title: Dense RNNs with Attention Gate

final_score: rating: 2: Strong rejection
review_content:  This paper proposes a new type of RNN architectures called Dense RNNs. The authors combine several different RNN architectures and claim that their RNN can model long-term dependencies better, can learn multiscale representation of the sequential data, and can sidestep the exploding or vanishing gradients problem by using parametrized gating units.

Unfortunately, this paper is hard to read, it is difficult to understand the intention of the authors. The authors make several claims without any supportive reference or experimental evidence. Both intuitive and theoretical justifications of the proposed architecture are not so convincing. The experiment is only done on PTB dataset, and the reported numbers are not that promising either. 

This paper tries to combine three different features from previous works, and unfortunately, it is not so well conducted.

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose an RNN that combines temporal shortcut connections from [Soltani & Jang, 2016] and Gated Recurrent Attention [Chung, 2014]. However, their justification about the novelty and efficacy of the model is not well demonstrated in the paper. The experiment part is modest with only one small dataset Penn Tree Bank is used. The results are not significant enough and no comparisons with models in [Soltani & Jang, 2016] and [Chung, 2014] are provided in the paper to show the effectiveness of the proposed combination. To conclude, this paper is an incremental work with limited contributions.

Some writing issues:
1. Lack of support in arguments,
2. Lack of referencing to previous works. For example, the sentence “By selecting the same dropout mask for feedforward, recurrent connections, respectively, the dropout can apply to the RNN, which is called a variational dropout” mentions “variational dropout” with no citing. Or “NARX-RNN and HO-RNN increase the complexity by increasing recurrent depth. Gated feedback RNN has the fully connection between two consecutive timesteps” also mentions a lot of models without any references at all.
3. Some related papers are not cited, e.g., Hierarchical Multiscale Recurrent Neural Networks [Chung, 2016]

review_title: Not exciting

Meta-review: meta score: 4
This paper concerns a variant to previous RNN architectures using temporal skip connections, with experimentation on the PTB language modelling task
The reviewers all recommend that the paper is not ready for publication and thus should be rejected from ICLR.  The novelty of the paper and its relation to the state-of-the-art is not clear.  The experimental validation is weak.
Pros:
 - possibly interesting idea
Cons:
 - weak experimental validation
 - weak connection to the state of the art
 - precise original contribution w.r.t state-of-the-art is not clear

Decision: Reject