Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a modification of RNN that does not suffer from vanishing and exploding gradient problems. The proposed model, GATO partitions the RNN hidden state into two channels, and both are updated by the previous state. This model ensures that the state in one of the parts is time-independent by using residual connections. The experiments on the long copy and adding tasks, as well as language modeling on the Penn TreeBank dataset show the performance improvement against the basic LSTM and GRU. 

The paper tackles an interesting and challenging problem with a novel approach in sequence modeling. The idea is clear and the paper is well-written. The mathematical insights are well reasoned. 

The proposed method outperforms LSTM and RNN with much fewer number of parameters. However, there is no regularization is used for such big LSTM/GRU models. There is a chance that such a big LSTM/GRU model increased the chance of overfitting, and therefore the performance is low. I would like to see the comparison after adding any common regularization that prevents overfitting across the recurrent connections. 

There are many advanced RNN/LSTMs proposed in recent years [1-5] addressing the vanishing gradient problem. It is hard to judge the quality of the proposed method due to the lack of evaluation/comparisons. This paper needs more intensive evaluations with recent RNN-based methods. For instance based on [6], AWD-LSTM [6] and RHN [4] achieved 52.8 and 65.4 test perplexity scores on the Penn TreeBank dataset respectively. The the best score in this paper is 112.85. 

[1] "Phased LSTM: Accelerating recurrent network training for long or event-based sequences." 2016.
[2] "Fast-slow recurrent neural networks." 2017.
[3] "Skip RNN: Learning to skip state updates in recurrent neural networks." 2017.
[4] "Recurrent highway networks." 2017.
[5] "Dilated recurrent neural networks." 2017.
[6] "Regularizing and optimizing LSTM language models." 2017

All experiments are performed with 1 or 2 layers. Hierarchical RNN/LSTM performs much better in sequence learning. Is there any reason authors only showed 1 or 2 layers? How does GATO change with more than 2 layers?

How do other choices of non-linear functions affect the performance in practice?   


Typo
r -> r_t in Eq. 6

--- 
After rebuttal:
One of my main concerns, weak baselines and unfair comparisons, was partially answered in the updated paper.  I am not fully convinced by their new comparisons.
For instance, authors mentioned that 'RHN and EURNN performed poorly because they have unbounded forward propagation'. To overcome this, they introduced 'Bounded RHN' in Append G and it performs similarly to GATO and GRU. However, this 'Bounded RHN' is the one used in the original RHN paper. Overall, it is hard to trust their additional comparisons. 
Although, I believe that this paper is well-structured and justified. Also it has high potential for the community. However, the paper itself is not ready to be published.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors propose a novel recurrent architecture called GATO. 
Specifically, the authors focused on sequence modeling tasks and developed criteria for RNN models on such tasks. 
Tha GATO model can resolve the vanishing/exploding gradient issue and is robust to initializations. 
Empirical results show GATO can outperform LSTM and RNN in both synthetic datasets and real datasets.

The key insight of the proposed model is that only part of the hidden states is recurrently updated. 
The GATO achieves this by adding the skip connection channel (or residual connection) along the temporal dimension.
GATO summarizes the hidden states r_t by recurrently adding (transformed) r_t to s_t.
This idea also appears in many previous RNN models, such as highway RNN/LSTM, Statistical/Fourier Recurrent Units [1][2].
Specifically, the proposed GATO is a special case of SRU ( alpha=1). This limits the novelty of the paper and thus make the contribution marginal.

As for the experimental studies, the authors only provide comparisons with LSTM and GRU. There are a lot of advanced RNN architectures to address vanishing/exploding gradient issues, such as uRNN[3], oRNN[4], Spectral-RNN[5] and SRU/FRU [1][2]. It would be more convincing if the 
authors could include these models into comparison.

Overall I think this paper should be further improved before being accepted.  



[1] Oliva, J.B., Póczos, B. and Schneider, J., The statistical recurrent unit. 
In ICML 2017 (pp. 2671-2680).

[2] Zhang, J., Lin, Y., Song, Z. and Dhillon, I., Learning Long Term Dependencies via Fourier Recurrent Units. 
In ICML 2018 (pp. 5810-5818).

[3] Arjovsky, M., Shah, A. and Bengio, Y., Unitary evolution recurrent neural networks. 
In ICML 2016 (pp. 1120-1128).

[4] Mhammedi, Z., Hellicar, A., Rahman, A. and Bailey, J., Efficient orthogonal parametrisation of recurrent neural networks using householder reflections. 
In ICML 2017 (pp. 2401-2409).

[5] Zhang, J., Lei, Q. and Dhillon, I., Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization. 
In ICML 2018 (pp. 5801-5809).
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper proposes a new RNN architecture designed to overcome vanishing/exploding gradient problems and to improve long-term memory for sequence modelling. The main ideas are (i) to split the hidden state into two parts, one of which does not influence the recurrence relation, and can therefore not blow up or contract by self-feedback; and (ii) to use periodic functions, in particular the cosine, as non-linearity in the decoder, so that the output is bounded but does not saturate.

The paper puts forward a fairly systematic analysis of the gradients in RNNs. The analysis appears correct, and is in fact quite similar to considerations in earlier RNN work (which is correctly cited), and forms the basis for the proposed GATO unit. There are two loose ends in this part:
1) the cosine non-linearity results from a purely negative selection - the function should be bounded, but not saturating. The paper does not even ask the question which periodic function might be a good choice.
2) While the method is presented as a grand theory, with the only constraint that a part of the hidden state does not influence the recurrence function; the actual implementation and experiments are limited to the narrow special case of "non-interacting" GATO, where the "passive" variables make up exactly half of the hidden vector, and the update of each individual hidden variable is influenced only by a single variable from the previous state. So there are in fact no empirical results, not even on toy data, for the general case that the paper claims to introduce.

In the experiments, there are two artificial problems (copying, adding) for sequences of symbols. These are illustrative and sensible to verify and analyse  the behaviour of GATO in a controlled setting, but rather far from most real sequence modelling tasks. In a third experiment the task is to classify whether or not patients will stay in intensive care for >1 week, based on a (seemingly private) database of time series of vital parameters. Unfortunately, the results for that experiment do not go beyond the usual "ours is better". While the numbers clearly support GATO, it would have been nice to look a bit closer and pinpoint what makes the difference. Also, the comparison is a bit loose. It would have been better to add additional baselines where also LSTM and GRU are restricted to "non-interacting", element-wise recurrence (as far as technically feasible). As it stands, it is unfair to claim "we can do it with much fewer parameters" - perhaps LSTM / GRU could, too. In fact, it could even be that the task is just simple, so that more restricted model with fewer parameters generally perform better - I do not claim this is the case, but the experiments do not rule it out and, hence, do not confirm that the clever GATO recurrence makes the difference.

A small gap is also that even the "real" experiment might not be completely realistic. Nowadays it is a popular strategy to use deep LSTMS / GRUs, i.e., stack multiple levels of recurrence, possibly with temporal sub-sampling, to better capture long-term relations. While this is potentially even more brittle, because of the additional gradient flow across layers, it does seem to work. But deep RNNs are not tested (in fact, not even mentioned) in the paper.

Overall, in spite of a few loose ends, I find both the design and the practical performance of the proposed GATO unit very interesting, and potentially valuable for the ICLR crowd. This is one of the more convincing RNN papers I have recently read.
review_title: Official Blind Review #2

Meta-review: This paper proposes a modification of RNN that does not suffer from vanishing and exploding gradient problems. The proposed model, GATO partitions the RNN hidden state into two channels, and both are updated by the previous state. This model ensures that the state in one of the parts is time-independent by using residual connections. 

The reviews are mixed for this paper, but the general consensus was that the experiments could be better (baseline comparisons could have been fairer). The reviewers have low confidence in the revised/updated results. Moreover, it remains unclear what the critical components are that make things work. It would be great to read a paper and understand why something works and not that something works. 

Overall: Nice idea, but the paper is not quite ready yet.


Decision: Reject