Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This work presents the Simple Recurrent Unit architecture which allows more parallelism than the LSTM architecture while maintaining high performance.

Significance, Quality and clarity:
The idea is well motivated: Faster training is important for rapid experimentation, and altering the RNN cell so it can be paralleled makes sense. 
The idea is well explained and the experiments convince that the new architecture is indeed much faster yet performs very well.

A few constructive comments:
- The experiment’s tables alternate between “time” and “speed”, It will be good to just have one of them.
- Table 4 has time/epoch yet only time is stated
review_title: Nice idea, tested extensively

final_score: rating: null
review_content: In Tables 6 / 9: 
It is not clear why SRU model capacity was increased in depth (to 12 layers) and not in width, which would give an even faster model I would think. As you mention for LSTM 5 layers appear to be optimal, so it is surprising that 12 were needed for SRU.
review_title: Nice work

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors introduce SRU, the Simple Recurrent Unit that can be used as a substitute for LSTM or GRU cells in RNNs. SRU is much more parallel than the standard LSTM or GRU, so it trains much faster: almost as fast as a convolutional layer with properly optimized CUDA code. Authors perform experiments on numerous tasks showing that SRU performs on par with LSTMs, but the baselines for these tasks are a little problematic (see below).

On the positive side, the paper is very clear and well-written, the SRU is a superbly elegant architecture with a fair bit of originality in its structure, and the results show that it could be a significant contribution to the field as it can probably replace LSTMs in most cases but yield fast training. On the negative side, the authors present the results without fully referencing and acknowledging state-of-the-art. Some of this has been pointed out in the comments below already. As another example: Table 5 that presents results for English-German WMT translation only compares to OpenNMT setups with maximum BLEU about 21. But already a long time ago Wu et. al. presented LSTMs reaching 25 BLEU and current SOTA is above 28 with training time much faster than those early models (https://arxiv.org/abs/1706.03762). While the latest are non-RNN architectures, a table like Table 5 should include them too, for a fair presentation. In conclusion: the authors seem to avoid discussing the problem that current non-RNN architectures  could be both faster and yield better results on some of the studied problems. That's bad presentation of related work and should be improved in the next versions (at which point this reviewer is willing to revise the score). But in all cases, this is a significant contribution to deep learning and deserves acceptance.

Update: the revised version of the paper addresses all my concerns and the comments show new evidence of potential applications, so I'm increasing my score.
review_title: Very useful RNN cell with ok results but over-hyped presentation.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose to drop the recurrent state-to-gates connections from RNNs to speed up the model. The recurrent connections however are core to an RNN. Without them, the RNN defaults simply to a CNN with gated incremental pooling. This results in a somewhat unfortunate naming (simple *recurrent* unit), but most importantly makes a comparison with autoregressive sequence CNNs [ Bytenet (Kalchbrenner et al 2016), Conv Seq2Seq (Dauphin et al, 2017) ] crucial in order to show that gated incremental pooling is beneficial over a simple CNN architecture baseline. 

In essence, the paper shows that autoregressive CNNs with gated incremental pooling perform comparably to RNNs on a number of tasks while being faster to compute. Since it is already extensively known that autoregressive CNNs and attentional models can achieve this, the *CNN* part of the paper cannot be counted as a novel contribution. What is left is the gated incremental pooling operation; but to show that this operation is beneficial when added to autoregressive CNNs, a thorough comparison with an autoregressive CNN baseline is necessary.

Pros:
- Fairly well presented
- Wide range of experiments, despite underwhelming absolute results

Cons:
- Quasi-RNNs are almost identical and already have results on small-scale tasks.
- Slightly unfortunate naming that does not account for autoregressive CNNs
- Lack of comparison with autoregressive CNN baselines, which signals a major conceptual error in the paper.
- I would suggest to focus on a small set of tasks and show that the model achieves very good or SOTA performance on them, instead of focussing on many tasks with just relative improvements over the RNN baseline.

I recommend showing exhaustively and experimentally that gated incremental pooling can be helpful for autoregressive CNNs on sequence tasks (MT, LM and ASR). I will adjust my score accordingly if the experiments are presented.


review_title: Low novelty and lacks comparison with obvious baselines

Meta-review: The paper presents Simple Recurrent Unit, which is characterised by the lack of state-to-gates connections as used in conventional recurrent architectures. This allows for efficient implementation, and leads to results competitive with the recurrent baselines, as shown on several benchmarks.

The submission lacks novelty, as the proposed method is essentially a special case of Quasi-RNN [Bradbury et al.], published at ICLR 2017. The comparison in Appendix A confirms that, as well as similar results of SRU and Quasi-RNN in Figures 4 and 5. Quasi-RNN has already been demonstrated to be amenable to efficient implementation and perform on a par with the recurrent baselines, so this submission doesn’t add much to that.
Decision: Reject