Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on accelerating RNN by applying the method from Blelloch (1990). The application is straightforward and thus technical novelty of this paper is limited. But the results are impressive. 

One concern is the proposed technique is only applied for few types of RNNs which may limit its applications in practice. Could the authors comment on this potential limitation?
review_title: simple but effective method for RNN speed up

final_score: rating: 7: Good paper, accept
review_content: # Summary and Assessment

The paper addresses an important issue–that of making learning of recurrent networks tractable for sequence lengths well beyond 1’000s of time steps. A key problem here is that processing such sequences with ordinary RNNs requires a reduce operation, where the output of the net at time step t depends on the outputs of *all* its predecessor. 
The authors now make a crucial observation, namely that a certain class of RNNs allows evaluation in a non-linear fashion through a so-called SCAN operator. Here, if certain conditions are satisfied, the calculation of the output   can be parallelised massively.
In the following, the authors explore the landscape of RNNs satisfying the necessary conditions. The performance is investigated in terms of wall clock time. Further, experimental results of problems with previously untacked sequence lengths are reported.

The paper is certainly relevant, as it can pave the way towards the application of recurrent architectures to problems that have extremely long term dependencies.
To me, the execution seems sound. The experiments back up the claim.

## Minor
- I challenge the claim that thousands and millions of time steps are a common issue in “robotics, remote sensing, control systems, speech recognition, medicine and finance”, as claimed in the first paragraph of the introduction. IMHO, most problems in these domains get away with a few hundred time steps; nevertheless, I’d appreciate a few examples where this is a case to better justify the method.
review_title: Authors propose a method to make recurrent learning over 1000s and more time steps possible.

final_score: rating: 7: Good paper, accept
review_content: This paper abstracts two recently-proposed RNN variants into a family of RNNs called the Linear Surrogate RNNs which satisfy  Blelloch's criteria for parallelizable sequential computation. The authors then propose an efficient parallel algorithm for this class of RNNs, which produces speedups over the existing implements of Quasi-RNN, SRU, and LSTM. Apart from efficiency results, the paper also contributes a comparison of model convergence on a long-term dependency task due to (Hochreiter and Schmidhuber, 1997). A novel linearized version of the LSTM outperforms traditional LSTM on this long-term dependency task, and raises questions about whether RNNs and LSTMs truly need the nonlinear structure.

The paper is written very well, with explanation (as opposed to obfuscation) as the goal. Linear Surrogate RNNs is an important concept that is useful to understand RNN variants today, and potentially other future novel architectures.

The paper provides argument and experimental evidence against the rotation used typically in RNNs. While this is an interesting insight, and worthy of further discussion, such a claim needs backing up with more large-scale experiments on real datasets.

While the experiments on toy tasks is clearly useful, the paper could be significantly improved by adding experiments on real tasks such as language modelling.
review_title: Faster RNNs, with novel insights on need for nonlinear recurrence; novel and clear presentation

Meta-review: Paper presents a way in which linear RNNs can be computed (fprop, bprop) using parallel scan. They show big improvements in speedups and show application on really long sequences. Reviews were generally favorable.
Decision: Accept (Poster)