Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Post-rebuttal update:
I am happy with the rebuttal and therefore I will keep the score of 7.

This is a very interesting paper. Training RNN's in an online fashion (with no backpropagation through time) is one of those problems which are not well explored in the research community. And I think, this paper approaches this problem in a very principled manner. The authors proposes to use forward approach for the calculation of the gradients. The author proposes to modify RTRL by maintaining a rank one approximation of jacobian matrix (derivative of state w.r.t parameters)  which was done in NoBackTrack Paper. The way I think this paper is different from NoBackTrack Paper is that this version can be implemented in a black box fashion and hence easy to implement using current DL libraries like Pytorch. 

Pros.

- Its an interesting paper, very easy to follow, and with proper literature survey.

Cons:

- The results are quite preliminary. I'll note that this is a very difficult problem.
- "The proof of UORO’s convergence to a local optimum is soon to be published Masse & Ollivier (To appear)."  I think, paper violates the anonymity.  So, I'd encourage the authors to remove this. 

Some Points: 

- I find the argument of stochastic gradient descent wrong (I could be wrong though). RNN's follow the markov property (wrt hidden states from previous time step and the current input) so from time step t to t+1, if you change the parameters, the hidden state at time t (and all the time steps before) would carry stale information unless until you're using something like eligibility traces from RL literature. I also don't know how to overcome this issue. 

- I'd be worried about the variance in the estimate of rank one approximation. All the experiments carried out by the authors are small scale (hidden size = 64). I'm curious if authors tried experimenting with larger networks, I'd guess it wont perform well due to the high variance in the approximation. I'd like to see an experiment with hidden size  = 128/256/512/1024. My intuition is that because of high variance it would be difficult to train this network, but I could be wrong. I'm curious what the authors had to say about this. 

- If the variance of the approximation is indeed high, can we use something to control the dynamics of the network which can result in less variance. Have authors thought about this ? 

- I'd also like to see experiments on copying task/adding task (as these are standard experiments which are done for analysis of long term dependencies) 

- I'd also like to see what effect the length of sequence has on the approximation. As small errors in approximation on each step can compound giving rise to chaotic dynamics. (small change in input => large change in output)

- I'd also like to know how using UORO changes the optimization as compared to Back-propagation through time in the sense, does the two approaches would reach same local minimum ? or is there a possibility that the former can reach "less" number of potential local minimas as compared to BPTT. 


I'm tempted to give high score for this paper( Score - 7) , as it is unexplored direction in our research community, and I think this paper makes a very useful contribution to tackle this problem in a very principled way.  But I'd like some more experiments to be done (which I have mentioned above), failing to do those experiments, I'd be forced to reduce the score (to score - 5) 
review_title: Very interesting paper that approaches online training of RNNs in a principled way, although more experiments would make it more convincing

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a generic unbiased low-rank stochastic approximation to full rank matrices that makes it possible to do online RNN training without the O(n^3) overhead of real-time recurrent learning (RTRL). This is an important and long-sought-after goal of connectionist learning and this paper presents a clear and concise description of why their method is a natural way of achieving that goal, along with experiments on classic toy RNN tasks with medium-range time dependencies for which other low-memory-overhead RNN training heuristics fail. My only major complaint with the paper is that it does not extend the method to large-scale problems on real data, for instance work from the last decade on sequence generation, speech recognition or any of the other RNN success stories that have led to their wide adoption (eg Graves 2013, Sutskever, Martens and Hinton 2011 or Graves, Mohamed and Hinton 2013). However, if the paper does achieve what it claims to achieve, I am sure that many people will soon try out UORO to see if the results are in any way comparable.
review_title: Clever trick for making general memory-efficient online unbiased RNN learning possible

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors introduce a novel approach to online learning of the parameters of recurrent neural networks from long sequences that overcomes the limitation of truncated backpropagation through time (BPTT) of providing biased gradient estimates.

The idea is to use a forward computation of the gradient as in Williams and Zipser (1989) with an unbiased approximation of Delta s_t/Delta theta to reduce the memory and computational cost.

The proposed approach, called UORO, is tested on a few artificial datasets.

The approach is interesting and could potentially be very useful. However, the paper lacks in providing a substantial experimental evaluation and comparison with other methods.
Rather than with truncated BPTT with smaller truncation than required, which is easy to outperform, I would have expected a comparison with some of the other methods mentioned in the Related Work Section, such as NBT, ESNs, Decoupled Neural Interfaces, etc. Also the evaluation should be extended to other challenging tasks. 

I have increased the score to 6 based on the comments and revisions from the authors.
review_title: Review

Meta-review: The reviewers agree that the proposed method is theoretically interesting, but disagree on whether it has been properly experimentally validated.   My view is that the the theoretical contribution is interesting enough to warrant inclusion in the conference, and so I will err on the side of accepting.
Decision: Accept (Poster)