Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors reformulate the RNN training objective to explicitly learn the state vectors, and propose an algorithm called Fixed Point Propagation (FPP Algorithm 1). The authors motivate the derivation of FPP in Section 3, provide some theoretical convergence results in Section 4, and demonstrate experiment results in Section 5.

In general, this paper is interesting and well written. The experiment results in Section 5 seem to be very strong. However, I am not familiar with the relevant literature, thus, I am not sure if the authors have compared with the strongest baselines in this field. 

I think the paper suffers from the following limitations:

1) Theorem 1 shows that the FPP algorithm converges to a stationary point. However, this result seems to be too weak. Can we say something about the stationary point? Is it a local minimum under some conditions?

2) In the experiments, the authors choose \lambda=1. My understanding is that \lambda is a key meta-parameter of FPP. Please do a better job in justifying this choice.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes an alternative to the truncated back-propagation through time (BPTT) algorithm for training RNNs. An online setting is assumed, which I understand as training RNN as data arrives and not storing too much of the arriving data (although notably the proposed method uses a buffer). The proposed Fixed-Point Propagation algorithm works as follows. It maintains a buffer of the last N RNN states that can be updated. From this buffer at every time step it samples two states that are T steps apart from each other (s_i and s_{i - T}). The RNN is run for T steps starting from s_{i - T}. A loss function is constructed that takes into account the output loss at time i as well as the mismatch between s_i and the new state constructed by running the RNN. The states s_i and s_{i-T}, as well as the RNN parameters are updated based on this loss function. 

The novel idea of the paper is therefore a modifiable state buffer for the RNN states. The goal is better computational efficiency than that of T-BPTT. 

The paper is mostly clearly written, but I think it is absolutely necessary to move Algorithm 1 to the main text, as well as to add the mini-batch processing (B) and multiple updates (M) to it. This pseudocode was very instrumental for me to understand the algorithm. I confess that I did not read the theory; I don’t think it’s super relevant because in practice convergence to fixed-point will require too many updates. 

The empirical comparison with T-BPTT is substantial, but the waters are muddied a bit by imprecise presentation of baselines. For example, when T-BPTT is used for e.g. language modelling, it doesn’t make sense for back-propagate the loss from only the last time step, losses from all time-steps can be back-propagated together. Was this done in T-BPTT and/or FPP? Does T-BPTT use the 100 step buffer somehow? NoOverlap T-BPTT is not explained very well. A very interesting and absolutely necessary baseline is FPP without state updates, but for such a baseline the loss comparing s_t and s_{i-T} should be disabled. Was this done?

In short, the paper must clear show that updating the states in the buffer allows to get same performance with smaller T, compared to the best possible baseline that also uses the buffer but does not update states in it. I am not sure this case is clearly made at the moment.
One further direction authors could explore is that using a very small T but larger B could be more computationally inefficient because parallel computations would be used instead of sequential ones. Besides, from a practical viewpoint and I think it could make sense to also update intermediate states, and not just s_i and s_{i-T}. 

Other remarks:
- the legend in Figure 4 is a dashed line, but the curves in the plots are dotted
- the max. cycle length in CycleWorld is not clearly explained in the text, the name CycleWorld is not properly introduced

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: 
Background: The authors consider the problem of training RNNs in an online fashion. The authors note that RNNs are trained using BPTT, which prevents them from being trained in an online fashion.  There have been various approximations which has been proposed which are based on RTRL or approximations to RTRL, as current approximations based on RTRL has high computational complexity. 

Proposed Method: The authors propose to learn the state of the RNNs explicitly by improving the prediction accuracy at each time step as well as predicting the "next" state of the RNN. The authos note that the constraint of predicting the next state  is a fixed-point formula for the states underthe given RNN dynamics.

Clarity of the paper: The paper is clearly written. 

Related work : Most of the relevant related work has been covered in the paper and discussed. I like it. These two related work could also be cited. Here, authors approximate the RTRL with random kronecker factors.
https://papers.nips.cc/paper/7894-approximating-real-time-recurrent-learning-with-random-kronecker-factors
https://www.biorxiv.org/content/10.1101/458570v1

Experiment section: The authors evaluate the proposed method on both synthetic as well as real experiments. 

Simulation Problems: The authors use simulation problems to note the robustness of the proposed method to increasing termporal delay in online learning. These tasks show the soundness of the proposed method. Its actually difficult to tell how the proposed method is performing because of the selection of tasks. It might be more interesting to choose same tasks as in UORO paper (https://arxiv.org/abs/1702.05043) and it could be another "baseline" for the proposed method.

Ablations: I liked the fact that the authors consider conducting experiments without state updating, as it could also be  due to using a large buffer rather than explicitly optimizing for the prediction objective.

 Positive: The proposed method could be interesting for learning the state representation for policy gradient RL methods (specifically POMDPs) as the proposed method can leverage use of mini-batches, as well as multiple-updates which is a crucial ingredient to make best use of data collected by the agent interacting with the environment. 

review_title: Official Blind Review #1

Meta-review: The paper proposes an alternative to BPTT for training recurrent neural networks based on an explicit state variable, which is trained to improve both the prediction accuracy and the prediction of the next state. One of the benefits of the methods is that it can be used for online training, where BPTT cannot be used in its exact form. Theoretical analysis is developed to show that the algorithm converges to a fixed point. Overall, the reviewers appreciate the clarity of the paper, and find the theory and the experimental evaluation to be reasonably well balanced. After a round of discussion, the authors improved the paper according to the reviews. The final assessments are overall positive, and I’m therefore recommending accepting this paper.
Decision: Accept (Poster)