Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a new method for aggregating temporal information in reinforcement learning policies. The method takes the difference of latent representations between consecutive frames and concatenates this difference with the latent representations for downstream processing.

Strengths:
- The method is simple and easy to understand and implement.
- The method outperforms a baseline on average based on experiments on 5 Deepmind control benchmark suite tasks under a limited budget of training samples.
- Ablations show an interesting result that the fusion of latent representations performs better than pixel-based fusion.

Weaknesses:
- The contribution is incremental in my opinion. It makes a small change over existing RL methods. The improvement in performance on several tasks is also marginal. 
- One of the most common methods of aggregating temporal information is just using a recurrent layer in RL policies. It’s unclear if the baseline consists of a recurrent layer. If it does not, a simple recurrent policy needs to be added as a baseline. If it does, it is unclear to me why a simple subtraction of latent representations would result in performance improvement. I would imagine that it should be very easy for a recurrent layer such as an LSTM to learn differences between latent representations if they were useful. Some discussion on this would be useful.
- The performance gain is only demonstrated on 5 tasks and it’s unclear whether it would translate to gains in other tasks as well. Results in the appendix on 6 easier tasks show results comparable to or worse than the baseline.

Overall, I believe that the paper proposes a simple and interesting method, but I am not convinced that it would lead to better results consistently. The main experiments are conducted only on 5 tasks and the performance gains are marginal in my opinion.

Update after rebuttal: 
The authors have added a recurrent SAC baseline to one set of experiments. The results indicate that the recurrent SAC is a much stronger baseline, and the variance of results is high enough that I am not convinced of the benefits of the proposed method.

The authors argue that "many state-of-the-art RL algorithms" "are non-recurrent", and "frame stacking" is "largely untouched since its inception and is used in most state-of-the-art RL architectures", "recurrent architectures" have "additional overhead from training and implementation". I do not believe this is true. Recurrent architectures are commonly used in RL algorithms (for example RSSM in Dreamer) and are widely available in open-source implementations (for example https://github.com/openai/baselines/blob/master/baselines/common/models.py). There are some prior papers which use the frame stacking heuristic for a fair comparison with DQN, but this heuristic or the non-recurrent model architecture is not a part of the RL algorithm itself. Since this paper proposes a method for extracting temporal information, LSTM/GRU are very natural baselines in my opinion and should be added to all experiments.

The authors argue that "it is very reasonable and common to have a new method improving a majority of the environments but not all", I agree with this, but the examples given by the authors such as Rainbow DQN, Dueling DQN, Dreamer etc perform experiments in many more environments and performance improvements are larger. I believe a much larger scale study is needed to compare Flare with recurrent baselines and make conclusive statements about performance gains.




review_title: Review 1

final_score: rating: 7: Good paper, accept
review_content: - Summary:
    - This paper presents Flare, an RL method that replaces frame stacking (early fusion) with latent vector stacking (late fusion) and then further improve upon this by adding in latent flow vectors (the difference between adjacent latent vectors)
    - The method is demonstrated on DM control using RAD-SAC as the baseline.  In all but one case, Flare outperforms the baseline
    - The authors then present a series of ablations to show that latent flow outperforms pixel flow and that stacking latent flow is better than pure latent stacking
- Strengths:
    - Relatively straightforward idea that improves upon the common technique of frame stacking, making this likely a technique with broad appeal and impact
    - Good ablation study showing how the individual components.
    - Well motivated idea.  I liked the experiment of position only SAC to set the motivation.
- Weaknesses
    - Concat + MLP seems like a poor way to do sequence modeling (a series of frames is, after all, a sequence).  How does Flare and the latent frame baseline behave if a GRU or LSTM is used to combine the the sequence of latent vectors?  This would avoid the normal challenges of training a recurrent policy while also benefiting from the superior sequence modeling of an RNN.
    - Performance on pendulum degrading as the number of frames is increased is concerning.  If possible, I wold like to see the hypothesis posed in Sec 6.3 Q3 validated by training longer.  Another possible hypothesis is that latent vector stacking increases the number of parameters and causes Q function overfitting.
    - Questions:
        - How was RAD applied to series of frames?  Was the same translation applied to all or was a different one applied too each?
- Overall
    - This paper presents and effective idea, however, there are some additional experiments (using an RNN to combine latent vectors) that I think would strength the paper considerably


## Post Rebuttal

I thank the authors for their response.  The addition of the recurrent SAC baseline helps the paper.  I disagree with R1 that it is a stronger baseline as FLARE outperforms it in all tasks and stack SAC similar or better three (arguably four) of five tasks. Instead it shows that recurrence isn't common in off-policy RL because it doesn't always perform better.  While recurrence is considerably more common in embodied 3D environments and this work may be less applicable there, I don't foresee DM control style RL benchmarks going away anytime soon and this believe this method will be useful.
review_title: Official Blind Review #4

final_score: rating: 3: Clear rejection
review_content: Significance:
The paper brings very little novelty or insight. It is unclear that the introduced architecture complexity worth marginal improvements (given high variance and only 5 random seeds) on 2 out of 11 tasks (5 from the main paper and 6 from appendix). This might be a good workshop paper but it clearly does not meet the high acceptance threshold of ICLR.

Pros:
-A simple architecture modification that might be beneficial to impose a stronger inductive bias on temporal dynamics.

Cons:
-The proposed algorithm is a trivial architecture change to the conv encoder, the introduced novelty is limited. Injecting the temporal difference inductive bias obviously will be beneficial, given that the sole reason for frame stacking is to infer velocity and acceleration. 
-In Fig 3 the authors chose to only use 2 consecutive frames for State SAC, while a common practice is to use 3 frames. Using only 2 consecutive frames is not enough to infer acceleration and thus it is not a realistic setup, which makes this comparison meaningless. Also given that the variance is pretty high here, comparing performance over just 3 random seeds is not statistically conclusive. 
-In Fig 6 the method is only evaluated only on 5 seeds and given that it demonstrates very high variance on the 3 tasks (out of 5) where it outperforms RAD it makes me think that the performance improvements are marginal and not worth introducing complexity. 
-The ablation study is not very illuminating, this partially comes from the fact that the results are inconclusive (due to the high error bars), and partially because the experiments themselves are not very interesting. 


Quality:
While the paper is well executed and made it significantly lacks on novelty and significance fronts.

Clarity:
The paper in general is clearly written and well organized.

review_title: Frame stacking that is done on image embeddings

final_score: rating: 7: Good paper, accept
review_content: Summary:
This work presents a simple technique (Flare) to incorporate explicit temporal information to enable effective RL policy learning in challenging continuous control environments using pixel-based state representations. The approach is inspired from recent advances in the video recognition approaches which employ optical flow information and late fusion to incorporate temporal information. Typically, RL algorithms employ a frame-stacking heuristic to incorporate temporal information (early fusion). Though computing optical flow is slow and can been prohibitive for real-time applications, the authors present a simple alternative to using optical flow, i.e. difference of latent state vectors as a proxy for explicitly encoding motion information with latent vectors representing the observed state (late fusion). Experimental results on challenging continuous control tasks in the DMControl Suite show that Flare can achieve up to 1.9x higher scores than a baseline algorithm (RAD) which uses the frame-stacking heuristic to incorporate temporal information.

########################

Pros:
- The presented approach provides an effective alternative to the frame stacking heuristic for incorporating temporal information in RL with pixel-based state representations. The presented methodology (concatenation of latent state vector differences) thus learns effective policies on challenging continuous control environments.
- The presented approach can easily modify any RL algorithm operating on pixel-based state representations to encode temporal information. 
- The paper is well-written, clear and easy to follow.
- The idea is well-motivated with experiments in environments with low-dimensional state spaces. The results from the motivation section show the importance of temporal information, and specifically explicit temporal information to learn effective policies.
- Ablation study clearly highlights the merits of including explicit temporal information with late fusion ruling out other techniques like pixel-based flow (early fusion) and the effect of independent convolutional feature extraction for different image frames.

########################

Cons:
- Comparisons to other approaches mentioned in Section 2 are missing.  For example, how would Flare compare in performance and sample efficiency to LSTM based RL methods (such as those under the “neural network architectures” subsection listed under Section 2)?
- Flare does not outperform RAD in all environments (such as hopper hop and walker run). It is unclear why Flare works well on some environments and not so on others. In fact, Flare performs worse than RAD on walker run. Why so?


########################

Reason for score:
The approach is well-motivated, and the paper is clearly written. The experiments comparing with an early fusion approach (RAD) and related ablative analysis highlighting that explicit late fusion of temporal information is key to improved performance are well-done and prove the effectiveness of the Flare. However, comparison to other approaches incorporating explicit temporal information for RL is missing (eg: LSTM based approaches). 

########################

Questions during rebuttal:
- Please refer to questions in the Cons section and other feedback
- For results from Figure 6, why does Flare not outperform RAD for the hopper hop and walker run environments? Is there a way to visualize the temporal information to further investigate why Flare outperforms RAD in some environments (like quadruped walk) and not on others?

########################

Some typos and other feedback:
- Figure 2 and Section 5, paragraph 1, sentence 3: What is meant by proprioceptive state input? Consider formally defining it in the text for the reader.
- Section 5, paragraph 1, sentence 3: Consider ending the sentence by stating the fact that the experiments suggest the alternative to frame stacking heuristic is also effective in terms of performance.
- Consider defining p(a_t+1|a_t,o_t) as the transition function in the text for the reader.
- Section 5.1, last sentence: “… are done in the same except with augmented observations.” -> “… are done in the same way except with augmented observations.”
- Section 6, paragraph 1, sentence 1: “… that are experiments focus on.” -> “… that our experiments focus on.”
- Section 6.2, sentence 2:  You seem to have forgotten to mention the environments for which Flare outperforms RAD. What are you referring to with the phrase “remaining environments”?
- Section 6.2, sentence 5: “… walker run shown visualized in Figure 6.” -> “… walker run as shown in Figure 6.“
- Section 6.2, Figure 6: Why does Flare not perform as well as RAD for the Walker run environment?
- Conclusion, last sentence: Consider replacing “We would like to integrate Flare with model-based RL in the future” with “Integrating Flare with model-based RL is a potential direction for future work.”
- Consider replacing the usage of the phrase “state-based RL” with “RL on low-dimensional state space”.
review_title: Official Blind Review #3

Meta-review: This paper provides a simple approach to incorporate temporal information in RL algorithms. AC agrees with authors that simplicity is a virtue. As reviewers point out that experimentally the approach is not conclusively better (given that environments might be hand-chosen). Even R3 believes some reported improvements is within variance. Given the discussions, AC agrees that results do not seem convincing enough.
Decision: Reject