Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper suggests using an optical flow to encode pixel observations in contrast to simple concatenation over the temporal dimension. This proposed method enables efficient recovery of dynamics information such as velocity and acceleration, this in turn, improves performance on several image-based tasks from DMControl and Atari.
main_review: Originality: I think the paper is limited in its originality. While the particular application of an optimal flow like architecture to this type of setup is novel, it still a rather incremental derivation from prior work (both in CV and RL). This should not be mistaken with the simplicity aspect of Flare, which I like. 


Quality: The paper is well executed. The experimentation is sound and extensive. I like the fact that the authors consider several different domain (DMControl and Atari) and several different algorithm, which makes it easier to justify their method.

Clarity:
The paper is very well written and easy to follow. I like the motivation example and figures 2 and 3. I also like how the ablation section is structured as a Q&A, which makes it easy to comprehend.

Significance:
I believe the significance of this paper is limited. While it proposes a better way to deal with encoding temporal information in image-based RL, it feels more like a hack, rather than a systematic solution. For example, it is not hard to imagine scenarios where it is not enough to extract linear relationships between consecutive frames. In these cases more general architectures such as RNNs or Transformers will be still required.

Overall:
I think it is a well written and executed paper that demonstrates some interesting findings. I also like the fact that the authors attempt to investigate alternative architectures in RL. However, the novelty and significance aspects are limited. I'm thus borderline on this paper and biasing towards rejection.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes adding temporal information in the form of latent flow to deep RL agents. The idea is simple: temporal differences between latent embeddings of consecutive frames are computed; the differences are then concatenated to the embeddings for downstream processing by an RL controller. This idea is inspired by idea of late fusion in video processing and stands in contrast to the current approach of frame-stacking, which is more akin to early fusion. The authors have validated their approach on a series of DMC tasks as well as Atari games.
main_review: Strengths:
* the paper is well-written and the ideas clearly laid out
* the method is easily applicable to many different RL architectures
* empirical results show a clear improvement over baselines (including RNN baselines) on a selection of DMC and Atari environments
* informative ablation studies comparing frame-stacking, temporal pixel differences, and pure latent stacking

Weaknesses:
* number of frames to include adds an extra hyperparameter that needs to be tuned. The appropriate number will probably be task-dependent. The authors have demonstrated one case where extra frames do nothing and another case where it actually hurts (presumably because the agent must learn to ignore the superfluous information). However, one can conceivably imagine cases where having extra frames is necessary (e.g. if accelerations are important, then using two frames won't be enough).

Minor typos:
* line 60: “Notably” instead of “Notebly”
* line 212: “Algorithm 1” instead of “Algorithm 5.2”
* line 238: different set of games listed here compared to the table

Overall, while the idea is simple, I believe the authors have demonstrated the method does benefit RL algorithms. I am also satisfied that the authors have addressed concerns raised in their previous submission regarding a missing RNN baseline. I recommend accepting this paper.



limitations_and_societal_impact: The authors have given thoughtful consideration to the limitations of their work, including instances where their method may not work. I would have also appreciated some discussion of needing to tune the number of frames as a hyperparameter. The authors have also discussed the broader impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors modify two state-of-the-art reinforcement learning algorithms (RAD-SAC and Rainbow-DQN) by changing the way information from multiple timesteps is aggregated. Instead of stacking frames at a pixel level, the authors propose to encode each frame individually and then pass the encodings and their pairwise differences to the actor/critic networks. It is shown that when learning policies from pixels this way of fusing information substantially outperforms the baseline variants in 3/5 environments from the DeepMind control suite and 5/8 environments from Atari2600. Furthermore, experiments with low-level observations on 5 DeepMind Control Suite tasks support the particular choice of information fusion over using recurrent networks or stacking states directly without including state differences.

main_review: **Originality:**
The proposed way to fuse information from multiple frames / timesteps is simple but novel. It questions the "standard way" of stacking frames to incorporate information from multiple timesteps into the RL algorithm, which is interesting.


**Quality:**

* The authors perform reasonable experiments to evaluate the advantage of their proposed method for low-dimensional and high-dimensional observations

* The motivation for selecting the subset of environments reported in Table 1 is unclear to me. In other work such as CURL, Dreamer and PlaNet, different environments and timesteps were selected (namely Finger Spin, Cartpole Swingup, Reacher Easy, Cheetah Run, Walker Walk, Cup Catch) at 100k steps / 500k steps. Evaluating FLARE on these environments and timestep limits would greatly simplify comparison with the above methods.
  
* I am concerned with the fairness of comparing the "stack frames" and "stack latents" variants to FLARE. In FLARE the "fused" (concatenated) vector of latents and differences is passed through a fully connected network before entering the actor and critic networks (Algorithm 1). Is this fully connected network also active for the "stack frames" / "stack latent" variants? If not, don't the FLARE actor/critic networks effectively have one more layer compared to the baseline variants? What happens when just z_t is passed to actor/critic, without f_FC and LayerNorm?


**Clarity:**
* The paper is well-written and easy to follow and sufficiently covers preliminary methods and related work. Figure 4 is very helpful to understand the proposed method fusioning information from multiple timesteps.

* The inputs to the "Stack SAC" baseline as denoted in the caption of Figure 3 does not align with Figure 4c. In the Stack SAC baseline, the current state and 3 differences comprise the input. In Figure 4, it is 3 states and 2 differences. Is this intended?

* The set of environments used for Fig. 2, 3 differs from the set of environments on which numerical results are reported (Table 2). Is there a particular reason for this?

* Checklist item 3d) is not fulfilled, I did not find a report on the type and amount of computational resources used.

* Questions:
  * l. 208: Why exactly do you detach z_{t-1} gradients?
  * l. 209: Why should a ReLU non-linearity not be used as activation function when negative input values possess semantic meaning?


**Significance:**
If further experiments and clarifications (see 'Weaknesses') support the claim that the proposed modifications to state-of-the-art model free RL algorithms substantially and consistently improve their performance this would be of high interest to the RL community.


**Conclusion:**
Although the methodological contribution is rather small it would be of interest for the RL community in my opinion, under the condition that the proposed modification substantially and consistently improves performance over the base variants RAD and RAINBOW. However, in my eyes, some experiments and clarifications are missing to support this claim. First, reporting (numerical) results on (Finger Spin, Cartpole Swingup, Reacher Easy, Cheetah Run, Walker Walk, Cup Catch) would greatly simplify comparison to other state-of-the-art RL algorithms. Second, I am concerned about the fairness of the comparison between baseline variants ("stack frames" / "stack latents") and FLARE (see 'Weaknesses'). For this reason, I do not see the paper ready for publication in NeurIPS yet.


**Minor comments:**
* In Table 1 in the appendix, "FLARE" is erroneously marked as bold for the "Berzerk" environment although Rainbow from "DQN Zoo" performs better
limitations_and_societal_impact: The authors sufficiently cover the limitations of their work. Societal impacts are not explicitly addressed. In my opinion, this work does not exhibit extraordinary societal impacts  in addition to those induced by basic reasearch on reinforcement learning methods in general.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a simple modiﬁcation to existing deep RL architectures when the input is a sequence of frames. It makes use of a specific encoding of the frames and a specific way of combining CNN and fusion of the information from the different frames. This architecture does not require any additional auxiliary tasks or supervision signals as compared to traditional RL losses. The results of the paper show that the developed approaches are significantly better than baselines.
main_review: The proposed technique is built on top of existing techniques such as Rainbow DQN and RAD and it achieves a gain in performance in many cases. The benchmarks considered are pixel-based continuous control tasks within the DeepMind control benchmark suite and Atari games.

The paper also provides different ablations, which contribute to understanding the importance of the different components of the proposed method.

This paper provides interesting insights about a late fusion architecture and a specific encoding in the context of RL from pixels. Even though the technique is quite simple and the improvements are in most cases not large (but still significant), the experiments and the ablations are conducted in good way with a few interesting baselines. The paper is also quite well-written and provides clear illustrations of the approach and of its results on the benchmarks.

Additional positive points: a source code is provided, the error bars are reported (except on Figure 6?).

Additional comment: 
- there are quite a few typos: "Notebly", "signifcant", "an SAC variant", "pipelinee"
- a few sentences at the beginning of section 3 that defines the setting formally with o_t, a_t, etc. could be useful.
limitations_and_societal_impact: The limitations and broader impact are discussed in a suitable way.
review_title: null

Meta-review: The paper proposes a simple but effective method to increase sample efficiency of RL in environments with pixel-based observations, by capturing temporal information via differences of consecutive image observation encodings. While, like framestacking, the method is expected to be needed and effective only in environments where capturing a non-trivial observation history is necessary for good performance and despite the performance advantage over alternatives being moderate, the experiments contain an informative ablation study that makes this work useful to other researchers from this field.
Decision: Accept (Poster)