Reinforcement Learning with Latent Flow
Authors: Wenling Shang
Xiaofei Wang
Aravind Srinivas
Aravind Rajeswaran
Yang Gao
Pieter Abbeel
Michael Laskin
Venue: NeurIPS 2021 Poster (2021)
Keywords: Machine Learning
Reinforcement Learning
Latent Flow

Abstract: Temporal information is  essential to learning effective policies with Reinforcement Learning (RL). However, current state-of-the-art RL algorithms either assume that such information is given as part of the state space or, when learning from pixels, use the simple heuristic of frame-stacking to implicitly capture temporal information present in the image observations. This heuristic is in contrast to the current paradigm in video classification architectures, which utilize explicit encodings of temporal information through methods such as optical flow and two-stream architectures to achieve state-of-the-art performance. Inspired by leading video classification architectures, we introduce the Flow of Latents for Reinforcement Learning (Flare), a network architecture for RL that explicitly encodes temporal information through latent vector differences. We show that Flare recovers optimal performance in state-based RL without explicit access to the state velocity, solely with positional state information. Flare is the most sample efficient model-free pixel-based RL algorithm on the DeepMind Control suite when evaluated on the 500k and 1M step benchmarks across 5 challenging control tasks, and, when used with Rainbow DQN, outperforms the competitive baseline on Atari games at 100M time step benchmark across 8 challenging games. 

Reinforcement Learning with Latent Flow
Wenling Shang"
Xiaofei Wang" UC Berkeley xf wwffberkeeee.eeee
DeepMind wendyshangkddeepmind.com com
Aravind Srinivas OpenAl aravind www.idddsrinnwwsseeereeelyy edu
Aravind Rajeswaran
Facebook Al Research, University of Washington aravraj@fb..
Yang Gao
Picter Abbeel UC Berkeley, Covariant pabbeel@berkeley, edu
Michael Laskin
Tsinghua University
UC Berkeley
gooyangiisstssinghua.. edu. en
mlaskii@@errkeley.coomm edu
Abstract
Temporal information is essential to learning effective policies with Reinforcement Learning (RL). However, current state-of-the-ar RL algorithms either assume that such information is given part of the state space or, when learning from pixels, use the simple heuristic of frame-stacking to implicitly capture temporal information present in the image observations. This heuristic is in contrast the current paradigm in video classification architectures, which utilize explicit encodings of temporal information through methods such optical flow and two-stream architectures to achieve state-of-the-art performance. Inspired by introduce the Flow of Latents for
leading video classification architectures, architecture for RL that explicitly Reinforcement Learning (Flare), a network encodes temporal information through latent vector differences. We show that Flare recovers optimal performance in state-based RL without explicit access to the state velocity, solely with positional state information. Flare the most sample efficient model-free pixel-based RL algorithm on the DeepMind Control suite when evaluated on the 500k and 1M step benchmarks across 5 challenging control tasks, and, when used with Rainbow DQN, outperforms the competitive baseline on Atari games 100M time step benchmark across 8 challenging games.
Introduction
Reinforcement learning (RI.) 41 holds the promise of enabling artificial agents to solve diverse set of tasks in uncertain and unstructured environments. Recent developments in RL with deep neural networks have led to tremendous advances in autonomous decision making. Notable examples include classical board games [36] 37). video games [29] 6 45), and continuous control [34] 28
There has been a large body of research on extracting high quality features during the RL process, such with auxiliary losses 220 27 35 data augmentation 225 26). However, another important component in RL representation learning has been largely overlooked: more effective architecture to incorporate temporal features. This becomes especially crucial in unstructured real-world setup like the home when compact state representations such as calibrated sensory inputs are unavailable. Motivated by this understanding, explore architectural improvements better utilize temporal features for the problem of efficient and effective deep RI. from pixels.
Equual contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Average DMControl
and Modian Human Nomallaed Score on
Eptrode Return
-00
HNS
200
12
IM Steps
500K Steps
FLARC
RAINDOW
DON
Figure 1: Mean and median evaluation scores on DMControl 42 and Atari B Flare is an architectural modification that improves RAD and Rainbow, the base algorithms integrates with.
Current approaches in deep RL, for learning temporal features are largely heuristic in nature. commonly employed approach is to stack the most recent frames [29] as inputs to a convolutional neural network (CNN). This can be interpreted as form of early fusion (2]. where information from the recent time window is combined at the pixel level for input to the CNN. In contrast, modern video recognition systems use alternate architectures that employ optical flow and late fusion [38], where frames are processed individually with CNN layers before fusion and downstream processing. Late fusion is typically beneficial due to better performance, fewer parameters, and the ability to use multi-modal data g 21.. However, it is not straightforward how to directly extend such architectures to RL. Real-time computation of optical flow for action selection can be computationally infeasible for many applications with fast control loops like robotics. Furthermore, optical flow computation at training time can also be prohibitively expensive. In our experiments, we also observe that a naive late fusion architecture minus the optical flow yields poor results in RL settings (see Section 6.3). This observation is consistent with recent findings in related domains like visual navigation [46]
To overcome the above challenges, we develop Flow of Latents for Reinforcement Learning (Flare), a new architecture for deep RI. from pixels (Figure 4). Flare can be interpreted as structured late fusion architecture. It processes each frame individually to compute latent vectors, similar to standard late fusion approach. Subsequently, temporal differences between the latent feature vectors are computed and fused along with the latent vectors by concatenation for downstream processing. By incorporating this structure temporal difference latent feature space, we provide the learning agent with appropriate inductive bias.
We highlight the main empirical contributions from Flare in the following
1. Flare recovers optimal performance in state-based RL without explicit access to the state velocity, solely with positional state information.
2. Flare achieves state-of-the-an performance compared to model-free methods on several challenging pixel-based continuous control tasks within the DeepMind control benchmark suite [42], while being the most sample efficient model-free pixel-based RL algorithm across these tasks, outperforming the prior model-free sate-of-the-ard RAD on the 500k and IM environment step benchmarks respectively (Figure [ A video demonstration of Flare achieving SOTA performance Quadruped Walk is in the supplementary materials.
3. When augmented over Rainbow DQN, Flare outperforms the baseline on 5 out of 8 chal- lenging Atari games at 100M step benchmark. Notebly, Flare scores 1668 on Montezuma's Revenge, signifcant gain over the baseline Rainbow DQN's 900.
2 Related Work
Pixel-Based RL The ability of an agent to autonomously learn control policies from visual inputs can greatly expand the applicability of deep RL [00 32]. Prior works have used CNNs to extend RL algorithms like PPO [34], SAC [14, and Rainbow [19] pixel-based tasks. Such direct extensions have typically required substantially larger number of environment interactions when compared to the state-based environments. In order to improve sample efficiency, recent efforts have studied the
use of auxiliary tasks and loss functions 50 27 35) data augmentation 26 25 and latent space
"ooee: https/gggtubb.co//Weenyhhhheeee

---

dynamics modeling [3]. Despite these advances, there is still a large gap between the learning efficiency in state-based and pixel-based environments number of challenging benchmark tasks. Our goal in this work is to identify where and how to improve pixel-based performance on this set of challenging control environments.
Neural Network Architectures in RL Mnih et al. [29] combined Q-learning with CNNs to achieve human level performance in Atari games, wherein Mnih et al. [29] concatenate the most recent 4 frames and use convolutional neural network to output the Q values. In 2016, Mnih et al. 30 proposed use a shared CNN among frames extract visual features and aggregate the temporal information with LSTM. The same architectures have been adopted by most works to date 27 35 25 26]. Recently, new architectures for RL have been explored that explore dense connections 39 as well as residual connections and instance norm [4)] However, the development stream of images has received little
of new architectures to better capture temporal information attention in deep RI., and our work fills this void. Perhaps the closest to our motivation is the work of Amiranashvili et al. [ who explicitly use optical flow as an extra input to the RL policy. However, this approach requires additional information and supervision signal to train the flow estimator, which could be unavailable or inaccurate in practice. In contrast, our approach is a simple modification to existing deep RL architectures and does not require any additional auxiliary tasks supervision signals.
Two-Stream Video Classification In video classification tasks, such as activity recogmition 40)] there are a large body of works on how to utilize temporal information 9 22 7
12 Of
particular relevance is the two-stream architecture of Simonyan and Zisserman
38 where one
CNN stream takes the usual RGB frames, while the other the optical flow computed from the RGB values. The features from both streams are then late-fused to predict the activity class. That the two-stream architecture yields a significant performance gain compared to the single RGB stream counterpart, indicating the explicit temporal information carried by the flow plays an essential role in video understanding. Instead of directly computing the optical flow, we propose to capture the motion information in latent space to avoid computational overheads and potential flow approximation errors. Our approach also could focus on Nomain-speciffe motions that might be overlooked in a generic optical flow representation.
3 Background
Soft Actor Critic (SAC) 14 is an off-policy actor-critic RL algorithm for continuous control with an entropy maximization term augmented to its score function to encourage exploration. SAC leams a policy network n. (a:jo;) and critic networks Qo (o,, as) and Qb, (o,, az) to estimate state-action values. The critic Qo, (o,, ax) is optimized to minimize the (soft) Bellman residual error:
(o( [Qo, (o,, ax) - (r+ + ()))++)))
where is the reward, y the discount factor, (or, de, 0,+1, r2) is transition sampled from replay buffer B, and ((0+1) is the (soft) target value estimated by:
((0441)=min min Q3, (0.41,a+41) - a log n. ((c+1
where a is the entropy maximization coefficient. For stability, Q3, the exponential moving average of Qo 's over training iterations. The policy T is trained to maximize the expected return estimated by Q together with the entropy term
(v) = Bar [min Qo, (Or, a.) - a log क. (arlo.),
where a is also learnable parameter.
Reinforcement Learning with Augmented Data, or RAD 226.. is recently proposed training technique. In short, RAD pre-pree prepprcessses raw pixel observations by applying random data augmenta- tions, such as random translation or cropping, for RL training. As simple as is, RAD has taken many existing RL algorithms, including SAC, to the next level. For example, on many DMCon- trol [42 benchmarks, while vanilla pixel-based SAC performs poorly, RAD-SAC-ie. applying data augmentation to pixel-based SAC achieees sate-of-the-art results both in sample efficiency and final performance, In this work, we refer RAD RAD-SAC and use random translation as data augmentation.
3

---

walk
Tapper
caripll swingup
3023
930
302
200
202
493
433
400
292
208
17
386
36c
Ervisement Dap
1ed
Iec
Fidi-ttde
Pastion-orly
Figure 2: Flare enables an RL agent with only access to positional state to recover a near-optimal policy relative to RL with access to the full state. In the above learning curves we show test-time performance for (i) full-state SAC (blue), where both pose and temporal information given (ii) position-only SAC (green), and (iii) state-hased Flare (orange), where only pose information is provided and velocities are approximated through pose offsets. Unlike full-state SAC, which learns the optimal policy, position-only SAC either fails or converges at suboptimal policies. Mcanwhile, the fusion of positions and approximated velocities in Flare efficiently recovers near-optimal policies in most cases. This motivates using Flare for pixel-based input, where velocities are not present in the observation. These results show mean performance with standard deviations averaged over 3 seeds.
Rainbow DQN is an extension of the Deep Q Network (DQN) (29], which combines multiple follow-up improvements of DQN to a single algorithm [19]. In summary. DQN [29] is an off-policy RL algorithm that leverages deep neural networks (DNN) to estimate the Q value directly from the pixel space. The follow-up works Rainbow DQN bring together to enhance the original DQN include double learning [17). prioritized experience replay (3), dueling network (48), noisy network [33,, distributional RL (5) and multi-step returns [41]. Rainbow DQN is one of the state-of- the-art RL algorithms on the Atari 2600 benchmark [3] We thus adopt an official implementation of Rainbow 31 as our baseline to directly augment Flare on top.
4 Motivation
We motivate Flare by investigating the importance of temporal information in state-based RL. Our investigation utilizes diverse DMControl 42 tasks. The full state for these environments includes both the agent's 's pose information, such as the joints' positions and angles, as well as temporal information, such as the joints' translational and angular velocities.
First, we train two variants with SAC-one where the agent receives the full state as input (full-state SAC), and the other with the temporal information masked out, i.e. the agent only receives the pose information as its input (position-on) only SAC). The resulting learning curves are in Figure 2 While the full-state SAC learns the optimal policy quickly, the position-only SAC learns much sub-optimal policies, which often fail entirely. Therefore, we conclude that effective policies cannot be learned from positions alone, and that temporal information is crucial for efficient learning.
While full-state SAC can receive velocity information from internal sensors in simulation, in the more general case such learning from pixels, such information is often not readily available. For this rea- son, we attempt to approximate temporal information as the difference between two consecutive states' positions. Concretely, we compute the positional offset 6,- " - -6-2- sg-3), and provide the fused vector (s"' 6.) to the SAC agent. This procedure describes the state-based version of Flare. Results shown in Figure 2 demonstrate that based Flare significantly outperforms the position-only SAC. Furthermore, achieves optimal asymptotic performance and a learning efficiency comparable to full-state SAC in most environments.
Given that the position-only SAC utilizes sf compared to Flare that utilizes sf and oG, we also investi-
gate a variant (stack SAC) where the SAC agent takes consecutive positions (s", s." 1...1... s. sya)). Stack SAC reflects the frame-stack heuristic used in pixel-based RL. Results in Figure 3 show that Flare still significantly outperforms stack SAC. It suggests that the well-structured inductive bias in the form of remporal-position fusion is essential for efficient learning.
Lastly, since a recurrent structure is an alternative approach to process temporal information, we im- plement an SAC variant with recurrent modules (Recurrent SAC) to compare with Flare. Specifically,

---

we pass a sequence of poses sy. 1 -. through an LSTM cell. The number of the LSTM hidden units h is set be the same as the dimension of ò, in Flare. The trainable parameters of the L.TTM cell are updated to minimize the critic loss. Recurrent SAC is more complex to implement and requires longer wall-clock training time, but performs worse than Flare as shown in Figure
Our findings from the state experiments in FigureBlnd Figure suggest that (i) temporal information crucial learning effective policies in RL, (ii) using Flare approximate temporal information in the absence of sensors that provide explicit measurements is sufficient in most cases, and (iii) to incorporate temporal information via naively staking position states or a recurrent module are less effective than Flare. In the next section, we carry over these insights to pixel-space RL.
cendulum swingup
QUAREE upod
hopper hop
cartzolo swingup
1000
.500
3023
300
3230
000
500
622
200
200
200
200
420
100
233
200
500
2-2
0.2
3.0
0.55
2.0
0.0
0.5
Envirammant
Flore
Recumeett
Stock SAC
Figure 3: We compare 3 ways to incorporate temporal information: i) Flare (orange) receives 2 -
sp-s), ii) stack SAC (green) stacks (sp, s Sp 26-2, sp-)) as inputs,
and iii) recurrent SAC (blue) uses recurrent layers to process (>) 18-1, 8 22--2---- 8 3). Stack SAC and recurrent SAC perform significantly worse than Flare on most environments, highlighting the benefit of how Flare handles temporal information. Results are averaged over 3 seeds.
5 Reinforcement Learning with Latent Flow
To date, frame stacking is the most common way of pre-processing pixel-based input to convey temporal information for RL algorithms. This heuristic, introduced by Mnih al. [2]]. has been largely untouched since its inception and is used in most state-of-the art RL architectures. However, our observations from the experiments run on state inputs in Section suggest an alternative to the frame stacking heuristic through the explicit inclusion of temporal information as part of the input. Following this insight, we seek a general alternative approach to explicitly incorporate temporal information that can be coupled to any base RI. algorithm with minimal modification. To this end, we propose the Flow of Latents for Reinforcement Learning (Flare) architecture. Our proposed method calculates differences between the latent encodings of individual frames and fuses the feature differences and latent embeddings before passing them as input to the base RL algorithm, as shown in Figure 4 We demonstrate Flare on top of 2 state-of-the- thefthth-artt art model-free off-policy RL baselines, RAD-SAC 26 and Rainbow DQN 19], though principle any RL algorithm can be used in principle.
Vodaaa
Laae:
Laker Dilvanses
Gubenctice
5 3-11
Frame stacking heuresic
Indivicuall
Row of Latents for (Fars)
Figure 4: Flow of Latents for Reinforcement Learning (Flare): (a) the architecture for the frame stacking heuristic, (b) an alternative to the frame stacking hueristic by encoding each image indi- vidually, and (c) the Flare architecture which encodes images individually, computes the feature differences, and fuses the differences together with the latents.
5

---

5.1 Latent Flow
In computer vision, the most common way to explicitly inject temporal information of video sequence is to compute dense optical flow between consecutive frames [38]. Then the RGB and the optical flow inputs are individually fed into two streams of encoders and the features from both are fused in the later stage of the pipelinee. But two-stream architectures with optical flow are not as applicable to RL, because is too computationally costly to generate optical flow on the fly.
To address this challenge and motivated by experiments in Section we propose an alternative architecture that is similar in spirit to the two-stream networks for video classification. Rather than computing optical flow directly, we approximate temporal information in the latent space. Instead of encoding stack of frames at once, we use frame-wise CNN to encode each individual frame. Then we compute the differences between the latent encodings of consecutive frames, which we refer to as latent flow. Finally, the latent features and the latent flow are fused together through concatenation before getting passed to the downstream RL algorithm. We call the proposed architecture as Flow of Latents for Reinforcement Learning (Flare).
5.2 Implementation and Architecture Details
We select RAD the base algorithm to elab- orate the execution of Flare. Similar adapta- tions can be seamlessly applied to other RI. al- gorithms such as Rainbow DQN. The RAD ar- chitecture, shown in Figure Ap. stacks multiple data augmented frames observed in the pixel space and encodes them altogether through an CNN. This can be viewed as form of early fusion (4
Algorithm Pixel-based Flare Inference
Given Tyl foNN
for each environment step t do
2j fcNN
k,..,
= -==zz=z-11--3 - k+1,
2. = = (z=-k11..., 2,, ,
2 = LaverNorm
Another preprocessing option to encode each frame individually through shared frame-wise encoder and perform late fusion the resulting latent features, shown in Figure 4b. However, we find that simply concatenating the latent fea- tures results inferior performance when compared the frame stacking heuristic, which we further elaborate in Section 6.3 We conjecture that pixel-level frame stacking benefits from leveraging both the CNN and the fully connected layers process temporal information, whereas latent-level stacking does not propagate temporal information back through the CNN encoder.
a, (
0 p 0r or
(or, (0: 0-1-000---1101
Based on this conjecture, explicitly compute the latent flow 6, 2, 2t-1 while detaching the 2t-1 gradients when computing dz. We then fuse together (&e, 2,). Next, since negative values in the fused latent embedding now possesses semantic meaning from 8,, instead of ReLJU non-linearity, pass the embedding through fully-connected layer followed by layer normalization, before entering the actor and critic networks as shown in Figure 4p. Pseudocode illustrates inference with Flare in Algorithm 5.2 during training, the encodings of latent features and flow are done in the same way except with augmented observations.
Experiments
In this section, we first present the main experimental results, where we show that Flare achieves substantial performance gains over the base algorithm RAD (26) and Rainbow DQN [19]. Then we conduct a series of ablation studies stress test the design choices of the Flare architecture.
6.1 Environments and Evaluation Metrics
The Med Control Suite (DMControl) [2), based on MuloCo [.) is commonly used bench- mark for continuous control from pixels. On simpler environments in the suite, prior works [2] 26 have made substantial progress on this benchmark and closed the gap between state-based and pixel-based efficiency. However, on more challenging environments that feature partial observability, sparse rewards, or precise manipulation, these algorithms struggle to learn optimal policies efficiently In this work, focus on 5 of these more challenging tasks. The 5 environments include Walker

---

Table 1: Mean returns and standard errors on challenging DMControl tasks evaluated at 500K and 1M environment steps over random seeds and 10 trajectories per seed. Flare substantially outperforms RAD on a majority (3 out of the 5) of environments, while remaining competitive in the remaining ones and achieving substantially higher aggregate scores.
IM STEPS
500K STEPS
TASK
FLARE
RAD
CURL
FLARE
RAD
CURL
QUADRUPED WALK
99
38 +
296 62
206 + 50
39
PENDULUM SWINGUP
809 = 14
520 144
151 151::8
242 = 68
79 / 33
46 207
HOPPER HOP
217 + 26
211 + 12
44
90 - 25
40 18
10 + 17
FINGER TURN HARD
661 = 141
249 +
222 + 14
282 30
137 44
207 + 32
WALKER RUN
556
628 +
323 + 35
426 +
547 +1
245 + 32
AVERAGE RETURN
546
388
156
267
202
109
Due to large computational requirements for 100M Atari runs, we randomly select 8 Atari games for evaluation. We random seeds for both Flare and Rainbow DQN [1]] evaluate scores at 100m training steps, and show the mean and standard error. Flare improves the Rainbow DQN in most games and achieves a subtantially higher mean and median human normalized scores (HNS). refers to a comparison being made between Flare and Flare's 's base algorithm Rainbow. Reference values for DQN, Random, and Human baselines are taken from Bellemare et
Table 2:
TASK
FLARE
RAINBOW
DQN
RANDOM
HUMAN
DEFENDER
83062+13065
44694=1782
23633
2874.5
18688.9
PHOENIX
60974+8870
16992-1474
8485.2
761.4
7242.6
BERZERK
2049-118
163611267
585.6
123.7
2630.4
MONTEZUMA
166811472
9001116
0
0
4753.3
ASSAULT
12724_1221
1522911611
4280.5
222.4
742
BREAKOUT
345 100
280118
385.5
1.7
30.5
SEAQUEST
1390111361
24090-15579
5860.6
68.4
42054.7
TUTANKHAM
248.19
247.55
68.1
11.4
167.6
MEDIAN HNS
3.4
2.0
0.8
0.0
1.0
AVERAGE HNS
6.7
5.8
3.0
0.0
1.0
Run (requires maintaining balance with speed), Quadruped Walk (partially observable agent morphol- ogy), Hopper Hop (locomotion with sparse rewards), Finger Turn-hard (precise manipulation), and Pendulum Swingup (torque control with sparse rewards). For evaluation, we benchmark performance at 500K and 1M environment steps and compare against RAD.
The Atari 2600 Games 3 is another highly popular RL benchmark. Recent efforts have led to solve Atari games directly from pixel. A representative state-of-the-an is Rainbow DQN (see Section 3). We adopt the official Rainbow DQN implementation (31 as our baseline and simply incorporate Flare while retaining all the other default settings, including hyperparameters and preprocessing. Note that the baseline Rainbow DQN's model architecture is also modified to the most comparable setup to that Flare as described in Section 5.2 including increasing the number of last layer convolutional channels (to match the number of parameters) and adding a fully-connected layer plus layer normalization before the Q networks. We evaluate Breakout, Freeway, Krull, Montezuma Revenge, Seaquest, Up n Down and Tutankham, to assess the effectiveness of Flare.
range of highly successful algorithms [I] 19 23
diverse subset of Atari games at 100M training steps, namely Assault,
common symptom in RL that evaluations appear noisy. To ensure the most fair presentation of results, we follow this protocol: the main results in Section 6.2 report the mean over random seeds with standard error, standard RI. practice 42 3T.
Evaluation and Training Protocol
Furthermore,
use the same 5 seeds for both the baseline and Flare.

---

6.2
Main Results
We present the main results of comparing Flare against the baselines, namely RAD and Rainbow DQN. Flare outperforms the baselines on the majority of the environments. It is worth noting that since these baselines already produce state-of-the ff-th-eartt art level performances, any steady improvement under our rigorous experimental protocol-eve even when the gain seems minor -is significant.
DMControl:
Our main experimental results on the DMControl tasks are presented in Table [ We find that Flare outperforms RAD in terms of both final performance and sample efficiency for majority (3 out of 5) of the environments, while being competitive on the remaining environments. Specifically, Flare attains similar asymptotic performance to state-based RL on Pendulum Swingup, Hopper Hop, and Finger Turn-hare For Quadruped Walk, a particularly challenging environment due to its large action space and partial observability, Flare learns much more efficiently than RAD and achieves a higher final score. Moreover, Flare outperforms RAD in terms of sample efficiency on all of the core tasks except for Walker Run. The 500k and 1M environment step evaluations in Table [ show that, on average, Flare achieves 1.4x and 1.3x higher scores than RAD at the 1M step and the 500K step benchmarks, respectively.
The results on the Atari games are in Table 2 Again, we observe substantial performance gain from Flare on the majority (5 out of 8) of the games, including the challenging Montezuma's Revenge. On most of the remaining games, Flare is equally competitive except for Seaquest. In the appendix, we also show that Flare performs competitively when comparing against other DQN variants at 100M training steps, including the original Rainbow implementations.
Atari:
6.3 Ablation Studies
We ablate number of components of the Flare architecture on the Quadruped Walk and Pendulum Swingup environments to stress test the Flare architecture. The results shown in Figure 5 aim to answer the following questions:
Q1: Do we need latent flow or is computing pixel differences sufficient?
A1: Flare proposes late fusion of latent differences with the latent embeddings, while a simpler approach is an early fusion of pixel differences with the pixel input, which we call pixel flow. We compare Flare to pixel flow in Figure 3 (left) and find that pixel flow is above RAD but significantly less efficient and less stable than Flare, particularly on Quadruped Walk. This ablation suggests that late fusion temporal information after encoding the image is preferred early fusion.
Q2: Are the gains coming from latent flow or individual frame-wise encoding?
A2: We address the potential concern that the performance gain of Flare stems from the frame-wise ConvNet architectural modification instead of the fusion of latent flow. Concretely, we follow the exact architecture and training Flare, but instead of concatenating the latent flow, concatenate each frame's 's latent vector after the convolution encoders directly as described in Figure 4b. This ablation is similar in spirit to the state-based experiments in Figure 3 The learning curves in Figure D (center) show that individual frame- wise encoding is not the source of the performance lift: frame- wise encoding, though par with RAD Pendulum Swingup, performs significantly worse on Quadruped Walk. Flare's improvements over RAD are hence most likely thanks to the explicit fusion of latent flow.
Q3: How does the input frame count affect performance?
A3: We compare stacking 2, 3, and 5 frames in Flare in Figure 5 (right). (right.. We find that changing the number of stacked frames does not significantly impact the locomotion task, quadruped walk, but Pendulum Swingup tends to be more sensitive to this hyperparameter. Interestingly, the optimal num- ber of frames for Pendulum Swingup is 2, and more frames can in fact degrade Flare's performance, indicating the immediate position and velocity information is the most critical to learn effective policies on this task. We hypothesize that Flare trains more slowly with increased frame count on Pendulum Swingup due to the presence of unnecessary information that the actor and critic networks need to learn to ignore.
Q4: Do we need latent flow or is RNN over latent sufficient?
A4: Another approach of latent fusion would be applying recurrent neural network on the latent embeddings. We compare FLARE with LSTM basclines DMControl. We found that RNNs
8

---

Perdolum Swingvp
1000
povel flw vs Iatent flww
RAD
1000
with
frame stack (RAD)
Flarc 2.3,5
frames
8000
latent (Flere)
ECO
latentt stack Row (Flere)
600
(pacde
poool flow
frames
000
Ialent stack only
200
1.5
0.0
6-5
1.3
3.5
140
106
1000
2000
Quedruped, Walk
800
600
0.7
0.5
3.0
2.5
2.0
Envtronment
306
306
(a) pixal flow ablation
(b) latent stacking ablation
(c) frame count ablation
Figure 5: We perform ablation studies: (a) Pixel flow ablation: we compare using pixel-level and latent-level (Flare) differences. Flare is more stable and performs better. (b) Latent stack ablation: we compare using latent stack with and without the latent flow. The latter performs significantly worse, suggesting that the latent flow is crucial. (c) Frames count ablation: we test using different number of frames for Flare.
perform worse than FLARE (see Fig 6. which is in agreement with our findings over the poor performance of RNN on coordinate (i.e. compact) state representations in Section
Broader Impacts and Limitations
Conclusion We propose Flare, an architecture for RL. that explicitly encode temporal information by computing flow in the latent space. In experiments, we show that in the state space, Flare can recover the optimal perfor- mance with only state positions and no access to the state velocities. In the pixel space, Flare improves upon the state-of-the-an model-free RL algorithms on the majority of selected tasks in the DMControl and Atari suites, while matching in the remaining. All code assets used for this project came with MIT licenses. For more details, refer the reader to the appendix.
FLARE RAD vs RNN RAD or IM steps
000
700
600
5c0
420
200
200
120
Penoudem
Walker quadripe
Swinggp Hun!
Figure 6: The bars show the difference in episode return between FLARE RAD and RNN at IM env steps. All experi- ments are run on a fixed seed. FLARE outperform the RNN baseline in all of the.experimeens
Limitations Flare is a general approach to improve RI. algorithms on many environments and tasks but not the panacea to handelyy solve all. For instance, learn- ing to control humanoid from pixels remains a challenge even when augmenting RAD with Flare. Also, RAD without www.e fact is preferred for Seaquest, one of the Atari games. An additional limitation is that Flare is only useful if temporal information such as velocities is visible in the input pixel images and may not work as well for partially observed environments. Finally, Flare only tested on model-free algorithms and it would be informative to investigate its upplicability in the model- based regime, which we leave for future work.
Broader Impacts Simple architectures that improve performance can be impactful due to their ease of use and wide applicability. Prime examples of such innovations in vision are ResNets [18] which have been widely adopted. potential negative consequence Flare and supervised RL algorithms in general is that they rely on hand-designed reward functions which can be exploited. For example, in the Quadruped task Flare learns a more optimal policy than prior model-free methods but the resulting policy is extremely jittery since the simulator does not penalize jerk. This policy

---

would likely wear out the joints if deployed on real robot. To make architectures like Flare and RL in general more applicable to real-world scenarios it would be useful to investigate how to generate safe policies perhaps through constrained optimization or offline RI..
References
[1] Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, and Thomas Brox. Motion perception in reinforcement learning with dynamic objects. In Conference on Robot Learning, pages 156-168. PMLR, 2018.
[2] Adrià Puigdomenech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark, In International Conference on Machine Learning. 2020.
[3] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[4] Marc G. Bellemare, Will Dabney, and R. Munos. A distributional perspective reinforcement learning. In ICMI., 2017.
[5] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforce- ment learning. arXiv preprint 1.07:0777..6888 2017.
[6] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota with large scale deep reinforcement learning. arXiv preprint arXiv: /9tiv::///..0 19/2.06680 2019.
[7] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[8] Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, S. Schaal, and S. Levine Path integral guided policy search. 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 338 3381-3388, 2017.
[9] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2625-2634, 2015.
[10] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. arXiv preprint arXiv:1711.039%3 2017.
[11] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et Impala: Scalable distributed deep-rl with importance weighted actor-leamer architectures. arXiv preprint arXiv: 1802.01561, 201 8.
[12] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE international conference on computer vision, pages 6202-6211, 2019.. 2019.
[13] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, al. Noisy networks for exploration. arXiv preprint arXiv: 10295, 2017.
[14] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arxiv 2018.
[15] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1xXiv:1912.0166 2019.
10

---

[16] Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555-2565. PMLR, 2019.
[17] Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23: 2613-2621, 2010.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2.16. doi: 10.1 19110099..2206..00
Utt.ttps::////// h org/ 10. 1109/CVPR..22 2016//.....22......... 90
[19] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv:1710.02290 2017.
[20] Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.00 105397, 2016.
[21] Divye Jain, Andrew Li, Shivam Singhal, Aravind Rajeswaran, Vikash Kumar, and Emanuel Todorov. Learning Deep Visuomotor Policies for Dexterous Hand Manipulation. In International Conference on Robotics and Automation (ICRA), 2019.
[22] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1): 221-231, 2012.
[23] Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney, Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
[24] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages
2014.
[25] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforeement learning from pixels. arXiv preprint arXiv:2004.13649, 2020.
[26] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. 14990, 2020.
[27] Michael* Laskin, Aravind* Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. Proceedings of the 37th International Conference on Machine Learning. Vienna, Austria, PMLR 119, 2020. arXiv:2004.04136.
[28) Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
[29] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5189753 2015.
[30] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lilli- crap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937, 2016.
[31] John Quan and Georg Ostrovkki. DQN Zoo: Reference implementations of DQN- -ased agents, 2020. CRRLhttp:/ //gi thub. cow wwtmm/deepminddddg..aa//

---

[32] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE International Conference Computer Vision, pages 9239-9347,229 2019.
[33] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint
2015.
[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:170700644 2017.
[35] Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with momentum predictive representations. arXiv preprint arXiv:2007.05999 2020.
[36] David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvan, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lil- licrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-94444 503, 2016. URL / 16961. .tml
http://wwe //www//wwww. nature. wwwpe//teeoeann/noooooaaaa
[37] David Silver, Julian Schrittiesser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai,, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-359, 10 2017. doi: 10. 12000//a
[38] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pages 568-576, 2014.
[39] Samarth Sinha, Homanga Bharadhwaj. Aravind Srinivas, and Animesh Garg. D2rl: Deep dense architectures in reinforcement learning. 2020.
[40] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv: 1212.000, 2012.
[41] Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
[42] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 1001.08801100660 2018.
[43] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In IROS, 2012.
[44] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages
2015.
[45] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforce- ment learning. Nature, 37507782):550-354, 2019. doi: 10. 586-019--11111 1724-2.. URI. https://doi. org/ 10. /
12

---

[46] Aaron Walsman, Yonatan Bisk, Saadia Gabriel, Dipendra Misra, Yoav Artzi, Yejin Choi, and Dieter Fox. Early Fusion for Goal Directed Robotic Vision. In International Conference on Intelligent Robots and Systems (IROS), 2019.
[47] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018.
[48] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003, 2016.
[49] Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression, 2020.
[50] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint 1rX14:1910001741, 2019.
13


Reviews:
[{"review_id": "TllFIqXx6j", "reviewer": "Reviewer_oLLK", "summary": "In this paper, the authors modify two state-of-the-art reinforcement learning algorithms (RAD-SAC and Rainbow-DQN) by changing the way information from multiple timesteps is aggregated. Instead of stacking frames at a pixel level, the authors propose to encode each frame individually and then pass the encodings and their pairwise differences to the actor/critic networks. It is shown that when learning policies from pixels this way of fusing information substantially outperforms the baseline variants in 3/5 environments from the DeepMind control suite and 5/8 environments from Atari2600. Furthermore, experiments with low-level observations on 5 DeepMind Control Suite tasks support the particular choice of information fusion over using recurrent networks or stacking states directly without including state differences.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:**\nThe proposed way to fuse information from multiple frames / timesteps is simple but novel. It questions the \"standard way\" of stacking frames to incorporate information from multiple timesteps into the RL algorithm, which is interesting.\n\n\n**Quality:**\n\n* The authors perform reasonable experiments to evaluate the advantage of their proposed method for low-dimensional and high-dimensional observations\n\n* The motivation for selecting the subset of environments reported in Table 1 is unclear to me. In other work such as CURL, Dreamer and PlaNet, different environments and timesteps were selected (namely Finger Spin, Cartpole Swingup, Reacher Easy, Cheetah Run, Walker Walk, Cup Catch) at 100k steps / 500k steps. Evaluating FLARE on these environments and timestep limits would greatly simplify comparison with the above methods.\n  \n* I am concerned with the fairness of comparing the \"stack frames\" and \"stack latents\" variants to FLARE. In FLARE the \"fused\" (concatenated) vector of latents and differences is passed through a fully connected network before entering the actor and critic networks (Algorithm 1). Is this fully connected network also active for the \"stack frames\" / \"stack latent\" variants? If not, don't the FLARE actor/critic networks effectively have one more layer compared to the baseline variants? What happens when just z_t is passed to actor/critic, without f_FC and LayerNorm?\n\n\n**Clarity:**\n* The paper is well-written and easy to follow and sufficiently covers preliminary methods and related work. Figure 4 is very helpful to understand the proposed method fusioning information from multiple timesteps.\n\n* The inputs to the \"Stack SAC\" baseline as denoted in the caption of Figure 3 does not align with Figure 4c. In the Stack SAC baseline, the current state and 3 differences comprise the input. In Figure 4, it is 3 states and 2 differences. Is this intended?\n\n* The set of environments used for Fig. 2, 3 differs from the set of environments on which numerical results are reported (Table 2). Is there a particular reason for this?\n\n* Checklist item 3d) is not fulfilled, I did not find a report on the type and amount of computational resources used.\n\n* Questions:\n  * l. 208: Why exactly do you detach z_{t-1} gradients?\n  * l. 209: Why should a ReLU non-linearity not be used as activation function when negative input values possess semantic meaning?\n\n\n**Significance:**\nIf further experiments and clarifications (see 'Weaknesses') support the claim that the proposed modifications to state-of-the-art model free RL algorithms substantially and consistently improve their performance this would be of high interest to the RL community.\n\n\n**Conclusion:**\nAlthough the methodological contribution is rather small it would be of interest for the RL community in my opinion, under the condition that the proposed modification substantially and consistently improves performance over the base variants RAD and RAINBOW. However, in my eyes, some experiments and clarifications are missing to support this claim. First, reporting (numerical) results on (Finger Spin, Cartpole Swingup, Reacher Easy, Cheetah Run, Walker Walk, Cup Catch) would greatly simplify comparison to other state-of-the-art RL algorithms. Second, I am concerned about the fairness of the comparison between baseline variants (\"stack frames\" / \"stack latents\") and FLARE (see 'Weaknesses'). For this reason, I do not see the paper ready for publication in NeurIPS yet.\n\n\n**Minor comments:**\n* In Table 1 in the appendix, \"FLARE\" is erroneously marked as bold for the \"Berzerk\" environment although Rainbow from \"DQN Zoo\" performs better", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors sufficiently cover the limitations of their work. Societal impacts are not explicitly addressed. In my opinion, this work does not exhibit extraordinary societal impacts  in addition to those induced by basic reasearch on reinforcement learning methods in general.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "GkpaYsWaPUq", "reviewer": "Reviewer_odFP", "summary": "The paper proposes a simple modiﬁcation to existing deep RL architectures when the input is a sequence of frames. It makes use of a specific encoding of the frames and a specific way of combining CNN and fusion of the information from the different frames. This architecture does not require any additional auxiliary tasks or supervision signals as compared to traditional RL losses. The results of the paper show that the developed approaches are significantly better than baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed technique is built on top of existing techniques such as Rainbow DQN and RAD and it achieves a gain in performance in many cases. The benchmarks considered are pixel-based continuous control tasks within the DeepMind control benchmark suite and Atari games.\n\nThe paper also provides different ablations, which contribute to understanding the importance of the different components of the proposed method.\n\nThis paper provides interesting insights about a late fusion architecture and a specific encoding in the context of RL from pixels. Even though the technique is quite simple and the improvements are in most cases not large (but still significant), the experiments and the ablations are conducted in good way with a few interesting baselines. The paper is also quite well-written and provides clear illustrations of the approach and of its results on the benchmarks.\n\nAdditional positive points: a source code is provided, the error bars are reported (except on Figure 6?).\n\nAdditional comment: \n- there are quite a few typos: \"Notebly\", \"signifcant\", \"an SAC variant\", \"pipelinee\"\n- a few sentences at the beginning of section 3 that defines the setting formally with o_t, a_t, etc. could be useful.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and broader impact are discussed in a suitable way.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "9Tipp1_cfUw", "reviewer": "Reviewer_FyFS", "summary": "The paper suggests using an optical flow to encode pixel observations in contrast to simple concatenation over the temporal dimension. This proposed method enables efficient recovery of dynamics information such as velocity and acceleration, this in turn, improves performance on several image-based tasks from DMControl and Atari.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: I think the paper is limited in its originality. While the particular application of an optimal flow like architecture to this type of setup is novel, it still a rather incremental derivation from prior work (both in CV and RL). This should not be mistaken with the simplicity aspect of Flare, which I like. \n\n\nQuality: The paper is well executed. The experimentation is sound and extensive. I like the fact that the authors consider several different domain (DMControl and Atari) and several different algorithm, which makes it easier to justify their method.\n\nClarity:\nThe paper is very well written and easy to follow. I like the motivation example and figures 2 and 3. I also like how the ablation section is structured as a Q&A, which makes it easy to comprehend.\n\nSignificance:\nI believe the significance of this paper is limited. While it proposes a better way to deal with encoding temporal information in image-based RL, it feels more like a hack, rather than a systematic solution. For example, it is not hard to imagine scenarios where it is not enough to extract linear relationships between consecutive frames. In these cases more general architectures such as RNNs or Transformers will be still required.\n\nOverall:\nI think it is a well written and executed paper that demonstrates some interesting findings. I also like the fact that the authors attempt to investigate alternative architectures in RL. However, the novelty and significance aspects are limited. I'm thus borderline on this paper and biasing towards rejection.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No issues", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "8nV2hmAtiS", "reviewer": "Reviewer_NvL5", "summary": "This paper proposes adding temporal information in the form of latent flow to deep RL agents. The idea is simple: temporal differences between latent embeddings of consecutive frames are computed; the differences are then concatenated to the embeddings for downstream processing by an RL controller. This idea is inspired by idea of late fusion in video processing and stands in contrast to the current approach of frame-stacking, which is more akin to early fusion. The authors have validated their approach on a series of DMC tasks as well as Atari games.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* the paper is well-written and the ideas clearly laid out\n* the method is easily applicable to many different RL architectures\n* empirical results show a clear improvement over baselines (including RNN baselines) on a selection of DMC and Atari environments\n* informative ablation studies comparing frame-stacking, temporal pixel differences, and pure latent stacking\n\nWeaknesses:\n* number of frames to include adds an extra hyperparameter that needs to be tuned. The appropriate number will probably be task-dependent. The authors have demonstrated one case where extra frames do nothing and another case where it actually hurts (presumably because the agent must learn to ignore the superfluous information). However, one can conceivably imagine cases where having extra frames is necessary (e.g. if accelerations are important, then using two frames won't be enough).\n\nMinor typos:\n* line 60: “Notably” instead of “Notebly”\n* line 212: “Algorithm 1” instead of “Algorithm 5.2”\n* line 238: different set of games listed here compared to the table\n\nOverall, while the idea is simple, I believe the authors have demonstrated the method does benefit RL algorithms. I am also satisfied that the authors have addressed concerns raised in their previous submission regarding a missing RNN baseline. I recommend accepting this paper.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have given thoughtful consideration to the limitations of their work, including instances where their method may not work. I would have also appreciated some discussion of needing to tune the number of frames as a hyperparameter. The authors have also discussed the broader impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a simple but effective method to increase sample efficiency of RL in environments with pixel-based observations, by capturing temporal information via differences of consecutive image observation encodings. While, like framestacking, the method is expected to be needed and effective only in environments where capturing a non-trivial observation history is necessary for good performance and despite the performance advantage over alternatives being moderate, the experiments contain an informative ablation study that makes this work useful to other researchers from this field.
Decision: Accept (Poster)