Reinforcement Learning with Latent Flow
Authors: Wenling Shang
Xiaofei Wang
Aravind Rajeswaran
Aravind Srinivas
Yang Gao
Pieter Abbeel
Michael Laskin
Venue:  (2021)
Keywords: reinforcement learning
deep learning
machine learning
deep reinforcement learning

Abstract: Temporal information is essential to learning effective policies with Reinforcement Learning (RL). However, current state-of-the-art RL algorithms either assume that such information is given as part of the state space or, when learning from pixels, use the simple heuristic of frame-stacking to implicitly capture temporal information present in the image observations. This heuristic is in contrast to the current paradigm in video classification architectures, which utilize explicit encodings of temporal information through methods such as optical flow and two-stream architectures to achieve state-of-the-art performance. Inspired by leading video classification architectures, we introduce the Flow of Latents for Reinforcement Learning Flare, a network architecture for RL that explicitly encodes temporal information through latent vector differences. We show that Flare (i) recovers optimal performance in state-based RL without explicit access to the state velocity, solely with positional state information, (ii) achieves state-of-the-art performance on pixel-based continuous control tasks within the DeepMind control benchmark suite, (iii) is the most sample efficient model-free pixel-based RL algorithm on challenging environments in the DeepMind control suite such as quadruped walk, hopper hop, finger turn hard, pendulum swing, and walker run, outperforming the prior model-free state-of-the-art by 1.9 and 1.5 on the 500k and 1M step benchmarks, respectively, and (iv), when augmented over rainbow DQN, outperforms or matches the baseline on a diversity of challenging Atari games at 50M time step benchmark.

Under review as a conference paper at ICLR 2021
REINFORCEMENT LEARNING WITH LATENT FLOW
Anonymous authors
Paper under double-blind review
ABSTRACT
Temporal information essential to learning effective policies with Reinforcement Learning (RL). However, current state-of-the-art RL algorithms either assume that such information given as part of the state space or, when learning from pixels, use the simple heuristic of frame-stacking to implicitly capture temporal information present in the image observations. This heuristic is in contrast to the current paradigm in video classification architectures, which utilize explicit encodings of temporal information through methods such as optical flow and two-stream architectures to achieve state-of-the-ar performance. Inspired by leading video classification architectures, we introduce the Flow of Latents for Reinforcement Learning (Flare), a network architecture for RL that explicitly encodes temporal information through latent vector differences. We show that Flare (i) recovers optimal performance state-based RL without explicit access to the state velocity, solely with positional state information, (ii) achieves state-of- the-art performance on pixel- base- continuous control tasks within the Deep.innd control benchmark suite, (iii) is the most sample efficient model-free pixel-based RI. algorithm on challenging environments in the DeepMind control suite such as quadruped walk, hopper hop, finger turn hard, pendulum swing, and walker run, outperforming the prior model-free state-of-the-an by 1.9x and 1.5x on the 500k and IM step benchmarks, respectively, and (iv), when augmented over rainbow DQN, outperforms or matches the baseline on diversity of challenging Atari games 50M time step benchmark.
INTRODUCTION
Reinforcement learning (RL) (Sutton & Barto, 1998) holds the promise of enabling artificial agents to solve a diverse set of tasks uncertain and unstructured environments. Recent developments in RL with deep neural networks have led to tremendous advances in autonomous decision making. Notable examples include classical board games (Silver al., 2016; 2017), video games (Mnih al., 2015; Berner et al., 2019; Vinyals et al., 2019), and continuous control (Schulman et al., 2017; Lillicrap et al., 2016; Rajeswaran et al., 2018). A large body research has focused on the case where an RL agent is equipped with compact state representation. Such compact state representations typically available in simulation (Todorov et al., 2012: Tassa et al., 2018) or in laboratories equipped with elaborate motion capture systems (OpenAl et al., 2018; Zhu et al., 2019; Lowrey et al., 201 8). However, state representations are seldom available in unstructured real-world settings like the home. For RL agents be truly autonomous and widely applicable, sample efficiency and the ability act using raw sensory observations like pixels is crucial. Motivated by this understanding, we study the problem of efficient and effective deep RL from pixels.
A number of recent works have made progress towards closing the sample-efficinncy and performance gap between deep RL from states and pixels (Laskin et al., 2020b;a: Hafner al., 201 Kostrikov et al., 2020). An important component in this endeavor has been the extraction of high quality visual features during the RL process. Laskin et (2020a) and Stooke et al. (2020) have shown that features learned either explicitly with auxiliary losses (reconstruction or contrastive losses) or implicitly (through data augmentation) are sufficiently informative to recover the agent's pose information. While existing methods can encode positional information from images, there has been little attention devoted to extracting temporal information from stream of images. As a result, existing deep RL methods from pixels struggle to learn effective policies more challenging continuous control environments that deal with partial observability, sparse rewards, or those that require precise manipulation.

---

Under review as a conference paper at ICLR 2021
Current approaches in deep RL for learning tem- poral features are largely heuristic in nature. A commonly employed approach is to stack the most recent frames as inputs to a convolutional neural network (CNN). This can be viewed as a form of early fusion (Karpathy et al., 2014), where information from the recent time window is combined immediately at the pixel level for in- put to the CNN. In contrast, modern video recog- nition systems use alternate architectures that employ optical flow and late fusion (Simonyan & Zisserman, 2014), where frames are processed individually with CNN layers before fusion and downstream processing. Such late fusion ap- proach is typically beneficial due to better per- formance, fewer parameters, and the ability to use multi-modal data (Jain et al., 2019; Chebotar straightforward
1-1
1-2
Latent
Latent Differences
Subtraction
Vectors
Figure 1: Flow of Latents for Reinforcement Learning (Flare) architecture. Input frames are first encoded individually by the same encoder. The resulting latent vectors are then concatenated with their latent differences before being passed to the downstream RL algorithm.
al., 2017). However, it how to port such architectures to RL. Comput- ing optical flow in real- time for action selection can be computationally infeasible in applications with fast control loops like robotics. In our experiments, we also find that a naive late fusion architecture minus the optical flow yields poor results in RL settings (see Section 6.3). This observation is consistent with recent findings in related domains like visual navigation (Walsman et al., 2019).
To overcome the above challenges, we develop Flow of Latents for Reinforcement Learning (Flare), a new architecture for deep RL from pixels (Figure 1). Flare can be interpreted as a structured late fusion architecture. Flare processes each frame individually to compute latent vectors, similar to standard late fusion approach (see Figure 1). Subsequently, temporal differences between the latent feature vectors are computed and fused along with the latent vectors by concatenation for downstream processing. By incorporating this structure of temporal difference in latent feature space, we provide the learning agent with appropriate inductive bias. In experiments, we show that Flare (i) recovers optimal performance in state-based RL without explicit access to the state velocity, solely with positional state information, (ii) achieves state-of-the-art performance compared to model-free methods on several challenging pixel-based continuous control tasks within the DeepMind control benchmark suite, namely Quadruped Walk, Hopper Hop, Finger Tum-hard, Pendulum Swingup. and Walker Run, and (iii) is the most sample efficient model-free pixel-basee based RL algorithm across these tasks, outperforming the prior model-free state-of-the-art RAD by 1.9x and 1.5x on the 500k and 1M environment step benchmarks, respectively.
RELATED WORK
Pixel-Based RL The ability of an agent to autonomously learn control policies from visual inputs greatly expand the applicability of deep RL (Dosovitskiy al., 2017; Savva et al., 2019). Prior works have used CNNs to extend RI. algorithms like PPO (Schulman et al., 2017), SAC (Haarnoja al., 2018), and Rainbow (Hessel et al., 2017) to pixel-based tasks. Such direct extensions have typically required substantially larger number of environment interactions when compared to the state-based environments. In order to improve sample efficiency, recent efforts have studied the use of auxiliary tasks and loss functions (Yarats et al., 2019; Laskin et al., 2020b; Schwarzer et al., 2020), data augmentation (Laskin al., 2020a; Kostrikov et al., 2020), and latent space dynamics modeling (Hafner et al., 2019b;a). Despite these advances, there is still large gap between the learning efficiency stste- and pixel-based environments a number of challenging benchmark tasks. Our goal in this work is to identify where and how to improve pixel-based performance this set of challenging control environments.
Neural Network Architectures in RL The work of Mnih et al. (2015) combined Q-learning with CNNs to achieve human level performance in Atari games. In this work, Mnih et al. (2015) con- catenate the most recent 4 frames and use convolutional neural network to output the Q values. In 2016, Mnih et al. (2016) proposed to use a shared CNN among frames to extract visual features and aggregate the temporal information with L.T.. The same architectures have been adopted by most
2

---

Under review as a conference paper at ICLR 2021
works till date (Laskin al., 2020b; Schwarzer al., 2020; Kostrikov et al., 2020; Laskin et al., 2020a). The development of new architectures to better capture temporal information in a stream of images has received little attention in deep RL, and our work aims to fill this void. Perhaps closest to our motivation is the work of Amiranashvili et al. (2018) who explicitly use optical flow as an extra input to the RL policy. However, this approach requires additional information and supervision signal to train the flow estimator, which could be unavailable or inaccurate in practice. In contrast, our approach is simple modification existing deep RL architectures and does not require any additional auxiliary tasks or supervision signals.
Two-Stream Video Classification In video classification tasks, such as activity recognition (Soomro et al., 2012), there are a large body of works on how to utilize temporal information (Donahue et al., 2015; al., 2012; Tran et al., 2015; Carreira & Zisserman, 2017; Wang et al., 2018; Feichtenhofer et al., 20. Of particular relevance is the two-stream architecture of Simonyan & Zisserman (2014), where one CNN stream takes the usual RGB frames, while the other the optical flow computed from the RGB values. The features from both streams are then late-fused to predict the activity class. Simonyan & Zisserman (2014) found that the two-stream architecture yielded a significant performance gain compared to the single RGB stream counterpart, indicating the explicit temporal information carried by the flow plays an essential role in video understanding. Instead of directly computing the optical flow, we propose capture the motion information in latent space to avoid computational overheads and potential flow approximation errors. Our approach also could focus on domain-specific motions that might be overlooked in a generic optical flow representation.
BACKGROUND
Soft Actor Critic (SAC) (Haarnoja et al., 2018) an off-policy actor-critic RL, algorithm for continuous control with an entropy maximization term augmented to its score function to encourage exploration. SAC learns a policy network (a, (o:) and critic networks Qº, (Or, at) and Qø, (O,, az) to estimate state-action values. The critic Qs (oe, ai) is optimized to minimize the (soft) Bellman residual error:
Co(d:) = ET~B [Co.. (or, ax) - (re + +V
(1)
where is the reward, y the discount factor, (or, de, 0141, mr) is transition sampled from replay buffer B, and (o++1) is the (soft) target value estimated by:
((o++1) = (min Qs, (or+1, 1++1) a log T (4t+1|0r+11)
(2)
where ar is the entropy maximization coefficient, For stability, in eq. 2, Q², is the exponential moving average of Qo, 's over training iterations. The policy Th. is trained to maximize the expected return estimated by Q together with the entropy term
La (v) = - Ba [min Qo, (or, a.) a log ne (arlo:).
where ar is also learnable parameter.
Reinforcement Learning with Augmented Data (RAD) (Laskin et al., 2020a) is a recently proposed training technique. In short, RAD pre-processes raw pixel observations by applying random data augmentations, such as random translation and cropping, for RL training. As simple as it is, RAD has taken many existing RL algorithms, including SAC, to the next level. For example, on many DMControl (Tassa et al., 2018) benchmarks, while vanilla pixel- SAC performs poorly, RAD- SAC-ie. applying data augmentation pixel-based SAC achieves state-of-the results both in sample efficiency and final performance. In this work, we refer RAD to RAD-SAC and the augmentation used random translation.
Rainbow DON is an extension of the Nature Deep Q Network (DQN) (Mnih et al., 2015), which combines multiple follow-up improvements of DQN to a single algorithm (Hessel et al., 2017). In summary, DQN (Mnih et al., 2015) is an off-policy RL algorithm that leverages deep neural networks (DNN) to estimate the Q value directly from the pixel space. The follow-up works Rainbow DQN bring together to enhance the original DQN include double Q learning (Hasselt, 2010), prioritized experience replay (Schaul et al., 2015), dueling network (Wang et al., 2016), noisy network (Fortunato al., 20, distributional RL (Bellemare al., 2017) and multi-step returns (Sutton & Barto, 1998).
3

---

Under review as a conference paper at ICLR 2021
Rainbow DQN is one of the state-of-the- art RL algorithms the Atari 2600 benchmark (Bellemare et al., 2013). We thus adopt an official implementation of Rainbow (Quan Ostrovski, 2020) as our baseline to directly augment Flare top.
4 MOTIVATION
pendulum Aa nguup
quadruped walk
happer hop
Wel bar walk
curipole swingup
1000
530
820
622
150
203
433
400
230
222
1.5
2/0
012
2.5
3/0
1.5
0.3
2/3
0.2
0.
1.2
355
29
259
Full state SNC
Flare
Positio only SAC
Figure 2: (i) full-state SAC (blue) where input contains both pose and temporal information; (ii) position-only SAC (green) with only pose information input; (iii) Flare applied to the state space (orange) with pose information and velocity approximations through pose offsets as input. While full-state SAC efficiently learns the optimal policy, position-only SAC recovers suboptimal policies and fails learning at all in some cases. Meanwhile, the fusion of approximated velocities in Flare is able recover the optimal policy nearly efficiently the full state SAC in most cases, Results are averaged over 3 seeds with standard deviation.
pendutum
Weller
2003
3530
800
630
(20
420
430
400
253
166
165
Envernment Chip
140
100
340
Flare
Recument
SAC
Figure 3: We compare Flare to 2 SAC variants: i) Stack SAC (green) receives consecutive positional states (St, St-1, $t-2, 8t-3) as input, whereas positional-only SAC receives (s:) and Flare receives (st, 5t) where 6, (s, 8t-1, 8t-1 $t-2; St-2 8t 8t-3). ii) Recurrent SAC (blue) uses recurrent layers to process a series of states. Despite of the implicit access to temporal information between consecutive states, Stack SAC and Recurrent perform significantly worse than Flare on most environ- ments, highlighting the benefit of explicit fusion of temporal information. Results are averaged over three seeds.
We motivate our method by investigating the importance of temporal information in state-based RL. Our investigation utilizes five diverse DMControl (Tassa et al., 2018) 8) tasks. The full state for these environments includes both the agent's pose information, such as the joints' positions and angles, as well as temporal information, such as the joints' translational and angular velocities. We train two variants with SAC-one variant where the agent receives the full state as input (full-state SAC), and the other with the temporal information masked out, i.c. the agent only receives the pose information as its input (position-only SAC). The resulting learning curves are in Figure 2. While the full-state SAC learns the optimal policy quickly, the position-only SAC learns much sub-optimal policies, which often fail entirely. It is therefore clearly shown that effective policies cannot be learned from positional information alone, and that temporal information is crucial for efficient learning.
While full-state SAC can receive velocity information from internal sensors in simulation, in the more general case such as learning from pixels, such information is often not readily available. For this reason, we investigate whether we can explicitly approximate temporal information as the difference between two consecutive states. If the input the positional state, then this positional difference roughly approximates the agent's velocity. Given poses sf, sp S% o. at time ł, 1,,,,,,-2, t-3, we compute the positional offset 6;=(s; - St-1. $t-1 Sc-2; 81-2 st-3), and provide the fused vector (st, 5,) to the SAC agent. This procedure precisely describes the state- mase-based version of Flare. Results shown in Figure 2 demonstrate that state- based Flare significantly outperforms the position- only SAC. Furthermore, state-based Flare achieves optimal asymptotic performance, and its learning
4

---

Under review as a conference paper at ICLR 2021
Lutent Vistors
Latent Bieeenox
Subsaction
Concutomaaion
0--1
Q
(nly Mp-is (p-2)
0p-2
stacking heuristic
(b) Individull encoding
Flow of Latents for RL (Flora)
Figure 4: Flow of Latents for Reinforcement Learning (Flare). In panel (a) we show the architecture for the frame stacking heuristic, in (b) we show alternative to the frame stacking hueristic by encoding each image individually, and in (c) we show the Flare architecture which encodes images individually, computes the feature differences, and fuses the differences together with the latents.
efficiency is comparable to full-state SAC in most environments. Given that the position-oni -onlo SAC (which utilizes s, alone) has only partial information compared to Flare that utilizes (st, 6t), we also investigate a variant where we provide consecutive positions (8t, 8t-1, 8t-2, 9x-3) to the SAC agent. We call this variant Stack SAC, since it is identical to the frame-stack heuristic used in pixel-based RL. Results in Figure 3 show that Flare still significantly outperforms the Stack SAC. It suggests that the well-structured inductive bias in the form of eemporal-position fusion is essential for efficient learning.
recurrent structure is an alternative approach process temporal information. We implement an SAC variant with recurrent modules (Recurrent SAC) and compare it with Flare. Specifically, we pass a sequence of poses s s 2: s/=3 through an LSTM cell. The number of the LSTM hidden units h is set to be the same as the dimension of 8, in Flare. The trainable parameters of the LSTM cell are updated to minimize the critic loss. Recurrent SAC more complex to implement and requires longer wall-clock training time, but performs worse than Flare shown in Figure 3.
Our findings from the state experiments Figure 2 and Figure suggest that (i) temporal information is crucial to learning effective policies in RI. and (ii) approximating temporal information in the absence of sensors that provide explicit measurements is sufficient in most cases. When learning from pixels, it is common to assume the absence of specialized sensors for reading out temporal information. We therefore hypothesize that explicit fusion of temporal information approximated directly from pixel-level inputs improve the efficiency of learning control policies.
REINFORCEMENT LEARNING WITH LATENT FLOW
To date, frame stacking is the most common way of pre-processing pixel-based input to convey temporal information for RL algorithms. This heuristic, introduced by Mnih et al. (2015), has been largely untouched since its inception and is used in most state-of-the-art RL architectures. However, our observations from the experiments run on state input in Section 4 suggest an alternative to the frame stacking heuristic through the explicit inclusion of temporal information as part of the input. To learn effective control policies from pixels, we seek a general approach to explicitly incorporate temporal information that can be coupled to any base RL algorithm with minimal modification. To this end, we propose the Flow of Latents for Reinforcement Learning (Flare) architecture. Our proposed method calculates differences between the latent encodings of individual frames and fuses the feature differences and latent embeddings before passing them as input to the base RL algorithm, shown in Figure 4. We demonstrate Flare top of 2 state-of-the ofetth-art model-free free off off-policy RL baselines, RAD-SAC (Laskin et al., 2020a) and Rainbow DQN (Hessel et al., 2017), though any RL algorithm can be used in principle.
5

---

Under review as a conference paper at ICLR 2021
Task
Flare (500K)
RAD (500K)
Flare (IM)
RAD (1M)
Quadruped Walk
296 + 139
206 = 112
488 + 221
322 + 229
Pendulum Swingup
242 152
79 73
809 31
520 321
Hopper Hop
90 - 55
40
217 59
211 27
Finger Turn hard
282 = 67
137 + 98
661 315
249 + 98
Walker Run
426 33
547 + 48
556 + 93
628 39
Table 1: Evaluation 5 benchmark tasks around 500K and 1M environment steps. We evaluate over 5 seeds,
each of 10 trajectories and show the mean + standard deviation across runs,
Flare (50M)
Rainbow (50M)
Flare (50M)
Rainbow (50M)
Assault
9926
1012312061
Breakout
330
321:334
Freeway
34:0
3410
Krull
8423=117
8030-1717
Montezuma
400+0
0=00
Seaquest
8362+1180
4521+3354
Up Down
44055:12276
24568+22216
Tutankham
240=7
148+16
Table 2: Evaluation on benchmark Atari games at 50M training steps over seeds.
5.1
LATENT FLOW
In computer vision, the most common approach to explicitly inject temporal information of a video sequence to compute dense optical flow between consecutive frames (Simonyan & Zisserman, 2014). Then the RGB and the optical flow inputs are individually fed into two streams of encoders and the features from both streams are fused the later stage of the network. However, two-stream architectures with optical flow are not directly applicable to RL. The main issue is that the computation of optical flow is slow: during inference, it often prohibitively expensive to compute real-time for applications with fast control loops like robotics; during training, optical flow calculation adds significant overhead to the wallclock training time in online learning settings like RL. While video architectures can utilize memory bank, such that optical flow need only be pre-computed once for the entire dataset, RL training is done dynamically and on the fly, and computing optical flow at each step therefore costly.
To address this challenge and motivated by ex- periments in Section 4, we propose alterna- tive architecture that is similar in spirit to the two-stream networks for video classification. Rather than computing optical flow directly, we approximate temporal information in the latent space. Instead of encoding a stack of frames at once, we use a frame-wise CNN to encode each individual frame. Then we compute the differences between the latent encodings of con- secutive frames, which we refer to as latent flow. Finally, the latent features and the latent flow fused together through concatenation before getting passed to the downstream RL algorithm. We call the proposed architecture as Flow of Latents for Reinforcement Learning (Flare).
Algorithm 1: Pixel-bascd based Flare Inference
Given Ty. SCNN;
for each environment step do
2j = fCNN fcNN(o)), k.,
2 2j-1, k+1
(2 12,,06-0 0t-k+1, ..,,,,,,,,,6.
a. (at Zz);
044 04 (pp11p(0++111 ( Oz = (or, ot- 10-
end
While Flare is broadly applicable technique, for clarity of exposition, select RAD the base algorithm to elaborate the execution of Flare. We also use RAD later on in our experiments as the comparative baseline (Section 6). The RAD architecture, shown in Figure 4a. stacks multiple data augmented frames observed in the pixel space and encodes them altogether through an CNN. This can be viewed as form of early fusion (Karpathy et al., 2014). Another preprocessing option is to encode each frame individually through a shared frame-wise encoder and perform late fusion of the resulting latent features, as shown in Figure 4b. However, we find that simply concatenating the latent features results in inferior performance when compared to the frame stacking heuristic, which we further elaborate in Section 6.3. We conjecture that pixel-level frame stacking benefits from leveraging both the CNN and the fully connected layers to process temporal information, whereas latent-level stacking does not propagate temporal information back through the CNN encoder. Based on this conjecture, gradients when computing &, We fuse the latent flow ó, with the latent embedding 2t, and pass the fused input to the actor and critic networks as shown in Figure 4c. We provide pseudocode that illustrates how to do inference with Flare in Algorithm 1; during training, the encodings of latent features and latent flow are done in the same way except with augmented observations.
explicitly compute the latent flow 8, 24 2t-1 while detaching the 2,-1

---

Under review as a conference paper at ICLR 2021
Quadruped Walk
Hopper Hop
Pendulum Swingup
Walker Run
Finger Turn Hard
We choose the following environments for our main experiments (i) quadruped walk, which requires coordination of multiple joints, (ii) hopper hop. which requires hopping while maintaining balance, (iii) pendulum swingup, an environment with sparse rewards, (iv) walker run, which requires the agent to maintain balance high speeds, and (v) finger turn hard, which requires precise manipulation of a rotating object. These environments are deemed challenging because prior state-of-the-ar model-free pixel-based methods (Laskin et al., 2020b; Kostrikov et al., 2020; Laskin et al., 2020a) either fail reach the asymptotic performance of state SAC or learn less efficiently.
Figure 5:
6 EXPERIMENTS
pendurum swingup
3000
stadruped mafc
hopper hop
1000
Tmpper turn hard
Yve ban
600
6CC
330
6000
200
282
200
200
03
RAD
Flare
Stato-SAC
Figure 6: We compare the performance of Flare to RAD, a state-of-the-art algorithm and the base algorithm used in Flare, on five challenging environments. Pendulum Swingup trained over 1.5e6 and the rest 2.5e6. We see that Flare substantially outperforms RAD on a majority (3 out the 5) of environments, while being competitive in the remaining. While not closing the gap between pixel and state-based performance entirely, Flare is closer to state-based performance than prior methods, and is the tftte-of-the-rr pixel-based model-free algorithm on most of these challenging environments. Results are averaged over 5 random seeds with standard deviation (shaded regions).
breakout
10303
33
310
23
3500
200
3500
6093
2000
mantezuma revenge
west
cown
tutarkhaam
10000
303
293
300
100
23303
195
1a7
Rainbow DON
Flare
Figure 7: We compare Rainbow DQN and Flare 8 Atari games over 50M training steps. Flare substantially enhances a majority (5 out of 8) of the games over the baseline Rainbow DQN while matching the rest. Results averaged over 3 random seeds with standard deviation (shaded regions).
7

---

Under review as a conference paper at ICLR 2021
1000
pixel flow vs latent flow
RAD
1000
with without latent flow
frame stack (RAD)
Flare 2.3.5 frame stacking
frames
Ponduilum Swir
Egiacdio fature
-00
latent flow (Flare)
B00
frames
laliet stack flow (Flare)
800
60D
pixal flow
400
latont stack only
200
200
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
166
1030
1000
Cuaadripad, Walk
800
600
000
600
600
400
200
220
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.5
3.5
2.0
les
Environment Stop
(a) pixel flow ablation
(b) latent stacking ablation
(c) frame count ablation
Figure 8: We perform three ablation studies. (a) flow ablation: we compare Flare to a variant where the differences are computed directly in pixel space (pixel flow) and find that latent flow is more stable and achieves better performance. (b) Latent stack ablation: in this experiment, the latent vectors without the temporal approximation. We find that this method performs significantly worse than Flare, and quadruped fails entirely, suggesting that fusing explicit temporal information is crucial. (c) Frames count ablation: We test whether adding more frames increases performance for Flare. We find that including additional input frames either does not change or degrades performance.
fuse
We first introduce the 5 core challenging continuous control tasks from DMControl suite (Tassa et al., 2018) that our experiments focus on. Next we present the main experimental results, where we show that Flare achieves substantial performance gains over the base algorithm RAD (Laskin et al., 2020a). Finally, we conduct a series of ablation studies to stress test the design choices of the Flare architecture.
6.1 ENVIRONMENTS AND EVALUATION METRICS
The DeepMind Control Suite (DMControl) (Tassa et al., 2018), based on MuJoCo (Todorov et al., 2012), is a commonly used benchmark for continuous control from pixels. Prior works such as DrQ (Kostrikov et al., 2020) and RAD (Laskin et al., 2020a) have made substantial progress on this benchmark and closed the gap between state-based and pixel-based efficiency on the simpler environments in the suite, such as Reacher Easy, Ball-in-cup Catch, Finger Spin, Walker Walk, Cheetah Run, Cartpole Swingup. However, current pixel-based RL algorithms struggle to learn optimal policies efficiently in more challenging environments that feature partial observability, sparse rewards, or precise manipulation. In this work, we study more challenging tasks from the suite to better showcase the efficacy of our proposed method. The 5 environments, listed in Figure 5, include Walker Run (requires maintaining balance with speed), Quadruped Walk (partially observable agent morphology). Hopper Hop (locomotion with sparse rewards), Finger Turn-hard (precise manipulation), and Pendulum Swingup (torque control with sparse rewards). For evaluation, we benchmark performance at 500K and 1M environment steps and compare against RAD.
The Atari 2600 Games (Bellemare et al., 2013) is another highly popular RL benchmark. Recent efforts have let range of highly successful algorithms (Espeholt et al., 2018: Hessel et al., 2017; Kapturowski et al., 2018; Hafner et al., 2019a; Badia et al., 2020) to solve Atari games directly from pixel space. A representative state-of-the-art is Rainbow DQN (see Section 3). We adopt the official Rainbow DQN implementation (Quan& & Ostrovski, 2020) as our baseline. Then we simply modify the model architecture to incorporate Flare while retaining all the other default settings, including hyperparameters and preprocessing. To ensure comparable model capacity, the Flare network halves the number of convolutional channels and adds a bottleneck FC layer to reduce latent dimension before entering the Q head (code in the Supplementary Materials). We evaluate Atari games at 50M training steps, namely Assault, Breakout, Freeway, Krull, Montezuma Revenge, Seaquest, Up n Down and Tutankham, to assess the effectiveness of Flare.
diverse subset of
8

---

Under review as a conference paper at ICLR 2021
6.2 MAIN RESULTS
DMControl: Our main experimental results on the five DMControl tasks are presented in Figure and Table I. We find that Flare outperforms RAD in terms of both final performance and sample efficiency for majority (3 out of 5) of the environments, while being competitive on the remaining environments. Specifically, Flare attains similar asymptotic performance to state-based RI. on Pendulum Swingup, Hopper Hop, and Finger Turn-hard. For Quadruped Walk, particularly challenging environment due to its large action space and partial observability, Flare learns much more efficiently than RAD and achieves a higher final score. Moreover, Flare outperforms RAD in terms of sample efficiency on all of the core tasks except for Walker Run as shown in Figure 6. The 500k and IM environment step evaluations in Table 1 show that, on average, Flare achieves x.9x and 1.5x higher scores than RAD at the 500k step and the IM step benchmarks, respectively. Though our investigation primarily focuses these 5 challenging environments, we also show in Appendix A. that Flare matches the state-of-the-art on the 6 simpler environments.
Atari: The results the 8 Atari games are in Figure 7 and Table 3. Again, we observe substantial performance gain from Flare on the majority of the games while being equally competitive to the baseline Rainbow DQN on the remaining games. In Appendix A.2, we also show that Flare performs competitively when comparing against other DQN variants at 50M training steps.
6.3 ABLATION STUDIES
We ablate number of components of the Flare architecture on the Quadruped Walk and Pendulum Swingup environments to stress test the Flare architecture. The results shown Figure 8 aim to answer the following questions:
Q1: Do we need latent flow or computing pixel differences sufficient? While Flare proposes a late fusion of latent differences with the latent embeddings, simpler approach is an early fusion of pixel differences with the pixel input, which we call pixel flow. We compare Flare to pixel flow in Figure 8 (left) and find that, while pixel flow outperforms RAD, it is significantly less efficient and less stable than Flare, particularly on Quadruped Walk. This ablation suggests that late fusion temporal information after encoding the image is preferable early fusion.
Q2: Are the gains coming from latent flow or individual frame-wise encoding? Next, we address the potential concern that the performance gain of Flare stems from the frame-wise ConvNet architectural modification instead of the fusion of latent flow. Concretely, we follow the exact architecture and training as Flare, but instead of concatenating the latent flow, we concatenate each frame's latent after the convolution encoders directly as described in Figure 4 (b). This ablation is similar in spirit to the state-based experiments in Figure 3. The learning curves in Figure (center) show that individual frame-wii wiie encoding is not the source of increased performance. While on par with RAD on Pendulum Swingup, on Quadruped Walk frame- wise encoding performs worse. Flare's improved performance over RAD is therefore most likely a result of the explicit fusion of latent flow.
Q3: How does the input frame count affect performance? Lastly, we compare stacking 2, 3, and frames in Flare in Figure 8 (right). We find that changing the number of stacked frames does not significantly impact the locomotion task, quadruped walk, but Pendulum Swingup tends to be more sensitive to this hyperparameter. Interestingly, the optimal number of frames for Pendulum Swingup 2, and more frames can in fact degrade Flare's performance, indicating that the immediate position and velocity information is the most critical to learn effective policies on this task. We hypothesize that Flare trains more slowly with increased frame count on Pendulum Swingup due to the presence of unnecessary information that the actor and critic networks need to learn to ignore.
CONCLUSION
We propose Flare, an architecture for RL that explicitly encode temporal information by computing flow in the latent space. In experiments, we show that in the state space, Flare can recover the optimal performance with only state positions and no access to the state velocities. In the pixel space, Flare improves upon the state-of-the- the model-free RL algorithms the majority of selected tasks in the DMControl and Atari suites, while matching in the remaining. Integrating Flare with model- based RL is a potential direction for future works.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, and Thomas Brox. Motion perception in reinforcement learning with dynamic objects. In Conference on Robot Learning, pp. 156-168.
PMLR, 2018.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, 2020.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning, arXiv preprint arXiv: 1xXiv:/707.06885 2017.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.0660 2019.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6360 -399-6308, 2017.
Yevgen Chebotar, Mrinal Kalakrishnan, Ali Yahya, Adrian Li, S. Schaal, and S. Levine. Path integral guided policy search. 2017 IEFE International Conference on Robotics and Automation (ICRA), pp. 3381-3388, 2017.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2015.
Alcxey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. arXiv preprint arXiv:/711.03955 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv: 1802.0156, 2018.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE international conference on computer vision, pp. 6202 2202-6211, 2019.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, al. Noisy networks for exploration. arXiv preprint arXiv: / /706. 10295, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.0122 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01660 2019a.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, pp.
PMLR, 2019b.
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23:2613-2622 2010.
10

---

Under review as a conference paper at ICLR 2021
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv: 2017.
Divye Jain, Andrew Li, Shivam Singhal, Aravind Rajeswaran, Vikash Kumar, and Emanuel Todorov. Learning Deep Visuomotor Policies for Dexterous Hand Manipulation. In International Conference on Robotics and Automation (ICRA), 2019.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence, 35(1):221-231, 2012.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp- 1725-1732, 2014.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 1.00044.3344. 13649, 2020.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbecl, and Aravind Srinivas. Reinforcement learning with augmented data, arXiv preprint ar arXiv:2004. 14990, 2020a.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, Proceedings of the 37th International Conference Machine Learning, Vienna, Austria, PMLR 119, 2020b. arXiv:2004.04136
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
Kendall Lowrey, Svetoslav Kolev, Jeremy Dao, Aravind Rajeswaran, and Emanuel Todorov. Rein- forcement learning for non-prehensile manipulation: Transfer from simulation to physical system. In JEEE SIMPAR, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-19999999 1938,197,240 2016.
OpenAl, Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray. Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in-hand manipulation. CoRR, abs//800..o. 001800077, 2018.
John Quan and Georg Ostrovski. DQN Zoo: Reference implementations of DQN-based agents, 2020.
URL http: /ttp::://gttiie thu.....tuuuuuuggg com/ deepmind/ /dgp//gnn_o__oo//22
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings Robotics: Science and Systems (RSS), 2018.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao Erik Wijmans, Bhavana Jain, Julian Straub. Jia Liu, Vladlen Koltun, Jitendra Malik, al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9347, 2019.
11

---

Under review as a conference paper at ICLR 2021
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05992 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.066: 2017.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with momentum predictive representations. arXiv preprint arXiv:2007.05999, 2020.
David Silver, Aja Huang, Christopher Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-503, 2016. URL http: //www. nature. com cwm/natttuuteeeee. http//atuurrreeee wttpr///oorraaa/vaa/ http://///////////1//// http:///www//aanteeeeeeeeeee html.
David Silver, Julian Schrittieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap. Fan Hui, Laurent Sifre, George Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-359, 10 2017. doi: 10. 10.00/aauture24270.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pp. 568-576, 2014.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2/2.4002, 2012.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning, 2020.
Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 18010009000 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489-4444 110999449,,211111
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782): 350-354, 2019. doi: 10.1038/441166-01--11 s41586- 01 9- 9
URL https: //l:://doo http otgpi.org/1..10
Aaron Walsman, Yonatan Bisk, Saadia Gabriel, Dipendra Misra, Yoav Artzi, Yejin Choi, and Dieter Fox. Early Fusion for Goal Directed Robotic Vision. In International Conference Intelligent Robots and Systems (IROS), 2019.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
12

---

Under review as a conference paper at ICLR 2021
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003, 2016.
Denis Yarats, Amy Zhang, Iya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741. 2019.
H. Zhu, A. Gupta, Rajeswaran, Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. 2019 International Conference Rebotics and Automation (ICRA), pp. 3651-3657, 2019.
13


Reviews:
[{"review_id": "tIvuQp_bsKL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis work presents a simple technique (Flare) to incorporate explicit temporal information to enable effective RL policy learning in challenging continuous control environments using pixel-based state representations. The approach is inspired from recent advances in the video recognition approaches which employ optical flow information and late fusion to incorporate temporal information. Typically, RL algorithms employ a frame-stacking heuristic to incorporate temporal information (early fusion). Though computing optical flow is slow and can been prohibitive for real-time applications, the authors present a simple alternative to using optical flow, i.e. difference of latent state vectors as a proxy for explicitly encoding motion information with latent vectors representing the observed state (late fusion). Experimental results on challenging continuous control tasks in the DMControl Suite show that Flare can achieve up to 1.9x higher scores than a baseline algorithm (RAD) which uses the frame-stacking heuristic to incorporate temporal information.\n\n########################\n\nPros:\n- The presented approach provides an effective alternative to the frame stacking heuristic for incorporating temporal information in RL with pixel-based state representations. The presented methodology (concatenation of latent state vector differences) thus learns effective policies on challenging continuous control environments.\n- The presented approach can easily modify any RL algorithm operating on pixel-based state representations to encode temporal information. \n- The paper is well-written, clear and easy to follow.\n- The idea is well-motivated with experiments in environments with low-dimensional state spaces. The results from the motivation section show the importance of temporal information, and specifically explicit temporal information to learn effective policies.\n- Ablation study clearly highlights the merits of including explicit temporal information with late fusion ruling out other techniques like pixel-based flow (early fusion) and the effect of independent convolutional feature extraction for different image frames.\n\n########################\n\nCons:\n- Comparisons to other approaches mentioned in Section 2 are missing.  For example, how would Flare compare in performance and sample efficiency to LSTM based RL methods (such as those under the “neural network architectures” subsection listed under Section 2)?\n- Flare does not outperform RAD in all environments (such as hopper hop and walker run). It is unclear why Flare works well on some environments and not so on others. In fact, Flare performs worse than RAD on walker run. Why so?\n\n\n########################\n\nReason for score:\nThe approach is well-motivated, and the paper is clearly written. The experiments comparing with an early fusion approach (RAD) and related ablative analysis highlighting that explicit late fusion of temporal information is key to improved performance are well-done and prove the effectiveness of the Flare. However, comparison to other approaches incorporating explicit temporal information for RL is missing (eg: LSTM based approaches). \n\n########################\n\nQuestions during rebuttal:\n- Please refer to questions in the Cons section and other feedback\n- For results from Figure 6, why does Flare not outperform RAD for the hopper hop and walker run environments? Is there a way to visualize the temporal information to further investigate why Flare outperforms RAD in some environments (like quadruped walk) and not on others?\n\n########################\n\nSome typos and other feedback:\n- Figure 2 and Section 5, paragraph 1, sentence 3: What is meant by proprioceptive state input? Consider formally defining it in the text for the reader.\n- Section 5, paragraph 1, sentence 3: Consider ending the sentence by stating the fact that the experiments suggest the alternative to frame stacking heuristic is also effective in terms of performance.\n- Consider defining p(a_t+1|a_t,o_t) as the transition function in the text for the reader.\n- Section 5.1, last sentence: “… are done in the same except with augmented observations.” -> “… are done in the same way except with augmented observations.”\n- Section 6, paragraph 1, sentence 1: “… that are experiments focus on.” -> “… that our experiments focus on.”\n- Section 6.2, sentence 2:  You seem to have forgotten to mention the environments for which Flare outperforms RAD. What are you referring to with the phrase “remaining environments”?\n- Section 6.2, sentence 5: “… walker run shown visualized in Figure 6.” -> “… walker run as shown in Figure 6.“\n- Section 6.2, Figure 6: Why does Flare not perform as well as RAD for the Walker run environment?\n- Conclusion, last sentence: Consider replacing “We would like to integrate Flare with model-based RL in the future” with “Integrating Flare with model-based RL is a potential direction for future work.”\n- Consider replacing the usage of the phrase “state-based RL” with “RL on low-dimensional state space”.", "extra_scores": {}, "extra_text": {}}, {"review_id": "t0FClChyvKw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Significance:\nThe paper brings very little novelty or insight. It is unclear that the introduced architecture complexity worth marginal improvements (given high variance and only 5 random seeds) on 2 out of 11 tasks (5 from the main paper and 6 from appendix). This might be a good workshop paper but it clearly does not meet the high acceptance threshold of ICLR.\n\nPros:\n-A simple architecture modification that might be beneficial to impose a stronger inductive bias on temporal dynamics.\n\nCons:\n-The proposed algorithm is a trivial architecture change to the conv encoder, the introduced novelty is limited. Injecting the temporal difference inductive bias obviously will be beneficial, given that the sole reason for frame stacking is to infer velocity and acceleration. \n-In Fig 3 the authors chose to only use 2 consecutive frames for State SAC, while a common practice is to use 3 frames. Using only 2 consecutive frames is not enough to infer acceleration and thus it is not a realistic setup, which makes this comparison meaningless. Also given that the variance is pretty high here, comparing performance over just 3 random seeds is not statistically conclusive. \n-In Fig 6 the method is only evaluated only on 5 seeds and given that it demonstrates very high variance on the 3 tasks (out of 5) where it outperforms RAD it makes me think that the performance improvements are marginal and not worth introducing complexity. \n-The ablation study is not very illuminating, this partially comes from the fact that the results are inconclusive (due to the high error bars), and partially because the experiments themselves are not very interesting. \n\n\nQuality:\nWhile the paper is well executed and made it significantly lacks on novelty and significance fronts.\n\nClarity:\nThe paper in general is clearly written and well organized.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qn9qxBV4wn1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary:\n    - This paper presents Flare, an RL method that replaces frame stacking (early fusion) with latent vector stacking (late fusion) and then further improve upon this by adding in latent flow vectors (the difference between adjacent latent vectors)\n    - The method is demonstrated on DM control using RAD-SAC as the baseline.  In all but one case, Flare outperforms the baseline\n    - The authors then present a series of ablations to show that latent flow outperforms pixel flow and that stacking latent flow is better than pure latent stacking\n- Strengths:\n    - Relatively straightforward idea that improves upon the common technique of frame stacking, making this likely a technique with broad appeal and impact\n    - Good ablation study showing how the individual components.\n    - Well motivated idea.  I liked the experiment of position only SAC to set the motivation.\n- Weaknesses\n    - Concat + MLP seems like a poor way to do sequence modeling (a series of frames is, after all, a sequence).  How does Flare and the latent frame baseline behave if a GRU or LSTM is used to combine the the sequence of latent vectors?  This would avoid the normal challenges of training a recurrent policy while also benefiting from the superior sequence modeling of an RNN.\n    - Performance on pendulum degrading as the number of frames is increased is concerning.  If possible, I wold like to see the hypothesis posed in Sec 6.3 Q3 validated by training longer.  Another possible hypothesis is that latent vector stacking increases the number of parameters and causes Q function overfitting.\n    - Questions:\n        - How was RAD applied to series of frames?  Was the same translation applied to all or was a different one applied too each?\n- Overall\n    - This paper presents and effective idea, however, there are some additional experiments (using an RNN to combine latent vectors) that I think would strength the paper considerably\n\n\n## Post Rebuttal\n\nI thank the authors for their response.  The addition of the recurrent SAC baseline helps the paper.  I disagree with R1 that it is a stronger baseline as FLARE outperforms it in all tasks and stack SAC similar or better three (arguably four) of five tasks. Instead it shows that recurrence isn't common in off-policy RL because it doesn't always perform better.  While recurrence is considerably more common in embodied 3D environments and this work may be less applicable there, I don't foresee DM control style RL benchmarks going away anytime soon and this believe this method will be useful.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lr3bBkjePkg", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new method for aggregating temporal information in reinforcement learning policies. The method takes the difference of latent representations between consecutive frames and concatenates this difference with the latent representations for downstream processing.\n\nStrengths:\n- The method is simple and easy to understand and implement.\n- The method outperforms a baseline on average based on experiments on 5 Deepmind control benchmark suite tasks under a limited budget of training samples.\n- Ablations show an interesting result that the fusion of latent representations performs better than pixel-based fusion.\n\nWeaknesses:\n- The contribution is incremental in my opinion. It makes a small change over existing RL methods. The improvement in performance on several tasks is also marginal. \n- One of the most common methods of aggregating temporal information is just using a recurrent layer in RL policies. It’s unclear if the baseline consists of a recurrent layer. If it does not, a simple recurrent policy needs to be added as a baseline. If it does, it is unclear to me why a simple subtraction of latent representations would result in performance improvement. I would imagine that it should be very easy for a recurrent layer such as an LSTM to learn differences between latent representations if they were useful. Some discussion on this would be useful.\n- The performance gain is only demonstrated on 5 tasks and it’s unclear whether it would translate to gains in other tasks as well. Results in the appendix on 6 easier tasks show results comparable to or worse than the baseline.\n\nOverall, I believe that the paper proposes a simple and interesting method, but I am not convinced that it would lead to better results consistently. The main experiments are conducted only on 5 tasks and the performance gains are marginal in my opinion.\n\nUpdate after rebuttal: \nThe authors have added a recurrent SAC baseline to one set of experiments. The results indicate that the recurrent SAC is a much stronger baseline, and the variance of results is high enough that I am not convinced of the benefits of the proposed method.\n\nThe authors argue that \"many state-of-the-art RL algorithms\" \"are non-recurrent\", and \"frame stacking\" is \"largely untouched since its inception and is used in most state-of-the-art RL architectures\", \"recurrent architectures\" have \"additional overhead from training and implementation\". I do not believe this is true. Recurrent architectures are commonly used in RL algorithms (for example RSSM in Dreamer) and are widely available in open-source implementations (for example https://github.com/openai/baselines/blob/master/baselines/common/models.py). There are some prior papers which use the frame stacking heuristic for a fair comparison with DQN, but this heuristic or the non-recurrent model architecture is not a part of the RL algorithm itself. Since this paper proposes a method for extracting temporal information, LSTM/GRU are very natural baselines in my opinion and should be added to all experiments.\n\nThe authors argue that \"it is very reasonable and common to have a new method improving a majority of the environments but not all\", I agree with this, but the examples given by the authors such as Rainbow DQN, Dueling DQN, Dreamer etc perform experiments in many more environments and performance improvements are larger. I believe a much larger scale study is needed to compare Flare with recurrent baselines and make conclusive statements about performance gains.\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a simple approach to incorporate temporal information in RL algorithms. AC agrees with authors that simplicity is a virtue. As reviewers point out that experimentally the approach is not conclusively better (given that environments might be hand-chosen). Even R3 believes some reported improvements is within variance. Given the discussions, AC agrees that results do not seem convincing enough.
Decision: Reject