Reinforcement Learning with Latent Flow
Year: 2022

Abstract:
Temporal information is essential to learning effective policies with Reinforcement Learning (RL). However, current state-of-the-art RL algorithms either assume that such information is given as part of the state space or, when learning from pixels, use the simple heuristic of frame-stacking to implicitly capture temporal information present in the image observations. This heuristic is in contrast to the current paradigm in video classification architectures, which utilize explicit encodings of temporal information through methods such as optical flow and two-stream architectures to achieve state-of-the-art performance. Inspired by leading video classification architectures, we introduce the Flow of Latents for Reinforcement Learning (Flare), a network architecture for RL that explicitly encodes temporal information through latent vector differences. We show that Flare recovers optimal performance in state-based RL without explicit access to the state velocity, solely with positional state information. Flare is the most sample efficient model-free pixel-based RL algorithm on the DeepMind Control suite when evaluated on the 500k and 1M step benchmarks across 5 challenging control tasks, and, when used with Rainbow DQN, outperforms the competitive baseline on Atari games at 100M time step benchmark across 8 challenging games.

heading: 1 Introduction
text: Reinforcement learning (RL) [41] holds the promise of enabling artificial agents to solve a diverse set of tasks in uncertain and unstructured environments. Recent developments in RL with deep neural networks have led to tremendous advances in autonomous decision making. Notable examples include classical board games [36, 37], video games [29, 6, 45], and continuous control [34, 28]. There has been a large body of research on extracting high quality features during the RL process, such as with auxiliary losses [20, 27, 35] or data augmentation [25, 26]. However, another important component in RL representation learning has been largely overlooked: a more effective architecture to incorporate temporal features. This becomes especially crucial in an unstructured real-world setup like the home when compact state representations such as calibrated sensory inputs are unavailable. Motivated by this understanding, we explore architectural improvements to better utilize temporal features for the problem of efficient and effective deep RL from pixels. ∗Equal contribution
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Current approaches in deep RL for learning temporal features are largely heuristic in nature. A commonly employed approach is to stack the most recent frames [29] as inputs to a convolutional neural network (CNN). This can be interpreted as a form of early fusion [24], where information from the recent time window is combined at the pixel level for input to the CNN. In contrast, modern video recognition systems use alternate architectures that employ optical flow and late fusion [38], where frames are processed individually with CNN layers before fusion and downstream processing. Late fusion is typically beneficial due to better performance, fewer parameters, and the ability to use multi-modal data [8, 21]. However, it is not straightforward how to directly extend such architectures to RL. Real-time computation of optical flow for action selection can be computationally infeasible for many applications with fast control loops like robotics. Furthermore, optical flow computation at training time can also be prohibitively expensive. In our experiments, we also observe that a naive late fusion architecture minus the optical flow yields poor results in RL settings (see Section 6.3). This observation is consistent with recent findings in related domains like visual navigation [46].
To overcome the above challenges, we develop Flow of Latents for Reinforcement Learning (Flare), a new architecture for deep RL from pixels (Figure 4). Flare can be interpreted as a structured late fusion architecture. It processes each frame individually to compute latent vectors, similar to a standard late fusion approach. Subsequently, temporal differences between the latent feature vectors are computed and fused along with the latent vectors by concatenation for downstream processing. By incorporating this structure of temporal difference in latent feature space, we provide the learning agent with appropriate inductive bias.
We highlight the main empirical contributions from Flare in the following2:
1. Flare recovers optimal performance in state-based RL without explicit access to the state velocity, solely with positional state information.
2. Flare achieves state-of-the-art performance compared to model-free methods on several challenging pixel-based continuous control tasks within the DeepMind control benchmark suite [42], while being the most sample efficient model-free pixel-based RL algorithm across these tasks, outperforming the prior model-free state-of-the-art RAD on the 500k and 1M environment step benchmarks respectively (Figure 1). A video demonstration of Flare achieving SOTA performance on Quadruped Walk is in the supplementary materials.
3. When augmented over Rainbow DQN, Flare outperforms the baseline on 5 out of 8 challenging Atari games at 100M step benchmark. Notebly, Flare scores 1668 on Montezuma’s Revenge, a signifcant gain over the baseline Rainbow DQN’s 900.

heading: 2 Related Work
text: Pixel-Based RL The ability of an agent to autonomously learn control policies from visual inputs can greatly expand the applicability of deep RL [10, 32]. Prior works have used CNNs to extend RL algorithms like PPO [34], SAC [14], and Rainbow [19] to pixel-based tasks. Such direct extensions have typically required substantially larger number of environment interactions when compared to the state-based environments. In order to improve sample efficiency, recent efforts have studied the use of auxiliary tasks and loss functions [50, 27, 35], data augmentation [26, 25], and latent space
2Code: https://github.com/WendyShang/flare
dynamics modeling [16, 15]. Despite these advances, there is still a large gap between the learning efficiency in state-based and pixel-based environments in a number of challenging benchmark tasks. Our goal in this work is to identify where and how to improve pixel-based performance on this set of challenging control environments.
Neural Network Architectures in RL Mnih et al. [29] combined Q-learning with CNNs to achieve human level performance in Atari games, wherein Mnih et al. [29] concatenate the most recent 4 frames and use a convolutional neural network to output the Q values. In 2016, Mnih et al. [30] proposed to use a shared CNN among frames to extract visual features and aggregate the temporal information with LSTM. The same architectures have been adopted by most works to date [27, 35, 25, 26]. Recently, new architectures for RL have been explored that explore dense connections [39] as well as residual connections and instance norm [49]. However, the development of new architectures to better capture temporal information in a stream of images has received little attention in deep RL, and our work fills this void. Perhaps the closest to our motivation is the work of Amiranashvili et al. [1] who explicitly use optical flow as an extra input to the RL policy. However, this approach requires additional information and supervision signal to train the flow estimator, which could be unavailable or inaccurate in practice. In contrast, our approach is a simple modification to existing deep RL architectures and does not require any additional auxiliary tasks or supervision signals.
Two-Stream Video Classification In video classification tasks, such as activity recognition [40], there are a large body of works on how to utilize temporal information [9, 22, 44, 7, 47, 12]. Of particular relevance is the two-stream architecture of Simonyan and Zisserman [38], where one CNN stream takes the usual RGB frames, while the other the optical flow computed from the RGB values. The features from both streams are then late-fused to predict the activity class. That the two-stream architecture yields a significant performance gain compared to the single RGB stream counterpart, indicating the explicit temporal information carried by the flow plays an essential role in video understanding. Instead of directly computing the optical flow, we propose to capture the motion information in latent space to avoid computational overheads and potential flow approximation errors. Our approach also could focus on domain-specific motions that might be overlooked in a generic optical flow representation.

heading: 3 Background
text: Soft Actor Critic (SAC) [14] is an off-policy actor-critic RL algorithm for continuous control with an entropy maximization term augmented to its score function to encourage exploration. SAC learns a policy network πψ(at|ot) and critic networks Qφ1(ot, at) and Qφ2(ot, at) to estimate state-action values. The critic Qφi(ot, at) is optimized to minimize the (soft) Bellman residual error:
LQ(φi)=Eτ∼B [( Qφi(ot, at)− (rt + γV (ot+1)) )2] ,
where r is the reward, γ the discount factor, τ = (ot, at,ot+1, rt) is a transition sampled from replay buffer B, and V (ot+1) is the (soft) target value estimated by:
V (ot+1)=min i Qφ̄i(ot+1, at+1)− α log πψ(at+1|ot+1)],
where α is the entropy maximization coefficient. For stability, Qφ̄i is the exponential moving average of Qφi ’s over training iterations. The policy πψ is trained to maximize the expected return estimated by Q together with the entropy term
Lπ(ψ) = −Eat∼π [min i Qφi(ot, at)− α log πψ(at|ot)],
where α is also a learnable parameter.
Reinforcement Learning with Augmented Data, or RAD [26], is a recently proposed training technique. In short, RAD pre-processes raw pixel observations by applying random data augmentations, such as random translation or cropping, for RL training. As simple as it is, RAD has taken many existing RL algorithms, including SAC, to the next level. For example, on many DMControl [42] benchmarks, while vanilla pixel-based SAC performs poorly, RAD-SAC—i.e. applying data augmentation to pixel-based SAC—achieves state-of-the-art results both in sample efficiency and final performance. In this work, we refer RAD to RAD-SAC and use random translation as data augmentation.
Rainbow DQN is an extension of the Deep Q Network (DQN) [29], which combines multiple follow-up improvements of DQN to a single algorithm [19]. In summary, DQN [29] is an off-policy RL algorithm that leverages deep neural networks (DNN) to estimate the Q value directly from the pixel space. The follow-up works Rainbow DQN bring together to enhance the original DQN include double Q learning [17], prioritized experience replay [33], dueling network [48], noisy network [13], distributional RL [5] and multi-step returns [41]. Rainbow DQN is one of the state-ofthe-art RL algorithms on the Atari 2600 benchmark [3]. We thus adopt an official implementation of Rainbow [31] as our baseline to directly augment Flare on top.

heading: 4 Motivation
text: We motivate Flare by investigating the importance of temporal information in state-based RL. Our investigation utilizes 5 diverse DMControl [42] tasks. The full state for these environments includes both the agent’s pose information, such as the joints’ positions and angles, as well as temporal information, such as the joints’ translational and angular velocities.
First, we train two variants with SAC—one where the agent receives the full state as input (full-state SAC), and the other with the temporal information masked out, i.e. the agent only receives the pose information as its input (position-only SAC). The resulting learning curves are in Figure 2. While the full-state SAC learns the optimal policy quickly, the position-only SAC learns much sub-optimal policies, which often fail entirely. Therefore, we conclude that effective policies cannot be learned from positions alone, and that temporal information is crucial for efficient learning.
While full-state SAC can receive velocity information from internal sensors in simulation, in the more general case such as learning from pixels, such information is often not readily available. For this reason, we attempt to approximate temporal information as the difference between two consecutive states’ positions. Concretely, we compute the positional offset δt=(s p t−spt−1, spt−1−spt−2, spt−2−spt−3), and provide the fused vector (spt , δt) to the SAC agent. This procedure describes the state-based version of Flare. Results shown in Figure 2 demonstrate that state-based Flare significantly outperforms the position-only SAC. Furthermore, it achieves optimal asymptotic performance and a learning efficiency comparable to full-state SAC in most environments.
Given that the position-only SAC utilizes spt compared to Flare that utilizes s p t and δt, we also investigate a variant (stack SAC) where the SAC agent takes consecutive positions (spt , s p t−1, s p t−2, s p t−3). Stack SAC reflects the frame-stack heuristic used in pixel-based RL. Results in Figure 3 show that Flare still significantly outperforms stack SAC. It suggests that the well-structured inductive bias in the form of temporal-position fusion is essential for efficient learning.
Lastly, since a recurrent structure is an alternative approach to process temporal information, we implement an SAC variant with recurrent modules (Recurrent SAC) to compare with Flare. Specifically,
we pass a sequence of poses spt , s p t−1, s p t−2, s p t−3 through an LSTM cell. The number of the LSTM hidden units h is set to be the same as the dimension of δt in Flare. The trainable parameters of the LSTM cell are updated to minimize the critic loss. Recurrent SAC is more complex to implement and requires longer wall-clock training time, but performs worse than Flare as shown in Figure 3.
Our findings from the state experiments in Figure 2 and Figure 3 suggest that (i) temporal information is crucial to learning effective policies in RL, (ii) using Flare to approximate temporal information in the absence of sensors that provide explicit measurements is sufficient in most cases, and (iii) to incorporate temporal information via naively staking position states or a recurrent module are less effective than Flare. In the next section, we carry over these insights to pixel-space RL.

heading: 5 Reinforcement Learning with Latent Flow
text: To date, frame stacking is the most common way of pre-processing pixel-based input to convey temporal information for RL algorithms. This heuristic, introduced by Mnih et al. [29], has been largely untouched since its inception and is used in most state-of-the-art RL architectures. However, our observations from the experiments run on state inputs in Section 4 suggest an alternative to the frame stacking heuristic through the explicit inclusion of temporal information as part of the input. Following this insight, we seek a general alternative approach to explicitly incorporate temporal information that can be coupled to any base RL algorithm with minimal modification. To this end, we propose the Flow of Latents for Reinforcement Learning (Flare) architecture. Our proposed method calculates differences between the latent encodings of individual frames and fuses the feature differences and latent embeddings before passing them as input to the base RL algorithm, as shown in Figure 4. We demonstrate Flare on top of 2 state-of-the-art model-free off-policy RL baselines, RAD-SAC [26] and Rainbow DQN [19], though in principle any RL algorithm can be used in principle.

heading: 5.1 Latent Flow
text: In computer vision, the most common way to explicitly inject temporal information of a video sequence is to compute dense optical flow between consecutive frames [38]. Then the RGB and the optical flow inputs are individually fed into two streams of encoders and the features from both are fused in the later stage of the pipelinee. But two-stream architectures with optical flow are not as applicable to RL, because it is too computationally costly to generate optical flow on the fly.
To address this challenge and motivated by experiments in Section 4, we propose an alternative architecture that is similar in spirit to the two-stream networks for video classification. Rather than computing optical flow directly, we approximate temporal information in the latent space. Instead of encoding a stack of frames at once, we use a frame-wise CNN to encode each individual frame. Then we compute the differences between the latent encodings of consecutive frames, which we refer to as latent flow. Finally, the latent features and the latent flow are fused together through concatenation before getting passed to the downstream RL algorithm. We call the proposed architecture as Flow of Latents for Reinforcement Learning (Flare).
5.2 Implementation and Architecture Details
Table 1: Evaluation on 5 benchmark tasks around 500K and 1M environment steps. We evaluate over 5 seeds, each of 10 trajectories and show the mean ± standard deviation across runs. Flare substantially outperforms RAD on a majority (3 out of the 5) of environments, while being competitive in the remaining
1M STEPS 500K STEPS
TASK FLARE RAD CURL FLARE RAD CURL
QUADRUPED WALK 488± 99 322± 102 38± 10 296± 62 206± 50 39± 22 PENDULUM SWINGUP 809± 14 520± 144 151± 48 242± 68 79± 33 46± 207
HOPPER HOP 217± 26 211± 12 44± 3 90± 25 40± 18 10± 17 FINGER TURN HARD 661± 141 249± 44 598± 100 282± 30 137± 44 328± 91
WALKER RUN 556± 42 628± 17 350± 17 426± 18 547± 21 255± 9 AVERAGE RETURN 546 388 236 267 201 136
To address this challenge and motivated by experiments in Section 4, we propose an alternative181 architecture that is similar in spirit to the two-stream networks for video classification. Rather than182 computing optical flow directly, we approximate temporal information in the latent space. Instead of183 encoding a stack of frames at once, we use a frame-wise CNN to encode each individual frame. Then184 we compute the differences between the latent encodings of consecutive frames, which we refer to as185 latent flow. Finally, the latent features and the latent flow are fused together through concatenation186 before getting passed to the downstream RL algorithm. We call the proposed architecture as Flow of187 Latents for Reinforcement Learning (Flare).188
5.2 Implementation and Architecture Details189
Algorithm 1 Pixel-based Flare Inference Given πψ , fCNN for each environment step t do zj=fCNN(oj), j=t−k, .., t δj=zj−zj−1, j=t−k+1, .., t zt=(zt−k+1, · · ·, zt, δt−k+1, · · ·, δt) zt = LayerNorm(fFC(zt)) at∼πψ(at|zt) ot+1∼p(ot+1|at,ot = (ot, ot−1..ot−k)) We190 se-191 lect192 RAD193 as194 the195 base196 al-197 go-198 rithm199 to200 elab-201 o-202 rate the execution of Flare. Similar adaptations can be seamlessly applied to other RL algorithms203 such as Rainbow DQN. The RAD architecture, shown in Figure 4a, stacks multiple data augmented204 frames observed in the pixel space and encodes them altogether through an CNN. This can be viewed205 as a form of early fusion [23].206
Another preprocessing option is to encode each frame individually through a shared frame-wise207 encoder and perform late fusion of the resulting latent features, as shown in Figure 4b. However, we208 find that simply concatenating the latent features results in inferior performance when compared to209 the frame stacking heuristic, which we further elaborate in Section 6.3. We conjecture that pixel-level210 frame stacking benefits from leveraging both the CNN and the fully connected layers to process211 temporal information, whereas latent-level stacking does not propagate temporal information back212 through the CNN encoder.213
Based on this conjecture, we explicitly compute the latent flow δt = zt − zt−1 while detaching the214 zt−1 gradients when computing δt. We then fuse together (δt, zt). Next, since negative values in the215 fused latent embedding now possesses semantic meaning from δt, instead of ReLU non-linearity, we216 pass the embedding through a fully-connected layer followed by layer normalization, before entering217 the actor and critic networks as shown in Figure 4c. Pseudocode illustrates inference with Flare in218 Algorithm 1; during training, the encodings of latent features and flow are done in the same way219 except with augmented observations.220
6
We select RAD as the base algorithm to elaborate the execution of Flare. Similar adaptations can be seamlessly applied to other RL algorithms such as Rainbow DQN. The RAD architecture, shown in Figure 4a, stacks multiple data augmented frames observed in the pixel space and encodes them altogether through an CNN. This can be viewed as a form of early fusion [24].
Another preprocessing option is to encode each frame individually through a shared frame-wise encoder and perform late fusion of the resulting latent features, as shown in Figure 4b. However, we find that simply concat nating the latent fea-
tures results inferior performance when compar d to the frame stacking heuri tic, which we further elaborate in Sectio 6.3. W conjecture that pixel-level frame stacking benefits from leveraging both the CNN and the fully connected layers to process temporal information, whereas latent-level stacking does not propagate temporal information back through the CNN encoder.
Based on this conjecture, we explicitly compute the latent flow δt = zt − zt−1 while detaching the zt−1 gradients when computing δt. We then fuse together (δt, zt). Next, since negative values in the fused latent embedding now possesses semantic meaning from δt, instead of ReLU non-linearity, we pass the embedding through a fully-connected layer followed by layer normalization, before entering the actor and critic networks as shown in Figure 4c. Pseudocode illustrates inference with Flare in Algorithm 5.2; during training, the encodings of latent features and flow are done in the same way except with augmented observations.

heading: 6 Experiments
text: In this section, we first present the main experimental results, where we show that Flare achieves substantial performance gains over the base algorithm RAD [26] and Rainbow DQN [19]. Then we conduct a series of ablation studies to stress test the design choices of the Flare architecture.

heading: 6.1 Environments and Evaluation Metrics
text: The DeepMind Control Suite (DMControl) [42], based on MuJoCo [43], is a commonly used benchmark for continuous control from pixels. On simpler environments in the suite, prior works [25, 26] have made substantial progress on this benchmark and closed the gap between state-based and pixel-based efficiency. However, on more challenging environments that feature partial observability, sparse rewards, or precise manipulation, these algorithms struggle to learn optimal policies efficiently . In this work, we focus on 5 of these more challenging tasks. The 5 environments include Walker
Run (requires maintaining balance with speed), Quadruped Walk (partially observable agent morphology), Hopper Hop (locomotion with sparse rewards), Finger Turn-hard (precise manipulation), and Pendulum Swingup (torque control with sparse rewards). For evaluation, we benchmark performance at 500K and 1M environment steps and compare against RAD.
The Atari 2600 Games [3] is another highly popular RL benchmark. Recent efforts have led to a range of highly successful algorithms [11, 19, 23, 15, 2] to solve Atari games directly from pixel. A representative state-of-the-art is Rainbow DQN (see Section 3). We adopt the official Rainbow DQN implementation [31] as our baseline and simply incorporate Flare while retaining all the other default settings, including hyperparameters and preprocessing. Note that the baseline Rainbow DQN’s model architecture is also modified to the most comparable setup to that of Flare as described in Section 5.2, including increasing the number of last layer convolutional channels (to match the number of parameters) and adding a fully-connected layer plus layer normalization before the Q networks. We evaluate on a diverse subset of Atari games at 100M training steps, namely Assault, Breakout, Freeway, Krull, Montezuma Revenge, Seaquest, Up n Down and Tutankham, to assess the effectiveness of Flare.
Evaluation and Training Protocol It is a common symptom in RL that evaluations appear noisy. To ensure the most fair presentation of results, we follow this protocol: the main results in Section 6.2 report the mean over 5 random seeds with standard error, a standard RL practice [42, 31]. Furthermore, we use the same 5 seeds for both the baseline and Flare.

heading: 6.2 Main Results
text: We present the main results of comparing Flare against the baselines, namely RAD and Rainbow DQN. Flare outperforms the baselines on the majority of the environments. It is worth noting that since these baselines already produce state-of-the-art level performances, any steady improvement under our rigorous experimental protocol—even when the gain seems minor —is significant.
DMControl: Our main experimental results on the 5 DMControl tasks are presented in Table 1. We find that Flare outperforms RAD in terms of both final performance and sample efficiency for majority (3 out of 5) of the environments, while being competitive on the remaining environments. Specifically, Flare attains similar asymptotic performance to state-based RL on Pendulum Swingup, Hopper Hop, and Finger Turn-hard. For Quadruped Walk, a particularly challenging environment due to its large action space and partial observability, Flare learns much more efficiently than RAD and achieves a higher final score. Moreover, Flare outperforms RAD in terms of sample efficiency on all of the core tasks except for Walker Run. The 500k and 1M environment step evaluations in Table 1 show that, on average, Flare achieves 1.4× and 1.3× higher scores than RAD at the 1M step and the 500K step benchmarks, respectively.
Atari: The results on the 8 Atari games are in Table 2. Again, we observe substantial performance gain from Flare on the majority (5 out of 8) of the games, including the challenging Montezuma’s Revenge. On most of the remaining games, Flare is equally competitive except for Seaquest. In the appendix, we also show that Flare performs competitively when comparing against other DQN variants at 100M training steps, including the original Rainbow implementations.

heading: 6.3 Ablation Studies
text: We ablate a number of components of the Flare architecture on the Quadruped Walk and Pendulum Swingup environments to stress test the Flare architecture. The results shown in Figure 5 aim to answer the following questions:
Q1: Do we need latent flow or is computing pixel differences sufficient? A1: Flare proposes a late fusion of latent differences with the latent embeddings, while a simpler approach is an early fusion of pixel differences with the pixel input, which we call pixel flow. We compare Flare to pixel flow in Figure 5 (left) and find that pixel flow is above RAD but significantly less efficient and less stable than Flare, particularly on Quadruped Walk. This ablation suggests that late fusion temporal information after encoding the image is preferred to early fusion.
Q2: Are the gains coming from latent flow or individual frame-wise encoding? A2: We address the potential concern that the performance gain of Flare stems from the frame-wise ConvNet architectural modification instead of the fusion of latent flow. Concretely, we follow the exact architecture and training as Flare, but instead of concatenating the latent flow, we concatenate each frame’s latent vector after the convolution encoders directly as described in Figure 4b. This ablation is similar in spirit to the state-based experiments in Figure 3. The learning curves in Figure 5 (center) show that individual frame-wise encoding is not the source of the performance lift: framewise encoding, though on par with RAD on Pendulum Swingup, performs significantly worse on Quadruped Walk. Flare’s improvements over RAD are hence most likely thanks to the explicit fusion of latent flow.
Q3: How does the input frame count affect performance? A3: We compare stacking 2, 3, and 5 frames in Flare in Figure 5 (right). We find that changing the number of stacked frames does not significantly impact the locomotion task, quadruped walk, but Pendulum Swingup tends to be more sensitive to this hyperparameter. Interestingly, the optimal number of frames for Pendulum Swingup is 2, and more frames can in fact degrade Flare’s performance, indicating the immediate position and velocity information is the most critical to learn effective policies on this task. We hypothesize that Flare trains more slowly with increased frame count on Pendulum Swingup due to the presence of unnecessary information that the actor and critic networks need to learn to ignore.
Q4: Do we need latent flow or is RNN over latent sufficient? A4: Another approach of latent fusion would be applying a recurrent neural network on the latent embeddings. We compare FLARE with LSTM baselines on DMControl. We found that RNNs
perform worse than FLARE (see Fig 6), which is in agreement with our findings over the poor performance of RNN on coordinate (i.e. compact) state representations in Section 4.

heading: 7 Broader Impacts and Limitations
text: Conclusion We propose Flare, an architecture for RL that explicitly encode temporal information by computing flow in the latent space. In experiments, we show that in the state space, Flare can recover the optimal performance with only state positions and no access to the state velocities. In the pixel space, Flare improves upon the state-of-the-art model-free RL algorithms on the majority of selected tasks in the DMControl and Atari suites, while matching in the remaining. All code assets used for this project came with MIT licenses. For more details, we refer the reader to the appendix.
Limitations Flare is a general approach to improve RL algorithms on many environments and tasks but not the panacea to single-handedly solve all. For instance, learning to control humanoid from pixels remains a challenge
even when augmenting RAD with Flare. Also, RAD without Flare in fact is preferred for Seaquest,
one of the Atari games. An additional limitation is that Flare is only useful if temporal information such as velocities is visible in the input pixel images and may not work as well for partially observed environments. Finally, Flare was only tested on model-free algorithms and it would be informative to investigate its applicability in the model-based regime, which we leave for future work.
Broader Impacts Simple architectures that improve performance can be impactful due to their ease of use and wide applicability. Prime examples of such innovations in vision are ResNets [18] which have been widely adopted. A potential negative consequence of Flare and supervised RL algorithms in general is that they rely on hand-designed reward functions which can be exploited. For example, in the Quadruped task Flare learns a more optimal policy than prior model-free methods but the resulting policy is extremely jittery since the simulator does not penalize jerk. This policy
would likely wear out the joints if deployed on real robot. To make architectures like Flare and RL in general more applicable to real-world scenarios it would be useful to investigate how to generate safe policies perhaps through constrained optimization or offline RL.

Review summaries:
This work presents a simple technique (Flare) to incorporate explicit temporal information to enable effective RL policy learning in challenging continuous control environments using pixel-based state representations. The approach is inspired from recent advances in the video recognition approaches which employ optical flow information and late fusion to incorporate temporal information. Typically, RL algorithms employ a frame-stacking heuristic to incorporate temporal information (early fusion). Though computing optical flow is slow and can been prohibitive for real-time applications, the authors present a simple alternative to using optical flow, i.e. difference of latent state vectors as a proxy for explicitly encoding motion information with latent vectors representing the observed state (late fusion). Experimental results on challenging continuous control tasks in the DMControl Suite show that Flare can achieve up to 1.9x higher scores than a baseline algorithm (RAD) which uses the frame-stacking heuristic to incorporate temporal information.

The paper brings very little novelty or insight. It is unclear that the introduced architecture complexity worth marginal improvements (given high variance and only 5 random seeds) on 2 out of 11 tasks (5 from the main paper and 6 from appendix). This might be a good workshop paper but it clearly does not meet the high acceptance threshold of ICLR.

In this paper, the authors modify two state-of-the-art reinforcement learning algorithms (RAD-SAC and Rainbow-DQN) by changing the way information from multiple timesteps is aggregated. Instead of stacking frames at a pixel level, the authors propose to encode each frame individually and then pass the encodings and their pairwise differences to the actor/critic networks. It is shown that when learning policies from pixels this way of fusing information substantially outperforms the baseline variants in 3/5 environments from the DeepMind control suite and 5/8 environments from Atari2600. Furthermore, experiments with low-level observations on 5 DeepMind Control Suite tasks support the particular choice of information fusion over using recurrent networks or stacking states directly without including state differences. 

This paper proposes adding temporal information in the form of latent flow to deep RL agents. The idea is simple: temporal differences between latent embeddings of consecutive frames are computed; the differences are then concatenated to the embeddings for downstream processing by an RL controller. This idea is inspired by idea of late fusion in video processing and stands in contrast to the current approach of frame-stacking, which is more akin to early fusion. The authors have validated their approach on a series of DMC tasks as well as Atari games.

The paper proposes a simple modiﬁcation to existing deep RL architectures when the input is a sequence of frames. It makes use of a specific encoding of the frames and a specific way of combining CNN and fusion of the information from the different frames. This architecture does not require any additional auxiliary tasks or supervision signals as compared to traditional RL losses. The results of the paper show that the developed approaches are significantly better than baselines.