Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Authors: Denis Yarats
Rob Fergus
Alessandro Lazaric
Lerrel Pinto
Venue: ICLR 2022 Poster (2022)
Keywords: Image-based RL
Data augmentation in RL
Continuous Control

Abstract: We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control. DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmentation to learn directly from pixels. We introduce several improvements that yield state-of-the-art results on the DeepMind Control Suite. Notably, DrQ-v2 is able to solve complex humanoid locomotion tasks directly from pixel observations, previously unattained by model-free RL. DrQ-v2  is conceptually simple, easy to implement, and provides significantly better computational footprint compared to prior work, with the majority of tasks taking just 8 hours to train on a single GPU. Finally, we publicly release DrQ-v2 's implementation to  provide RL practitioners with a strong and computationally efficient baseline.

Under review as a conference paper at ICLR 2022
MASTERING VISUAL CONTINUOUS CONTROL: IMPROVED LEARNING
CEMENTTTEMENT
Anonymous authors Paper under double-blind review
ABSTRACT
We present DrQ-v2, model-free reinforcement learning (RL) algorithm for visual continuous control. DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmentation to learn directly from pixels. We introduce several improvements that yield state-of-the ofte-offthhe-rt results on the DeepMind Control Suite. Notably, DrQ-v2 is able to solve complex humanoid locomotion tasks directly from pixel observations, previously unattained by model-free RL. DrQ-v2 is conceptually simple, easy implement, and provides significantly better computational footprint compared to prior work, with the majority of tasks taking just 8 hours to train on a single GPU. Finally, DrQ-v2's implementation publicly released to provide RI. practitioners with strong and computationally efficient baseline.
INTRODUCTION
Creating sample-cfficient continuous control methods that observe high-dimensional images has been a long standing challenge in reinforcement learning (RL) Over the last three years, the RL community has made significant headway this problem, improving sample-efficinncy significantly. The key insight to solving visual control is the learning of better low-dimensional representations, either through autoencoders (Yarats et al., 2019; Finn al., 2015), variational inference (Hafner et al., 2018; 2019; Lee et al., 2019), contrastive learning (Srinivas et al., 2020; Yarats et al., 2021 a). self-prediction (Schwarzer et al., 2020b), or data augmentations (Yarats al., 2021b; Laskin et al., 2020). However, current state-of-the tht-hherr art model-free methods are still limited in three ways. First, they are unable to solve the more challenging visual control problems such as quadruped and humanoid locomotion. Second, they often require significant computational resources, i.e. lengthy training times using distributed multi-GPP infrastructure. Lastly, it is often unclear how different design choices affect overall system performance.
12 DMC Tasks (Samples)
12 DMC Tasks (Time)
Humanold Walk (Samples)
Humanold Walk (Time)
Egisode Retum
800-
Episade Retur
800
Episode weturn
Epicode Return
sen
600
400
400
400-
200
200
200
0
20
200
Frames (x10F)
(for in Frames)
Frames (x11)")
(for 30" Frames)
SAC
CURL DrQ
DrQ-v2 (Ours)
Figure 1: DrQ-v2 demonstrates significantly better sample efficiency and computational footprint compared to state-of-the-art model-free methods for visual continuous control while being conceptu- ally simple and easy to implement. (Left two) Average performance results across 12 challenging tasks from the Deep Control Suite (the set of tasks can be seen in Figure 8). (Right two) Performance on the Humanoid Walk task from visual input, previously unsolved by model-free methods. In both cases we report sample complexity and wall-clock time axes for evaluation, with time being measured single GPU machine and using official implementations for each method.

---

Under review as a conference paper at ICLR 2022
In this paper we present DrQ-v2, simple model-free algorithm that builds the idea of using data augmentations (Yarats et al., 2021b; Laskin et al., 2020) to solve hard visual control problems. Most notably, it is the first model-free method that solves complex humanoid tasks directly from pixels. Compared to previous state-of-the-arr model-free methods, DrQ-v2 provides significant improvements in sample efficiency across tasks from the DeepMind Control Suite (Tassa al., 20. Conceptually simple, DrQ-v2 is also computationally efficient, which allows solving most tasks in DeepMind Control Suite in just hours on single GPU (see Figure 1). Recently, a model-based method, DreamerV2 (Hafner et al., 2020) was also shown to solve visual continuous control problems and it was first to solve the humanoid locomotion problem from pixels. While our model- free DrQ-v2 matches DreamerV2 in terms sample efficiency and performance, it does so 4x faster in terms of wall-clock time to train. We believe this makes DrQ-v2 a more accessible approach to support research in visual continuous control and it reinforces the question on whether model-free or model-based is the more suitable approach to solve this type of tasks.
DrQ-v2, which is detailed in Section 3, improves upon DrQ (Yarats et al., 2021b) by making several algorithmic changes: (i) switching the base RL algorithm from SAC (Haarnoja et al., 2018a) to DDPG (Lillicrap et al., 2015a) with clipped double Q-learning from TD3 (Fujimoto et al., 2018), (ii) this allows us straightforwardly incorporating multi-step return, (ii) adding bilinear interpolation to the random shift image augmentation, (iv) introducing an exploration schedule, (v) selecting better hyper-parameters including a larger capacity of the replay buffer. A careful ablation study of these design choices is presented in Section 4.4, Furthermore, we re-examine the original implementation of DrQ and identify several computational bottlenecks such as replay buffer management, data augmentation processing, batch size, and frequency of learning updates (see Section 3.2). To remedy these, we have developed new implementation that both achieves better performance and trains around 3.5 times faster with respect to wall-clock time than the previous implementation on the same hardware with an increase in environment frame throughput (FPS) from 28 to 96 (i.e., it takes 10 /96/3600 x 2.9 hours to train for 1M environment steps). DrQ-v2's implementation is available at https: / /ttp///nnnnnoosa//////// 40e.:...pppo.. www.n.sceeene.....oooo.p.oooeeee www.sceeeeeeeec.........e.eeeeee /tannce////// /rrqv2.
2
BACKGROUND
2.1 REINFORCHMENT LEARNING FROM IMAGES
We formulate image- based control an infinite-horize Markov Decision Process (MDP) (Bellman, 1957). Generally, in such a setting, an image rendering of the system is not sufficient to perfectly describe the system's 's underlying state. To this end and per common practice (Mnih et al., 2013), we approximate the current state of the system by stacking three consecutive prior observations. With this in mind, such MDP can be described as tuple (X,,,, P, R, 7, do), where X' is the state space (a three-stack of image observations). AA is the action space, P :: X' x A + ( is the transition function' that defines probability distribution over the next state given the current state and action, R R: X x A + [0, 1] is the reward function, E [0, 1) is discount factor, and da E A(X) is the distribution of the initial state Ty- The goal is find policy x X' - ((A) that maximizes the expected discounted sum of rewards E,, (2 7rr,), where Do do. and VL we have (-(2x)) 2441 P(- at), and r, = R(x,, az).
2.2 DEEP DETERMINISTIC POLICY GRADIENT
Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015a) is an actor-critic algorithm for continuous control that concurrently learns a Q-function Qo and a deterministic policy т&. For this, DDPG uses Q-learning (Watkins and Dayan, 1992) to learn Qu by minimizing the one-step Bellman residual Jo (D) = Ee (...........244.3 )> (app)>/(((000 [[Qa ax) Tt 7 g T (2( The policy πæ is learned by employing Deterministic Policy Gradient (DPG) (Silver al., 2014) and maximizing (D( = Ex (Qe (It, Ta (x,) so Tt (x;) approximates argmax, Q0( a). Here, D is a replay buffer of environment transitions and 0 is an exponential moving average of the weights. DDPG is amenable to incorporate n-step returns (Watkins, 1989; eng and Williams, 1996) when estimating TD error beyond single step (Barth-Maron et al., 2018). In practice, n-step returns allow for faster
Here, ((X)) denotes a distribution over the state space X.
2

---

Under review as a conference paper at ICLR 2022
tt
Environment
at
Humanoid Walk
DrQ-v2
Humanoid Stand
Figure 2: (Left): DrQ-v2 is an off-policy actor-critic algorithm for image-based RL. It alleviates encoder overfitting by applying random shift augmentation to pixel observations sampled from the replay buffer. (Right): Examples of walking and standing behaviors learned by DrQ-v2 for complex humanoid agent from DMC (Tassa et al., 2018) with 21 and 54 dimensional action and state spaces, respectively. DrQ-v2 does not have access to the internal state of the environment, only observing three consecutive pixel frames at a time. Despite this imperfect observational channel, our agent still manages solve the tasks. To the best of our knowledge. this is the first successful demonstration by a model-free method, using pixel-based inputs of these tasks.
reward propagation and has been previously used in policy gradient and Q-learning methods (Mnih et al., 2016b; Barth-Maron et al., 2018; Hessel et al., 2017).
2.3 DATA AUGMENTATION IN REINFORCEMENT LEARNING
Recently, has been shown that data augmentation techniques, commonplace Computer Vision, are also important for achieving the state-of-the-e performance in image-based RL (Yarats et al., 2021b; Laskin et al., 2020). For example, the state-of-the the-art algorithm for visual RL, DrQ (Yarats et al., 2021b) builds on top of Soft Actor-Critic (Haarnjj et al., 2018a), model-free actor-critic algorithm, by adding convolutional encoder and data augmentation the form of random shifts. forms essential component of several recent visual RL
The of such data augmentations algorithms (Srinivas et al., 2020; Raileanu et al., 2020; Yarats et al., 2021a; Stooke et al., 2020; Hansen and Wang, 2021; Schwarzer et al., 2020b).
D2Q- IMPROVED DATA AUGMENTED REINFORCEMENT LEARNING
In this section, we describe DrQ-v2, simple model-free actor-critic RL algorithm for image-based continuous control, that builds upon DrQ.
3.1 ALGORITHMICC DETAILS
Image Augmentation As in DrQ we apply random shifts image augmentation to pixel observations of the environment. In the settings of visual continuous control by DMC, this augmentation can be instantiated by first padding each side 84 x observation rendering by pixels (by repeating boundary pixels), and then selecting random 84 84 crop, yielding the original image shifted by +4 pixels. We also find it useful to apply bilinear interpolation on top of the shifted image (i.c, we replace each pixel value with the average of the four nearest pixel values). In experiments, this modification provides an additional performance boost across the board.
Image Encoder The augmented image observation is then embedded into a -iw-imenniionl latent vector by applying a convolutional encoder. We use the same encoder architecture as in DrQ, which first was introduced introduced in SAC-AE (Yarats al., 2019). This process can succinetly summarized as h fe (aug(a))) where fg is the encoder, aug is the random shifts augmentation, and æ is the original image observation.
Actor-Critic Algorithm We DDPG (Lillicrap al., 2015a) backbone actor-critic RL algorithm and, similarly to Sarth-Marow et al. (2018), augment with n-step returns to estimate TD emor.. This results into faster reward propagation and overall learning progress (Mnih al., 2016a).
3

---

Under review as a conference paper at ICLR 2022
While some methods (Hafner et al., 2020) employ more sophisticated techniques such TD(A) or Retrace(A) (Munos et al., 2016), they are often computationally demanding when n large. We find that using simple n-step returns, without an importance sampling conrection, strikes good balance between performance and efficiency. We also employ clipped double Q-learning (Fujimoto et al., 2018) reduce overestimation bias in the target value. Practically, this requires training two Q- functions Qe, and Q2. For this, we sample a mini-batch of transitions T = (æt, ar 11 P(taa) from the replay buffer D and compute the following two losses:
Le i (D) Er~D ((Q6, (h,,, ar) - 2) Vk E {1,2),
(1)
with the TD target defined as:
y rese + 7"h min Qā (he+as actn), k=1,2
where he - fg ((aug(a:)): hi-n- fc (aug((:+4))) )), aeen T To + E, 02 and 02 are the slow- moving weights for the Q target networks. We note, that contrast to DrQ, we do not employ a target network for the encoder fę and always use the most recent weights & to embed æ; and T++n- The exploration noise is sampled from clip(A((O, o²), -c, e) similar to TD3 (Fujimoto et al., 201 8), with the exception of decaying o, which we describe below. Finally, we train the deterministic actor To using DPG with the following loss:
(D) -EROND [ min Qu (h,, ah), k=1,2
where h, (aug(a)))) a, = NH (h+) c, and clip(N o2), -c, - Similar to DrQ, do not use actor's 's gradients to update the encoder's parameters &.
Scheduled Exploration Noise Empirically, we observe that is helpful to have different levels of exploration at different stages of learning. At the beginning of training we want the agent to be more stochastic and explore the environment more effectively, while at the later stages of training, when the agent has already identified promising behaviors, it is better to be more deterministic and master those behaviors. Similar to Amos et al. (2020), we instantiate this idea by using linear decay (() for the variance o² of the exploration noise defined as:
(L) Dinit + (1 - min( L 1))(aannll)) - -Oinit);
where iitt and Sinal are the initial and final values for standard deviation, and T is the decay horizon.
Key Hyper- Parametrrs We conduct an extensive hyper-paramett search and identify several hyper-parameter changes compared to DrQ. The three most important syper-parameters are: (i) the size of the replay buffer, (ii) mini-batch size, and (iii) learning rate. Specifically, larger replay buffer than DrQ. We also use a smaller mini-batch size of 256 without any noticeable performance degradation. This is in contrast to CURL (Srinivas et al., 2020) and DrQ (Yarats et al., 2021b) that both use a larger batch size of 512 to attain more stable training in the expense of computational efficiency. Finally, we find that using smaller learning rate of x 10-4, rather than DrQ's learning rate of x 10-3, results into more stable training without any loss in learning speed.
use 10 times
3.2 IMPLEMENTATION DETAILS
Faster Image Augmentation We replace DrQ's random shifts augmentation (i.e., kornia. atgmentatioon RandomCrop) by a custom implementation that uses flow- field image sampling provided in PyTorch (i.e., sprid_sample we noticed that Kornia's 's implementation does not fully utilize GPU pipelining since it has some intermediate CPU to GPU data transferring which breaks the computational flow. Second, using grid__sanae le allows straightforward addition of bilinear interpolation. Our custom random shifts augmentation improves training throughput by a factor of 2.
This is done for two reasons. First,
4

---

Under review as a conference paper at ICLR 2022
Faster Replay Buffer Another computational bottleneck of DrQ was the replay buffer. The specific implementation had poor memory management which resulted in slow CPU to GPU data transfer, which also restricted the number of image-based based transitions that could be stored. We reimplemented the replay buffer to address these issues which led to a ten-fold increase in storage capacity and faster data transfer. More details are available in our open-source release. We note that the improved training speed of DrQ-v2 was key to solving humanoid tasks as it enabled much faster experimentation.
EXPERIMENTS
In this section we provide empirical evaluation of DrQ-v2 on an extensive set of visual continuous control tasks from DMC (Tassa et al., 2018). We first present comparison to prior methods, both model-free and model-based, in terms of sample efficiency and wall-clock time. We then present a large scale ablation study that guided the final version of DrQ-v2.
4.1 SETUP
Environments We consider a set of MuJoCo tasks (Todorov et al., 2012) provided by DMC (Tassa et al., 2018), widely used benchmark for continous control. DMC offers environments of various difficulty, ranging from the simple control problems such as the single degree of freedom (DOF) pendulum and cartpool, to the control of complex multi-joint bodies such the humanoid (21 DOF). We consider learning from pixels. In this setting, environment observations are stacks 3 consecutive RGB images of size 84 x 84, stacked along the channel dimension to enable inference of dynamic information like velocity and acceleration. In total, we consider 24 different tasks, which we group into three buckets, easy, medium, and hard, according the sample complexity to reach near-optimal performance (see Appendix B). Our motivation for this is to encourage RL practitioners to focus on the medium and hard tasks and stop using the easy tasks for evaluation, as they are mostly solved at this point and may no longer provide any valuable signal in comparing different methods.
Training Details For all tasks in the suite an episode corresponds to 1000 steps, where per-step reward is in the unit interval [0, 1]. This upper bounds the episode return to 1000 making it easier to compute aggregated performance measures across tasks. To facilitate fair wall-clock time comparison all algorithms are trained on the same hardware (i.e., a single NVIDIA V100 GPU machine) and evaluated with the same periodicity of 20000 environment steps. Each evaluation query averages episode returns over 10 episodes. Per common practice (Hafner et al., 2019), we employ action repeat of 2 and measure sample complexity in the environment steps, rather than the actor steps. In all the figures we plot the mean performance over 10 seeds together with the shaded regions which represent 95% confidence intervals. A full list of papee-pprameterss can be found in Appendix E.
Comparison Axes In many real-world applications, taking a step in the environment incurs sig- nificant computational cost making sample efficiency a critical feature of an RL algorithm. It is hence important to compare RL algorithms in terms of their sample efficiency. We facilitate this comparison by computing an algorithm's 's performance measured by episode return with respect to environment steps. On the other end, striving low sample complexity often comes at the cost of a poor computational efficiency. Unfortunately, recent deep RL literature has paid very little attention to this important axis, which has led to skyrocketing hardware requirements. Such a trend has made it virtually impossible for an RL practitioner with modest hardware capacity to contribute to advance- RL, leaving research in this area to few well-equipped labs. To democratize research in visual RL, we additionally propose to compare the agents in terms of wall-clock training time given the same single GPU hardware. We note that it is possible to adapt DrQ-v2 to a distributed setup, as has been done for DDPG in prior work (Barth-Maron et al., 2018; Hoffman al., 2020).
ments in image-basce
4.2 COMPARISON TO MODEL-FREE METHIOSS
Baselines We compare our method to several state-of sfate-oftthe-ee model-fres algorithms for visual RL including CURL (Srinivas et al., 2020), DrQ (Yarats et al., 2021b), and vanilla SAC (Haarnjj et al., 2018a) augmented with the convolutional encoder from SAC-AE (Yarats et al., 2019). Vanilla SAC is weak baseline and only included as ground point to showcase the recent progress in visual RL.
5

---

Under review as a conference paper at ICLR 2022
Acrobot Swingup
Cheetah
Finger Turn Hard
Hopper Hop
1800
Episade Return
-n0
Episode Return
800
600
Episade Return
750
Episada Return
900
200
k00
500
200
200
250
100
0.0
Frames (H10F)
le6
Frames (x10f)
106
Frames (xiof)
Los
(x10")
166
Humanold Walk
Quadruped Run
Reach Duplo
Reacher Hard
Episade Return
Episode Return
100
Epicode Return
200
1000
600
600
400
100
Episode
500
200
200
250
10
20
30
Frames (x10F)
Frames (xi1f)
Frames (x109)
Frames (x109)
1c6
(a) Sample Efficiency.
Acrobot Swingup
Cheetah Run
Finger Tum Hard
Hopper Hop
Episade Return
200
Episode Return
800
Episodle Return
750
600
600
200
40c
500
Eplcade
200
200
105 tour
250
100
20
40
20
40
20
20
40
Hours (for a 10' Frames)
Hours (for x 10f Frames)
Hours (for 3 10t Frames)
Hours (for J 20 Fremes)
Humanold Walk
Quadruped Run
Reach Duplo
Reacher Hard
Episodee Return
200
Episade Return
80c
Epicade Return
200
Episople Return
1000
600 snc
750
400
100-
aous
500
200
200
40
(for 30 Frames)
Hours (for 3 10 Frames)
Hours (for 10f Frames)
(for Frames)
(b) Wall-clock Time.
SAC
CURL
DrQ
DrQ-v2 (Ours)
Figure 3: We compare DrQ-v2 subset of continuous control tasks that offer various challenges, including complex dynamics, sparse rewards, hard exploration, and more. (a) DrQ-v2 demonstrates favorable sample efficiency and comfortably outperforms leading model-free baselines, as well as requiring less wall-clock training image (b).
Sample Efficiency Axis We present results on several medium and hard tasks Figure 3a. Full results can be found in Appendix (Figure 6, Figure 8. and Figure 10). Our empirical study reveals that DrQ-v2 outperforms prior model-free methods in terms of sample efficiency across the three benchmarks with different levels of difficulty. Importantly, DrQ-v2's advantage is more pronounced on harder tasks (i.e., acrobot, quadruped, and humanoid), where exploration is especially challenging. Finally, DrQ-v2 solves the DMC humanoid locomotion tasks directly from pixels, which, the best of our knowledge, is the first successful demonstration of such feat by a model-free method.
Compute Efficiency Axis To facilitate a fair comparison in terms of sheer wall-clock training time, besides employee the identical training protocol (see Section 4.1), also use the same mini-batch size of 256 for each agent. In Figure 13, we evaluate DrQ-v2 on a subset of DMC tasks for the sake of brevity only, and note that the demonstrated results can be easily extrapolated the other tasks given the linear dependency between training time and sample complexity. In our benchmarks. DrQ-v2 is able achieve throughput of 96 FPS, which favorably compares to DrQ's 28 FPS 3.4x increase), and CURL's 16 FPS (a 6x increase) throughputs. Practically, DrQ-v2 solves easy, medium, and hard tasks within 2.9, 8.6, and 86 hours respectively. Full results can be found in Appendix (Figure 7, Figure 9, and Figure 11).
4.3 COMPARISON TO MODEL-BASED METHODS
Baseline To see how DrQ-v2 stacks up against model-based methods, which tend to achieve better sample complexity in expense of larger computational footprint, also compare recent and

---

Under review as a conference paper at ICLR 2022
Acrobot Swingup
Quadruped Run
Reacher Hard
Humanoid Walk
Esioade Relurn
Epioade Return
1000
Episode Return
1000
800
750
750
600
500
500
Ealcodo
400
250
230
200
Frames (x10F)
Frames (xtef)
Frames (x1of)
Frames (x:09)
(a) Sample Efficieney.
Acrobot Swingup
Quadruped Run
Reacher Hard
Humanoid Walk
Esionde Return
Epicade Return
Egisode Return
800-
600-
750
750-
Salsad Beome
400-
500
5000
400-
200
250
250-
200
hours
10
20
20
30
10
30
1.00
200
300
Hours (for S 10f Frames)
Hours (for 3 10f Frames!
Hours (for 3x 10 Frames)
Hours (for 30 10 Frames)
(b) Wall-clock Time.
Dreamer-v2
DrQ-v2 (Ours)
Figure 4: Model-based Dreamer-v2 needs to train a world model and thus performs more computations during training than model-free DrQ-v2. Still, (a) DrQ-v2 is able to match Dreamer-v2' 's sample efficiency, while (b) requiring much less wall-clock training time.
unpublished improvements to Dreamer-v2 (Hafner et al., 2020), a leading model-based approach for visual continuous control. The recent update shows that the model-based approach can solve the DMC humanoid tasks directly from pixel inputs. The open-soure implementation of Dreamer-v2 /ottps://gi hubb com/ hon/danija jar/ /dremme rv2) only provides learning curves for Humanoid Walk. For this reason we run their code to obtain results on other DMC tasks. To limit hardware requirements of compute-expensivy Dreamer-v2, we only run on subset of 12 out of 24 considered tasks. This subset, however, overlaps with all the three (i.e. easy, medium, and hard) benchmarks.
Sample Efficiency Axis Our empirical study in Figure 4a reveals that in many cases, DrQ-v2, despite being a model-free method, can rival sample efficiency of ofate-of-the-art model-based Dreamer-v2. We note, however, that several tasks (for example Acrobot Swingup) Dreamer-v2 outperforms DrQ-v2. We leave investigation of such discrepancy for future work. Full results are provided in Appendix D (Figure 12).
Compute Efficiency Axis A different picture emerges if comparison is done with respect to wall- clock training time. Dreamer-v2, being a model-based method, performs significantly more floating point operations to reach its sample efficiency. In our benchmarks, Dreamer-v2 records a throughput of 24 FPS, which is 4x less than DrQ-v2's 's throughput of 96 FPS, measured on the same hardware. In Figure 4b we plot learning curves against wall-clock time and observe that DrQ-v2 takes less time to solve the tasks. Full results can be found in Appendix (Figure 13).
4.4 ABLATION STUDY
In this section present an extensive ablation study that guided us to the final version of DrQ-v2. Here, for brevity we only discuss experiments that were most impactful and omit others that did not pan out. For computational reasons, we only ablate on 3 different control tasks of various difficulty levels. Our findings are summarized in Figure 5 and detailed below.
Switching from SAC to DDPG DrQ (Yarats et al., 2021b) leverages SAC (Haarnoja et al., 201 as the backbone RL algorithm. While it has been demonstrated by many works, including the original manuscripts (Haarnoja et al., 201 8a;b) that SAC is superior to DDPG (Lillicrap et al., 2015b), our careful examination identifies two shortcomings that preclude SAC (within DrQ) to solve hard exploration-wii wise image- magee-based tasks. First, the automatic entropy adjustment strategy, introduced in Haarnoja et (2018b), is inadequate and in some cases leads to a premature entropy collapse.
Arxiv v3 revision from May 3, 2021 introduces new result on the Humanoid Walk task in Appendix D.
7

---

Under review as a conference paper at ICLR 2022
Acrobot Swingup
Quadruped Walk
Reacher Hard
Episode Return
Episode Return
800
Episode Return
600
200
400
100
200
Frames (x110)
Frames (x10*)
Frames (x10f)
DiQ[SAC] DiQ[DDPG]
DrQ (dotted silver) relies on SAC as base RL algorithm. Replacing SAC with DDPG results a
significant performance gain (bluc).
Acrobot Swingup
Quadruped Walk
Reacher Hard
Episode Return
Episode Return
800
Episode Return
600
400
200
400
200
200
Frames (x10%)
Frames (x10*)
Frames (x109)
DrQ[DDPG,n=3] DrQ[DDPG.n=1]
DrQIDDPG,n=5]
(b) DDPG straightforwardyy incorporates n-step returns, critical tool for exploration. We observe that the 3 (blue) and 5 (red) steps variants provide additional improvements to the previous version that uses single step TD-targets (silver). Going forward, adopt 3-step returns (blue).
Acrobot Swingup
Quadruped Walk
Reacher Hard
Episode Return
Episode Return
Episode Return
1000
400
800
750
600
200
400
250
0
0-
Frames (x10*)
Frames (x10*)
Frames (x10f)
DrQIDDPG.n=3.B=100
DrQ3DDDG.n=33B-55 109]
DrQIDDPG.n=3.=-100%
(c) Increasing the size of the replay buffer (B) improves performance, over the original 10 used by DrQ (silver). Going forward, we use buffer size of 10" (red).
Acrobot Swingup
Quadruped Walk
Reacher Hard
Episode Return
Episode Return
1000
400
750
600
200
Episode
400
500
200
0
0
Frames (x10*)
Frames (x10*)
Frames (x10f)
DrQIDDPG.n=3,B=1100 noise==ixed]
DrQIDDDG,,==3B=-10" "0o.:---100,ooieddecaa (DrQ-v2)
(d) Finally, decaying schedule for the variance of the exploration noise (blue) helps hard exploration tasks, versus the fixed variance variant (silver).
Figure 5: An ablation study that led us to the final version of DrQ-v2. We incrementally show each of the four key improvements to DrQ that collectively form DrQ-v2. The silver dotted curves the first row show the original DrQ. In subsequent rows they show progressive improvements, using the optimal choice from the previous rows (i.e., the silver curve in the third row shows DrQ with a DDPG base RL algorithm and 3-step returns). The red and blue curves show the effect of individual modifications. In the last row the blue curve corresponds to DrQ-v2.
This prevents the agent from finding more optimal behaviors due to the insufficient exploration. In Figure 5a, we empirically verify our intuition and, indeed, observe that DDPG demonstrates better exploration properties than SAC. Here, DDPG uses constant 0.2 for the exploration noise.
8

---

Under review as a conference paper at ICLR 2022
N-step Returns The second issue concerns the inability of soft Q-learning incorporate n-step returns to estimate TD error in a straightforward manner. The reason for this is that computing a target value for soft Q-function requires estimating per-step entropy of the policy, which is challenging to do for large n in the off-policy regime. In contrast, DDPG does not require estimating per-step entropy to compute targets and is more amenable for n-step returns. In Figure 5h we demonstrate that estimating TD error with n-step returns improves sample efficiency over vanilla DDPG. We select 3-step returns as a sensible choice for our method.
Replay Buffer Size We hypothesize that larger replay buffer plays an important role in circum- venting the catastrophic forgetting problem (Fedus et al., 2020). This issue is especially prominent in tasks with more diverse initial state distributions (i.e., reacher or humanoid tasks), where the vast variety of possible behaviors requires significantly larger memory. We confirm this intuition by ablating the size of the replay buffer in Figure 5c, where observe that a buffer size of IM helps to improve performance on Reacher Hard considerably.
Scheduled Exploration Noise
Finally, we demonstrate that it is useful to decay the variance of the exploration noise over the course of training according to Equation (3), In Figure 5d, compare two versions of our algorithm, where the first variant uses a fixed standard deviation of a = 0.2, while the second variant employes the decaying schedule (t), with parameters diit 1.0, dinal = 0.1, and = 500000. Having the exploration noise to decay linearly over time turns out to be helpful and provide additional performance boost, which was especially useful for solving humanoid tasks.
RELATED WORK
Visual Reinforcement Learning Successes of visual representation learning in computer vi- sion (Vincent et al., 2008; Doersch et al., 2015; Wang and Gupta, 2015; Noroozi and Favaro, 2016; Zhang et al., 2017; Gidaris et al., 2018) has inspired successes in visual RL, where coherent representations are learned alongside RL. Works such as SAC-AE (Yarats et al., 2019), PlaNet (Hafner et al., 2018), and SLAC (Lee et al., 2019), demonstrated how auto-encoders (Finn et al., 2015) could improve visual RL. Following this, other self-supervised objectives such as contrastive learning in CURL (Srinivas et al., 2020) and ATC (Stooke et al., 2020), self self-prediction in SPR (Schwarzer et al., 2020a), contrastive cluster assignment in Proto-RU RL (Yarats et al., 2021a), and augmented data in DrQ (Yarats et al., 2021b) and RAD (Laskin et al., 2020), have significantly bridged the gap between state-based and image-based RI. Future prediction objectives (Hafner et al., 2018; 2019; Yan et al., 2020; Finn et al., 2015; Pinto et al., 2016; Agrawal et al., 2016) and other auxiliary objectives (Jaderberg et al., 2016; Zhan al., 2020; Young et al., 2020; Chen et al., 2020) have shown improvements on a variety of problems ranging from gameplay, continuous control, and robotics. In the context of visual control settings, clever use of augmented data ( (Yarats et al., 2021b; Laskin et al., 2020) currently produces state-of-the of-the--rttttttt results on visual tasks from DMC (Tassa et al., 2018).
Humanoid Control The humanoid control problem first presented in Tassa et al. (2012), has been studied as one of the hardest control problems due to its large state and action spaces. The earliest solutions to this problem use ideas in model-based optimal control to generate policies given an accurate model of the humanoid Subsequent works in RL have shown that model-free policies can solve the humanoid control problem given access to proprioceptive state observations. However, solving such problem from visual observations has been challenging problem, with leading RL algorithms making little progress to solve the task (Tassa et al., 2018). Recently, Hafner et al. (2020) was able to solve this problem through a model-based technique in around 30M environment steps and 340 hours of training on single GPU machine. DrQ-v2, presented in this paper, marks the first model-free RL method that can solve humanoid control from visual observations, taking also around 30M steps and 86 hours of training on the same hardware.
6 CONCLUSION
We have introduced conceptually simple model-free actor-critic RL agent for image- based con- tinuous control - DrQ-v2. Our method provides significantly better computational footprint and masters tasks from DMC directly from pixels, most notably the humanoid locomotion tasks that

---

Under review as a conference paper at ICLR 2022
were previously unsolved by model-free approaches. To support empirical results and in- spire further research in visual RL we provide an efficient PyTorch implementation of DrQ-v2 at https: / / anonymous 4open. science/ z drqv2.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: experiential learning of intuitive physics. In Proceedings of the 30th International Conference Neural Information Processing Systems, pages 5092-5100, 2016.
Brandon Amos, Samuel Stanton, Denis Yarats, and Andrew Gordon Wilson. On the model-based stochastic value gradient for continuous reinforcement learning. CoRR, 2020.
Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributional policy gradients. In International Conference on Learning Representations, 2018.
Richard Bellman. markovian decision process. Indiana Univ. Math. J., 1957.
Bryan Chen, Alexander Sax, Gene Lewis, Iro Armeni, Silvio Savarese, Amir Roshan Zamir, Jitendra Malik, and Lerrel Pinto. Robust policies via mid-level visual representations: An experimental study in manipulation and navigation. CoRR, 2020.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference Computer Vision, pages 1422-1430, 2015.
Jing eng and Ronald J. Williams. Incremental multi-step q-learning. Machine Learning, 1996.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. CoRR, 2020.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Learning visual feature spaces for robotic manipulation with deep spatial autoencoders. CoRR, 2015.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassaan Stockholm, Sweden, July 10-15, 2018, 2018.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, 2018.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor, arXiv preprint arXiv:1801.0 01290, aa.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta nd Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, 2018b.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint 1,04551, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.
Danijar Hafner, Timothy P. Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. CoRR, 2020.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmen- tation. International Conference on Robotics and Automation, 2021.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint 17Xiv:1710.020 2017.
11

---

Under review as a conference paper at ICLR 2022
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Marth,Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning, 2020.
Max Jaderberg. Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks, 2016.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data, 2020.
A. X. Lee, A. Nagabandi, Abbeel, and Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv e-prints, 2019.
Alex X Lee, Anusha Nagahandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv:1907.00953 2019.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR. 2015a.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509.02971, 2015b.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv e-prints, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR, 2016a.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, 2016b.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare G. Bellemare. Safe and efficient off-policy reinforcement learning. CoRR, 2016.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69-84. Springer, 2016.
Lerrel Pinto, Dhiraj Gandhi, Yuanfeng Han, Yong-Lae Park, and Abhinav Gupta. The curious robot: Learning visual representations via physical interactions. CoRR, 2016.
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning. CoRR, 2020.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with momentum predictive representations. arXiv preprint arXiv:2007.05999, 2020a.
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. Data-efficient reinforcement learning with momentum predictive representations. CoRR, 2020b.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference Machine Learning, 2014.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint 2011:,,00000136, 2020.
12

---

Under review as a conference paper at ICLR 2022
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. arXiv preprint arXiv, 2020.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 20/2 JEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoing Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103, ACM, 2008.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
Christopher C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 1992.
Christopher John Cornish Hellaby Watkins. Learning from Delayed Rewards. PhD thesis, King's College, 1989.
Wilson Yan, Ashwin Vangipuram, Pieter Abbeel, and Lerrel Pinto. Learning predictive representations for deformable objects using contrastive estimation. arXiv preprint arXiv:2003.054433 2020.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint 191i0: 2019.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. CoRR, 2021a.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. 9th International Conference on Learning Representations, ICLR 2021, 202
Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto. Visual imitation made easy. CoRR, 2020.
Albert Zhan, Philip Zhao, Lerrel Pinto, Pieter Abbeel, and Michael Laskin. A framework for efficient robotic manipulation. CoRR, 2020.
Richard Zhang, Phillip Isola, and Alexei Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1058-1067, 2017.
13


Reviews:
[{"review_id": "wfMB7XL38n", "reviewer": "Reviewer_7c6j", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##########################################################################\n\nSummary:\n\nThis paper introduces DrQ-v2, an improvement over DrQ, a model free off-policy actor-critic approach which uses SAC+data augmentation to learn directly from pixels.\nDrQ-v2, on the other hand, switches SAC with DDPG and proposes a series of algorithmic, hyperparameter choices and implementation improvements.\nThese improvements make it both faster computationally and better empirically as it is able to solve the Humanoid environment from pixel.\n\n\n##########################################################################\n\nPros:\n\n- The paper solves the humanoid environment from visual input, a task long overdue in the RL community, with an easy to implement and easy to understand algorithm.\n\n- To solve this task, using this algorithm, it is not required to have inaccessible hardware requirements.\n\n- The paper provides open-source implementation, pseudo-code, well designed and comprehensive experiments for the improvements proposed over DrQ, which are run over 10 seeds to provide more claims over reproducibility.\n \n##########################################################################\n\nCons:\n\n- With the adding of clipped double Q-learning and action noise, you are technically switching from SAC to TD3 more than DDPG. This could be at least highlighted in the last paragraph of the introduction where the algorithmic changes are summarized.\n\n- Although the proposed paper provides several ablation studies and extensive experiments, I still think the following experiments would be nice to have. From my understanding the improvements are:\n    a. swith sac to ddpg with clipped double Q-learning and action noise\n    b. multi step return\n    c. exploration schedule\n    d. adding bilinear interpolation to random shift of image\n    e. replay buffer size, learning rate and batch size hyperparameter\n    f. speed improvements provided by better implementations of replay buffer and data augmentation.\n\n1. Are you sure that the wall-clock time improvements are related to the algorithm and not to your better implementation of replay buffer and data augmentation (f)? Have you tried to run DrQ with the new implementations of replay buffer and data augmentation?\n\n2. The improvements above could be grouped into algorithmic improvements allowed by switching backbone to ddpg (a, b, c), and code implementation/hyperparameter improvements (d, e, f). For qualifying the importance of the algorithmic improvements, a stronger DrQ baseline could be introduced with the addition of (d), (e), (f) to DrQ and compared to DrQ-v2.\n\n##########################################################################\n\nQuestions during rebuttal period:\n\nAddress and/or clarify the cons above.\n\n##########################################################################\n\nReasons for score:\n\nOverall I am towards acceptance of the paper. I think it's straightforward to read and the claims are well documented and backed by empirical results. There are no real technical contributions but the empirical novelties of changing to DDPG/TD3 and the performances are good. Half of the contributions indicated are just related to hyperparameter tuning and optimized implementations of the same algorithm and for this reason it's not a higher score. Hopefully the authors can address my concern in the rebuttal period.\n", "summary_of_the_paper": "This paper introduces DrQ-v2, an improvement over DrQ, a model free off-policy actor-critic approach which uses SAC+data augmentation to learn directly from pixels.\nDrQ-v2, on the other hand, switches SAC with DDPG and proposes a series of algorithmic, hyperparameter choices and implementation improvements.\nThese improvements make it both faster computationally and better empirically as it is able to solve the Humanoid environment from pixel.", "summary_of_the_review": "Overall I am towards acceptance of the paper. I think it's straightforward to read and the claims are well documented and backed by empirical results. There are no real technical contributions but the empirical novelties of changing to DDPG/TD3 and the performances are good. Half of the contributions indicated are just related to hyperparameter tuning and optimized implementations of the same algorithm and for this reason it's not a higher score. Hopefully the authors can address my concern in the rebuttal period.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OkH4SCICVK", "reviewer": "Reviewer_K4wT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\n1. An improved implementation to decrease the training time for visual-based RL. And thanks for pointing out the importance of comparing wall clock time, which is usually neglected in related work.\n2. Solving the humanoid task, which is not possible with prior implementation/method.\n3. The approach is well documented with open source code.\n\nIssue\n1. . \"First, the automatic entropy adjustment strategy, introduced in Haarnoja et al. (2018a), is inadequate and in some cases leads to a premature entropy collapse.\" What about don't use the automatic entropy adjustment? By the way, the reference to Haarnoja et al is duplicated. Since the whole point of SAC is to encourage exploration with the additional entropy reward, it is interesting to see that the authors make the comment that \"DDPG demonstrates better exploration properties than SAC\". It will be great if this point can be investigated further.\n2. It is interesting that the buffer size for quadruped run is 10^5. It will be nice if the authors can provide some insights on why this is an exception. What happens if I set it to be the default 10^6? When and how should I tune this parameter? Wouldn't this contradict the result of Figure 5(c)?\n3. It is not obvious to me that a decaying schedule helps from Figure 5(d). Even for reacher hard, eventually the fixed variance curve reaches the optimal around the same time as schedule decay.\n\nAdditional comments:\n1. From the perspective of simulation only baseline, 96FPS is really slow. One would wonder if faster training time can be achieved with on policy algorithm such as PPO, which can leverage parallel simulations and also less gradient update per sample collected. Of course, I understand this is not in the scope of this paper.", "summary_of_the_paper": "This paper presents DrQ-v2,  an improvement over DrQ, for solving visual-based RL problems. Key components include (1) DDPG over SAC (2) n-step returns for the critic (3) replay buffer size and (4) decaying exploration. Implementation improvement is also discussed. Notably, DrQ-v2 is able to solve the humanoid task effectively compared to recent work in this domain.", "summary_of_the_review": "This paper presents various improvements for visual RL with continuous control. Various ablations support most of the claims in the paper and notably the approach presented can solve the humanoid task that is not possible with prior methods. \nOne interesting aspect of the paper is that it provides different perspectives on RL benchmark. e.g., more emphasis on wall clock time, DDPG is actually better than SAC in the visual domain.\nIf my concern in the main review is addressed appropriately, I believe this will be valuable to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "635jKXPdlql", "reviewer": "Reviewer_ncsH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to read. Results are well explained, ablated and compared with the current state of the art. The improvements in terms of sample efficiency and speed (on a single V100 GPU) are a step change with respect to results which have been published before. \n\nThe main limitation of the paper is its limited scope which significantly reduces the technical novelty and significance to the field. The paper  isn't substantially presenting anything new neither from the theoretical or empirical standpoint. The main contribution consists of a combination of previously existing techniques carefully optimized together. It isn't clear how the proposed sample efficiency and speed improvements could generalize beyond the DM control suite without further adjustments. In a sense, authors don't do any effort to prove that their improvements could go beyond the scope presented in the paper and this limits the significance to the field.", "summary_of_the_paper": "The paper presents DrQ-v2, a  model-free reinforcement learning (RL) algorithm for visual continuous control. The algorithm is tested on the Deepmind Control Suite and it is proven to learn visual tasks in a remarkably short wall-clock time and with an extremely high computational efficiency (i.e. small number of frames). The paper is an improvement over a previous algorithm DrQ. Improvements are achieved with: (1) a new learning backbone (DDPG vs SAC); (2) the introduction of n-steps returns; (3) a bigger reply buffer and (4) a better scheduled exploration strategy.", "summary_of_the_review": "The paper is overall a good paper but I think it's marginally below the acceptance threshold because of its limited technical novelty and significance. It is worth of a publication as a technical report but it has a marginal value for the audience of an international conference. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3FETYfScAz", "reviewer": "Reviewer_uMh5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n\n- This method is simple and efficient.\n\n**Weaknesses**:\n\n- The generalization of the DrQ-v2 method on different tasks and RL algorithms:\n  - Generalization on different tasks: How DrQ-v2 performs on Atari games? DreamerV2 can *master* Atari. How about DrQ-v2?\n  - Generalization on RL algorithms: How DrQ-v2 performs when incorporated with on-policy algorithms, such as TRPO and PPO? Can DrQ-v2-PPO achieve competitive or better performance?\n", "summary_of_the_paper": "This paper proposed a new model-free RL method for visual continuous control problems.", "summary_of_the_review": "This paper proposed a new model-free RL method for visual continuous control problems. This method is efficient and straightforward. But the evaluation of the proposed method is limited. If the author can conduct more experiments on Atari games and with on-policy algorithms (such as PPO) can further improve the value of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses various improvements in visual continuous RL, based on a previous RL algorithm (DrQ). As the reviewers point out, the main contribution of the paper is of empirical nature, demonstrating how several different choices relative to DrQ significantly improve data efficiency and wall-clock computation, such that several control problems of the DeepMind control suite can be solved more efficiently. The average rating for the paper is above the acceptance threshold, and some reviewers increased their rating after there rebuttal. While a mostly empirically motivated papers is always a bit more controversial, the paper may nevertheless stimulated an interesting discussion at ICLR that will be beneficial for the community, and should thus be accepted.
Decision: Accept (Poster)