Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
Authors: Nicklas Hansen
Hao Su
Xiaolong Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
generalization
data augmentation

Abstract: While agents trained by Reinforcement Learning (RL) can solve increasingly challenging tasks directly from visual observations, generalizing learned skills to novel environments remains very challenging. Extensive use of data augmentation is a promising technique for improving generalization in RL, but it is often found to decrease sample efficiency and can even lead to divergence. In this paper, we investigate causes of instability when using data augmentation in common off-policy RL algorithms. We identify two problems, both rooted in high-variance Q-targets. Based on our findings, we propose a simple yet effective technique for stabilizing this class of algorithms under augmentation. We perform extensive empirical evaluation of image-based RL using both ConvNets and Vision Transformers (ViT) on a family of benchmarks based on DeepMind Control Suite, as well as in robotic manipulation tasks. Our method greatly improves stability and sample efficiency of ConvNets under augmentation, and achieves generalization results competitive with state-of-the-art methods for image-based RL in environments with unseen visuals. We further show that our method scales to RL with ViT-based architectures, and that data augmentation may be especially important in this setting.

Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
Nicklas Hansen'
Hao Su
Xiaolong Wang'
University of California, San Diego
nihansen@ucsd.cm edu (haosu, xiwee2)@eng.....mm ucsd.
Abstract
While agents trained by Reinforcement Learning (RL) can solve increasingly chal- lenging tasks directly from visual observations, generalizing learned skills to novel environments remains very challenging. Extensive use of data augmentation is a promising technique for improving generalization RL, but is often found to decrease sample efficiency and even lead to divergence. In this paper, we in- vestigate causes of instability when using data augmentation common off- off-policy RL algorithms. We identify two problems, both rooted in high-variance Q Q-targets. Based on our findings, we propose a simple yet effective technique for stabilizing this class of algorithms under augmentation. We perform extensive empirical evaluation of image-based RI. using both ConvNets and Vision Transformers (ViT) on family of benchmarks based on DeepMind Control Suite, as well in robotic manipulation tasks. Our method greatly improves stability and sample efficiency of ConvNets under augmentation, and achieves generalization results competitive with state-of-the-and methods for image-based RL in environments with unseen visuals. We further show that our method scales to RL with ViT-based architectures, and that data augmentation may be especially important in this setting."
Introduction
Reinforcement Learning (RL) from visual observations has achieved tremendous success in various applications such as video-games [43, 4, 70]. robotic manipulation [37], and autonomous navigation [42, 83]. However, it is still very challenging for current methods to generalize the learned skills to novel environments, and policies trained by RL can easily overfit to the training environment [81, 13). especially for high-dimensional observation spaces such as images [8, 58].
Increasing the variability in training data via domain randomization [66, 50] and data augmenta- tion [57, 35, 33, 51] has demonstrated encouraging results for learning policies invariant to changes in environment observations. Specifically, recent works on data augmentation [35, 33] both show im- provements in sample efficiency from simple cropping and translation augmentations, but the studies also conclude that additional data augmentation in fact decrease sample efficiency and even cause divergence. While these augmentations have the potential improve generalization, the increasingly varied data makes the optimization more challenging and risks instability. Unlike supervised learning, balancing the trade-off between stability and generalization in RL requires substantial trial and error.
In this paper, we illuminate causes of instability when applying data augmentation to common off-policy RI algorithms [43, 38, 15, 18). Based on our findings, we provide an intuitive method for stabilizing this class of algorithms under use of strong data augmentation. Specifically, we find two main causes of instability in previous work's application of data augmentation: (i) indiscriminate application data augmentation resulting in high variance Qttargets; and (ii) that Q-value estimation strictly from augmented data results in over-regularization.
Website and code is available at: https://nickkasshasse too......o/VVEA... 1o/SVEA.
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

To address these problems, we propose SVEA:
Stabilized Q-Value Estimation under Augmentation, a simple yet effective framework for data augmentation in off pollyy RL that greatly improves stability of Q-value estimation. Our method consists of the following three components: Firstly, by only applying augmentation in Q-value estimation of the current state, without augmenting Q-targets used for bootstrapping, SVEA circumvents erroneous bootstrapping caused by data augmentation; Secondly, we formulate a modified Q-objective that optimizes Q- Q-value estimation jointly over both augmented and unaugmented copies of the observations; Lastly, for SVEA implemented with actor- critic algorithm, we optimize the actor strictly on unaugmented data, and instead learn a generalizable policy indirectly through parameter-shane sharing. Our framework can be implemented efficiently without additional forward passes nor introducing additional learnable parameters.
We perform extensive empirical evaluation on the DeepMind Control Suite [64] and extensions of it, including the DMControl Generalization Benchmark [21] and the Distracting Control Suite [60], as well as set of robotic manipulation tasks. Our method greatly improve Q-value estimation with ConvNets under a set of strong data augmentations, and achieves sample efficiency, asymptotic performance, and generalization that is competitive or better than previous state-of- -ft---rrtttttttt methods in all tasks considered, at a lower computational cost. Finally, we show that our method scales to RL with Vision Transformers (ViT) [10]. We find that ViT-based architectures are especially prone to overfitting, and data augmentation may therefore be a key component for large-scale RL.
2 Related Work
Representation Learning. Learning visual invariances using data augmentation and self superviee
objectives has proven highly successful in computer vision [46, 45, 82, 74, 68, 65, 75, 27, 7]. For example, Chen et al. [7] perform an extensive study on data augmentation (e.g. random cropping and image distortions) for contrastive learning, and show that representations pre-trained with such transformations transfer effectively to downstream tasks. While our work also uses data augmentation for learning visual invariances, we leverage the Q-bjjective of deep Q-learning algorithms instead of auxiliary representation learning tasks.
Visual Learning for RL. Numerous methods have been proposed with the goal of improving sample efficiency [29, 56, 68, 76, 40, 59, 61, 54, 77] of image-based RL. Recently, using self-supervision to improve generalization in RL has also gained interest [80, 47, 55, 1, 22, 21, 72]. Notably, Zhang et al. [80] and Agarwal al. [1] propose to Iearn behavioral similarity embeddings via auxiliary tasks (bisimulation metrics and contrastive learning, respectively), and Hansen et al. [21] learn visual invariances through an auxiliary prediction task. While these results are encouraging, it has also been shown in [29, 40, 22, 79, 41] that the best choice of auxiliary tasks depends on the particular RL task, and that joint optimization with sub-optimally chosen tasks can lead to gradient interference. We achieve competitive sample-efficienge and generalization results withou the need for carefully chosen auxiliary tasks, and our method is therefore applicable to a larger variety of RL tasks.
Data Augmentation and Randomization for RL. Our work is directly inspired by previous work on generalization in RL by domain randomization [66, 50, 48, 52, 6] and data augmentation [36, 71, 35, 33, 51, 61, 21J. For example, Tobin et al. [66] show that a neural network trained for object localization in a simulation with randomized visual augmentations improves real world generalization. Similarly. Lee et [l][ show that application of a random convolutional layer to observations during training improve generalization in 3D navigation tasks. More recently, extensive studies on data augmentation [35, 33] have been conducted with RL, and conclude that, while small random crops and translations can improve sample efficiency, most data augmentations decrease sample efficiency and cause divergence. We illuminate main causes of instability, and propose a framework for data augmentation in deep Q-lcarning algorithms that drastically improves stability and generalization.
Improving Deep Q-Learning, While deep Q-learning algorithms such as Deep Q-Networks (DQN) [43] have achieved impressive results in image-based RL, the temporal difference objective is known to have inherent instabilities when used in conjunction with function approximation and off- policy data [63]. Therefore, variety of algorithmic improvements have been proposed to improve convergence [24, 73, 25, 23, 53, 38, 15, 14, 28]. For example, Hasselt et al. [24] reduce overestimation of Q-values by decomposing the target Q-value estimation into action selection and action evaluation using separate networks. Lillicrap et al. [38] reduce target variance by defining the target Q-network as a slow-moving average of the online Q-network. Our method also improves Q-value estimation, but we specifically address the instability of deep Q-learnig algorithms on augmented data.
2

---

3 Preliminaries
Problem formulation. We formulate the interaction between environment and policy as a Markov Decision Process (MDP) [2] M = A,
A. P. 7), where S is the state space, A is the action space, P: x A + S is the state transition function that defines a conditional probability distribution ( a.) over all possible next states given state s, S and action a, A taken at time t, r: T:SxAH + is a reward function, and 7 E [o, 1) is the discount factor. Because image observations only offer partial state observability [30], we define state 8; sequence of k consecutive frames (o,, O1-1,... Or-k), o E O, where O is the high- sigh-dimensional image space, as proposed in Mnih et al. [43]. The goal is then to learn a policy :: S + A that maximizes discounted return R = Er EΣ 2"r(s,, at)] along a trajectory policy т from an initial state So E S to state Sy with state transitions sampled from P. and π is parameterized by a collection of learnable parameters For clarity, we hereon generically denote parameterization with subscript, e.g. To. We further aim to learn parameters @ s.t. To generalizes well (i.e., obtains high discounted return) to unseen MDPs, which is generally unfeasible without further assumptions about the structure of the space of MDPs. In this work, we focus on generalization to MDPs M = (S. A. P, r, 7), where states S, E S are constructed from observations o E O, O S O of a perturbed observation space O (c.g. unseen visuals), and M ~ M for a space of MDPs M..
(So. Si...... .S....,,T)))............ obtained by following
Deep Q-Learning Common model-free off-policy RL algorithms aim to estimate an optimal state- action value function Q": S x A + R as Qs a) Rs Q*(ss a) = max E RR (s: S, a, = al using function approximation. In practice, this is achieved by means of the single-step Bellman residual (r(s), ar) ar) + y maxa; Q af)+
Qo(s;, ar) [62]. where vA parameterizes a target state-action value function Q We can choose to minimize this residual (also known as the temporal difference error) directly wrt using a mean squared error loss, which gives us the objective
Lo, vb) = Ex 1
1 2
(((tt. ax)
an) + max Q (8)+1, - Qo(S,, at)
27
(1)
ni
where B is replay buffer with transitions collected by behavioral policy [39]. From here, we can derive a greedy policy directly by selecting actions a; = arg maxa, Qu(S,, az). While Q = Q and periodically setting v 4- 0 exactly recovers the objective of DQN [43], several improvements have been proposed to improve stability of Eq. 1, such as Double Q-learning [24], Dueling Q Q-networks [73], updating target parameters using a slow-moving average of the online Q-network [38]:
(2) for an iteration step and a momentum coefficient Ç E (0, 1). and others [25, 23,53, 14, 28]. As com- puting maxa; 1 ($1+1, a!) in Eq. 1 is intractable for large and continuous action spaces, a number of prominent actor-critic algorithms that additionally learn a policy (s:))) z arg maxa, Qo as) have therefore been proposed [38, 15, 18].
Pa+1 - (1 - <) + ç0.,
Soft Actor-Critic (SAC) [18] is an off-policy actor-critic algorithm that learns a state-action value function Qo and stochastic policy mo (and optionally temperature parameter), where Qo is optimized using a variant of the objective in Eq. 1 and ma is optimized using a 7-discounted maximum-entropy objective [84]. To improve stability, SAC also commonly implemented using Double Q-learning and the slow-moving target parameters from Eq. 2. We will in the remainder of this work describe our method in the context of a generic off-policy RL algorithm that learns parameterized state-action value function Qo. while we in our experiments discussed in Section 6 evaluate of our method using SAC as base algorithm.
4 Pitfalls of Data Augmentation in Deep Q-Learning
In this section, we aim illuminate the main causes of instability from naive application of data augmentation in Q-value estimation. Our goal is to learn a Q-function Qe for an MDP M that generalizes to novel MDPs M ~ M. with unseen visuals, and we leverage data augmentation as an aptimality-invarian state transformation 7 to induce a bisimulation relation [34, 17] between a state s and its transformed (augmented) counterpart = = T(s, v)) with parameters v/ ~ V.
Definition (Optimality-Invan Invariant State Transformation [33]). Given an MDP M, sstate transforma- tion T: - S is an optimality invariian state transformation if Q(s.a) = Q(T(s, v), a) UsE S, a A, E V, where E V parameterizes the transformation
3

---

Following our definitions of M.M from Section 3, we can further extend the concept of optimality- invariant transformations to MDPs, noting that a change of state space (e.g. perturbed visuals) itself can be described as a transformation 7: S x V + & with unknown parameters 7 E V. If we choose the set of parameters V/ of a state transformation T to be sufficiently large such that it intersects with V with high probability, we can therefore expect to improve generalization to state and observation spaces not seen during training. However, while naive application of data augmentation in previous work [35, 33, 61, 54] may potentially improve generalization, it can be harmful to Q-value estimation. We hypothesize that this is primarily because it dramatically increases the size of the observed state space, and consequently also increases variance Var (Q(T(s,)) 2 Var (Q(s)) : V ~ V when V is large. Concretely, we identify the following two issues:
Pitfall 1: Non-deterministic Qttarget For deep Q-learning algorithms, previous work [35, 33, 61, 54] applies augmentation to both state sug _ (Se v) and successor state sim - T(((+1, v/) where u, o/ V. Compared with DQN [43] that uses deterministic (more precisely, periodically updated) Q-target this practice introduces a non-deterministic Q-target r(s;, ax) + + maxa Q ($++1: af) depending on the augmentation parameters v'. As observed in the original DQN paper, high-variance target values are detrimental to Q-learning algorithms, and may cause divergence due to the "deadly triad" of function approximation, bootstrapping, and off-policy learning [63]. This motivates the work to introduce a slowly changing target network, and several other works have refined the Qttarget update rule [38, 15] to further reduce volatility. However, because data augmentation is inherently non-deterministic, it can greatly increase variance in Qtarget estimation and exacerbates the issue of volatility, as shown in Figure 1 (top). This is particularly troubling in actor-critic algorithms such DDPG [38] and SAC [18], where the Q-target is estimated from ($1+1, a'), a' ~ (T(ss++)), which introduces an additional source of error from π that is non non- negligglii especially when St+1 is augmented.
Pitfall 2: Over-regularization. Data aug- mentation was originally introduced in the supervised learning regime as a regularizer to prevent overfitting of high-capacity models. However, for RL, even learning a policy in the training environment is hard. While data augmentation may improve generalization, greatly increases the difficulty of policy learn- ing, i.c., optimizing 9 for Qo and potentially behavior network aa- Particularly, when the temporal difference loss from Eq. 1 cannot be well minimized, the large amount of aug- mented states dominate the gradient, which significantly impacts Q-value estimation of both augmented and unaugmented states. We refer to this issue as overlaegulai by data augmentation. Figure 1 (bottom) shows the mean difference in Q-predictions made with augmented vs. unaugmented data in fully converged DrQ [33] agents trained with shift augmentation. Augmentations such as affine-jitter, random convolution, and random overlay incur large differences in estimated Q-values. While such difference can be re- duced by regularizing the optimization with each individual augmentation, we empha- size that even the minimal shift augmentation used throughout training incurs non zero dif- ference. Since is commonly chosen to be moving average of 0 as in Eq. 2, such differences caused by over-regularization affect Qo and Q'sl equally, and optimization may therefore still diverge depending the choice of data augmentation. As such, there is an inherent trade-off between accurate Q-value estimation and generalization when using data augmentation. In the following section, we address these pitfalls.
10
Q-target variance
0.0
0.1
0.2
0.3
0.4
0.5
Number (x10*)
DrQ conv
SVEA w/ conv
Mean Q) etror
15
10
aftireJitter
blu
rotattion
Figure 1. (Top) Mean Q-target variance of DrQ [33] and SVEA (ours), both trained with conv augmenta- tion [36]. (Bottom) Mean difference in Q-value es- timation on augmented vs. non-augmented data. We measure mean absolute error in Q-value estima- tion from converged DrQ agents (trained with shift augmentation) on the same observations before and after augmentation. Both figures averages across seeds for each of the 5 tasks from DMControl-GB..

---

Target encoder
Target Q-function
Q-target
data
stop-grad
ESVEA
aug
S0
Gt
Encoder
Q-function
predictioon
Figure Overview. An observation S; transformed by data augmentation T(; V to produce a view aug The Q-function Qo is then jointly optimized on both augmented and unaugmented data wrt the objective in Eq. 7, with the Q-target of the Bellman equation computed
from an unaugmented observation S1+1- We illustrate our data-mixing strategy by the @ operator.
5 Method
propose SVEA: Stabilized Q-Value Estimation under Augmentation, general framework for visual generalization in RI. by use of data augmentation. SVEA applies data augmentation in a novel learning framework leveraging two data streams with and without augmented data, respectively. Our method is compatible with any standard off- off-policy RL algorithm without changes to the underlying neural network that parameterizes the policy, and requires no additional forward passes, auxiliary tasks, learnable parameters. While SVEA in principle does not make any assumptions about the structure of states s, E S, we here describe our method the context of image- based RL.
5.1 Architectural Overview
An overview of the SVEA architecture provided in Figure Our method leverages properties of common neural network architectures used in off-policy RL without introducing additional learnable parameters. We subdivide the neural network layers and corresponding learnable parameters of a state-action value function into sub-networks fe (denoted the state encoder) and Qo (denoted the Q-function) s.t 2 (s:) ar) is the predicted Q-value corresponding to a given state- action pair (st, at). We similarly define the target state-action value function s.t. 9t y maxa; 2 (f (s') is the target Q-value for (st, at). and define parameters & an exponential moving average of 0 as in Eq. 2. Depending the choice of underlying algorithm, may choose to additionally learn a parameterized policy ma that shares encoder parameters with Qa and selects actions a, (f)-)fesss)..
tet r(st, (x)
To circumvent erroneous bootstrapping from augmented data (as discussed in Section 4), we strictly apply data augmentation in Q-value estimation of the current state st, without applying data augmen- tation to the successor state St41 used in Eq. for bootstrapping with v. (and T if applicable), which addresses Pitfall 1. If Ty is learned (i.e., SVEA is implemented with an actor-critic algorithm), also optimize it strietly from unaugmented data. To mitigate over-regularization in optimization of fo and Qo (Pitfall 2), we further employ a modified Q-objective that leverages both augmented and unaugmented data, which introduce in the following section.
5.2 Learning Objective
Our method redefines the temporal difference objective from Eq. I better leverage data augmenta- tion. First, recall that tet r(s,, a+) + max,; Q (f a Instead of learning to predict get
dt only from state s,, we propose to minimize a nonnegative linear combination of LQ over two individual data streams, St and s,
_ug T)), v ~ V, which we define as the objective
SVEA (0, xb) A aLQ (st, ttt 0)+) + 3LQ (s. aug ; cg
(3)
E a a Qo (fo a.) a. 1:1 + (
18t
(4)
where a, constant coefficients that balance the ratio of the unaugmented and augmented data streams, respectively, and qt is computed strictly from unaugmented data. DQ data-mixing strategy that oversamples unaugmented data as implicit variance reduction technique.
LSVEA (0, e) serves
5

---

As we will verify empirically in Section 6, data-m data-mixing is simple and effective technique for variance reduction that works well in tandem with our proposed modifications bootstrapping, For 9, the objective in Eq. 4 can be evaluated in a single, batched forward-pas by rewriting it as:
g: [St, (SE, 1)JN
(5)
h 9t 9
unl 1n1 JN
(6)
ESVEA (0, v)) Es, as 8 Aa (a (a+33) (Qo hell2 aug E IR NxCxHW where HN is a concatenation operator along the batch dimension N for St, S;
and 18t E IRNxI, which is illustrated as & in Figure 2. Empirically, we find a 0.5, 8 0.5 to be both effective and practical to implement, which we adopt in the majority of our experiments. However, more sophisticated schemes for selecting a, 8 and/or varying them as training progresses could be interesting directions for future research. the base algorithm of choice learns policy Te. its objective C (0) is optimized solely on unaugmented states without changes to the objective, and a stop-grad operation is applied after fe to prevent non-stationary gradients of C (0) from interfering with Q-value estimation, i.c. only the objective from Eq. 4 optionally Eq. 7 updates fe using stochastic gradient descent. As described Scction 5.1, parameters &! are updated using an exponential moving average and a stop-gropeeeeeooe operation is therefore similarly applied after Q We summarize our method for a = 3 applied to a generic off-policy algorithm in Algorithm 1.
Algorithm 1 Generic SVEA off-policy algorithm naive augmentation,
our modifications)
0, 1z..v vh: randomly initialized network parameters, v/
Initialize y to be equal to
7, Ç: learning rate and momentum coefficient
a, 3: loss coefficients, default: (a = 0.5, 3 = 0.5)
1: for timestep (. = 1... do
act:
2:
a4 Me (s))f(s:))
Sample action from policy D Sample transition from environment D Add transition to replay buffer
3:
5
~ ((. ar)
4:
B a,, r (s,,, s.).
update:
5:
(s,, a,, r(s,, a,), s L...N ~ B
Sample batch of transitions Naäve application of data augmentation
6:
S; = --; 11), s = (() V4, 1/,1/~> v
7:
for transition i 1..N do
8:
0 + On - 11 v/ La (s;; 0z) (if applicable)
Optimize va with SGD A> Compute Q-target
9:
qi = r(si, a:) + maxa; Q (f (5(), a!)
1gt
10:
s; T($ 1/), ) 14 V
Apply stochastic data augmentation Pack data streams Optimize So and Qa with SGD Update & using EMA of
11:
gi [i,88 8; Luge
N h; [0,
tot 1917
12:
+ 0 - -1 V (gi, hi; 0, v)
13:
1b -- (1 - () 5
6 Experiments
training
natural videos
We evaluate both sample efficiency, asymptotic performance, and generalization of our method and a set of strong baselines using both Con- vNets and Vision Transformers (ViT) [10] in tasks from DeepMind Control Suite (DMCon- trol) [64] as well as set of robotic manip- ulation tasks. DMControl offers challenging and diverse continuous control tasks and is widely used as a benchmark for mmgee-assd
Figure 3. Experimental setup. Agents are trained in fixed environment and are expected to general- ize to novel environments with e.g. random colors, backgrounds, and camera poses.
RL [19, 20, 76, 59, 35, 33]. To evaluate generalization of our method and baselines, we test methods under challenging distribution shifts (as illustrated in Figure 3) from the DMControl Gen- eralization Benchmark (DMControl-GB) [21]. the Distracting Control Suite (DistractingCSS) [60], as well as distribution shifts unique to the robotie manipulation environment. Code available at https:/ //tg thub. ctw http://////icccaaaaane///////eee
genera lization -ew.oonnnnnnnrrr........

---

EStode returm
200
400
200
1000
Waker, walk (DrQ)
Walten stand (Dra)
Curipole, sningup (D:0)
Carppole balance (Drd)
Pinger; spin (Dro)
Edtrate retum
200
400
5.0
0.1
0.2
533
0.4
0.5
0.0
0.1
0.2
6.3
0.4
0.5 0.0
0.1
0. 0.0
0.1
0.20.00
6.05
0.10 0.15 0.20
Number of frames (410%)
Number of frames (x10%)
Number of frames (x10%))
Number frames (x101)
Number of frames (x109)
overlay
rotation
aftine-jiter
Figure 4. Data augmentations. Training performance of SVEA (top) and DrQ (bottom) under common data augmentations. Mean of 5 seeds. Red line at 800 return is for visual guidance only. We omit visualization of std. deviations for clarity, but provide per-augmeetation comparisons to DrQ (including std. deviations) across all tasks in Appendix B, and test performances in Appendix C.
1200
Wis ber, walk (traiting]
Walker, stand (traiing)
Cartpsle, sw swmp (traning)
Ball in cup, catch (trining)
Finger, spin ntuinnng
Eptore retum
203
200
1000
Walker, waic levalustion)
Welker, stand (evallation)
Cartpole swhpup levsliation
Ball cup. catch levsluation
Finger, spin (evalcarion)
Ephode retumn
400
0.0
6.3
0.3 Number frames (x11P)
9.2
0.5
9.3
Number of frames (x10%)
Number frames (X12))
Number of frames (x20*)
Number frames (X12%)
DrQ
DrQ + coPV
.....
SVEA wi comv (data mixing only)
..... SVEA wi conv la 0
SVEA w/ conv
Figure 5. Training and test performance. We compare SVEA to DrQ with and without random convolution augmentation, well as set of ablations. Data-mixing only indiscriminately applies our data-mixing strategy to all data streams, and (a = 00 3 = 1) only augments Q-predictions but without data-mixing. We find both components contribute SVEA's success. Top: episode return on the training environment during training. Bottom: generalization measured by episode return on the color hard benchmark of DMControl-GB, Mean of seeds, shaded area is 11 std. deviation.
Setup. We implement our method and baselines using SAC [18] as base algorithm, and we apply random shift augmentation to all methods by default. This makes our base algorithm equivalent to DrQ [33] when K=1,M=1; we refer to the base algorithm as unaugmented and consider stability under additional data augmentation. We use the same network architecture and hyperparameters for all methods (whenever applicable), and adopt the setup from Hansen and Wang [21]. Observations are stacks of 3 RGB frames of size 84 x 84 x (and 96 x 96 x 3 in ViT experiments). In the DMControl- GB and DistractingCS benchmarks, all methods are trained for 500k frames and evaluated on all 5 tasks from DMControl-GB used in prior work, and we adopt the same experimental setup for robotic manipulation. See Appendix H for hyperparameters and further details on experimental setup.
Baselines and data augmentations. We benchmark our method against the following strong base- lines: (1) CURL [59], contrastive learning method for RL; (2) RAD that applies a random crop: (3) DrQ that applies a random shift; (4) PAD [22] that adapts to test environments using self-supervision; (5) SODA [21] that applies data augmentation in auxiliary learning; as well number of ablations. We compare to the K=1,M=1 setting of DrQ by default, but also provide comparison to varying M. We experiment with diverse set of data augmentations proposed in previous work on RL and computer vision, namely random shift [33]. random convolution (denoted conv) [36], random overlay [21], random cutout [9], Gaussian blur, random affine-fitter, and random rotation [35, 16). We provide samples for all data augmentations in Appendix C. and test environments in Appendix E.
7

---

Table I. Comparison state-of-the-art. Test performance (episode return) of methods trained in a single, fixed environment and evaluated on (i) randomized colors, and (ii) natural video backgrounds from DMControi-GB, Results for CURL, RAD, PAD, and SODA are obtained from [21] and we report mean and std. deviation over 5 seeds. DrQ corresponds to our SAC base algorithm using random shift augmentation. SVEA matches or outperforms prior methods in all tasks considered.
DMControl-GB
CURL
RAD
DrQ
PAD
SODA (cOnV)
SODA (overlay)
SVEA (conv)
SVEA (overlay)
(random colors)
walker,
445
400
520
468
697
692
760
749
walk
+:9
261
=:1
1:47
+:68
2:68
144
2:01
walker, stand
662
644
770
797
930
803
942
933
154
188
171
1:46
1.12
1:12
126
1:24
cartpole, swingup
454
590
586
630
831
805
837
832
+110
4:3
4.02
1:29
4:23
1:20
ball in cup, catch
231
541
365
563
892
949
961
959
=210
+:00
#:19
finger, spin
691
667
776
803
901
793
977
972
112
1:54
=154
472
+:51
1128
255
26
DMControl-Gi
CURL
RAD
DrQ
PAD
SODA (coRv)
SODA (overlay)
SVEA (conv)
SVEA
(natural vidcos)
(overlay)
walker, walk
556 1133
606
682
717
635
768
612
819
1.80
1.38
171
walker, 5 stand
852
745
873
935
903
955
795
961
475
#148
1:8
1:20
4:36
1:1
=70
1.8
cartpole, swingup
404
373
485
521
474
758
606
782
+67
#72
=10G
#76
#143
#:02
#86
#:7
ball iun catch
316
481
318
436
539
875
659
871
#119
+:26
=107
4:55
#:11
4:50
#110
#106
finger, spin
502
400
533
691
363
695
764
808
#19
+64
=119
4:80
# 188
#97
=86
#38
DeupMind Control Suta (training)
Discruttng Central Saite
549
450
800
Epirodo return
400
250
400
200
259
200
203
0.0
0.1
0.2
0.3
0.4
0.5
24
48
72
36
120
144
100
159
Number of frames (x10%)
Wallitme (hours)
359
0.1
0.2
33
1.4
[K=1,M11]
conw (KmddMM44)
DrQ conv (K-2,M21
DrQ conw [K=B,M-B]
5r0
SVIA ml conV
SVTA overbyy
Figure 6. (Left) Comparison with additional DrQ baselines. We compare SVEA implemented with DrQ [K=1.M=1] as base algorithm to DrQ with varying values of its K, M hyperparameters. All methods use the conv augmentation (in addition to shift augmentation used by DrQ). Results are averaged over seeds for each of the tasks from DMControl-GB [21] and shaded area is +11 std. deviation across seeds. Increasing values of K, M improve sample efficiency of DrQ. but at a high computational cost; DrQ uses approx. 6x wall-time to match the sample efficiency of SVEA. (Right) DistractingCS, Episode return as a function of randomization intensity at test-time, aggregated across 5 seeds for each of the 5 tasks from DMControl-GB. See Appendix E for per-task comparison.
6.1 Stability and Generalization on DMControl
Stability. We evaluate the stability of SVEA and DrQ under ó common data augmentations; results are shown in Figure While the sample efficiency of DrQ degrades substantially for most augmentations, SVEA is relatively unaffected by the choice of data augmentation and improves sample efficiency in 27 out of 30 instances. While the sample efficiency of DrQ can be improved by increasing its K.M parameters, we find that DrQ requires approx. 6x wall-time to match the sample efficiency of SVEA; see Figure 6 (left). We further ablate each component of SVEA and report both training and test curves in Figure 5; we find that both components are key to SVEA'S success. Because we empirically find the cony augmentation to be particularly difficult to optimize, we provide additional stability experiments in Section 6.2 and 6.3 using this augmentation. See Appendix A for additional ablations.
8

---

Generalization. We compare the test performance of SVEA to 5 recent state-of-the- art methods for image-based RL on the color hard and video easy benchmarks from DMControl-GE (results in Table 1), as well as the extremely challenging DistractingCS benchmark, where camera pose, background, and colors are continually changing throughout an episode (results in Figure 6 (right)). We here use comv and overlay augmentations for fair comparison to SODA, and we report additional results on the video hard benchmark in Appendix F. SVEA outperforms all methods considered in 12 out of 15 instances and SODA that all learn auxiliary tasks. On DistractingCS, we observe that SVEA improves generalization by 42% low intensity, and its generalization degrades significantly slower than DrQ for high intensities. While generalization depends on the particular choice of data augmentation and test environments, this is encouraging result considering that SVEA enables efficient policy learning with stronger augmentations than previous methods.
DMControl-GB, and at lower computational cost than CURL, PAD,
6.2 RL with Vision Transformers
patchee
Vision Transformers (ViT) [10] have re- cently achieved impressive results on down- stream tasks in computer vision. We re- place all convolutional layers from the previous experiments with a 4-layer ViT encoder that operates on raw pixels in 8 x 8 space-tima timee-timepee patches, and evaluate our method using data augmentation in con- junction with VIT encoders. Importantly, we design the ViT architecture such that it roughly matches our CNN encoder in terms of learnable parameters. The VIT encoder is trained from scratch using RL, and we use the same experimental setup as our ConvNet experiments. In particular, it is worth emphasizing that both our ViT and CNN encoders trained using Adam [32] as optimizer and without weight decay. See Figure 7 (top) for architectural overview, and refer to Appendix H for additional im- plementation details.
Dug
Colorta
VIT wnsoodr
Trensform wiczole
MLP
Layerkorm
Mallihtad Atantion
Layerlorm
Caripols, setrgup
600
Our training and test results are shown Figure 7 (bottom). We are, to the best of our knowledge, the first to success- fully solve image-based RL tasks without CNNs. We observe that DrQ overfits sig- nificantly the training environment com- pared to its CNN counterpart (94 test re- turn color hard for DrQ with VIT vs. 569 with ConvNet on the Walker, walk task). SVEA achieves comparable sam- ple efficiency and improves generalization by 706% and 233% on Walker, walk and Cartpole, swingup, respectively, over DrQ, while DrQ conv remains unstable. In- terestingly, we observe that our Vis-b implementation of SVEA achieves mean episode return of 877 on the color hard benchmark of the challenging Walker, walk task (vs. 760 using CNNs). SVEA might therefore be promising technique for fu- ture research on RL with CNN-free architectures, where data augmentation appears to be especially important for generalization. We provide additional experiments with ViT encoders Section 6.3 and make further comparison to Nen encoders in Appendix A.
1000
Walker wait (evaluation)
Cartocle ealngup (nsmuaiion)
Certpore, balance (evalcatio)
400
0.2
0.2
framis
DrQ
DVQ
como
SVGA coru [CNN]
SVEA W/ COTE
Figure 7. (Top) VIT architecture. Observations are divided into 144 non-overlapping space-ting time patches and linearly projected into tokens. Each token uses a learned positional encoding and we also use learnable class token as [10]. The ViT encoder consists of 4 stacked Transformer encoders [69]. (Bottom) RL with a ViT encoder. Training and test performance of SVEA and DrQ using ViT encoders. We report results for three tasks and test performance is evaluated on the color hard benchmark of DMControl-GB, Mean of seeds, shaded area +1 std. deviation.

---

Table 2. Generalization in robotic manipulation. Task success rates of SVEA and DrQ with CNN and VIT encoders in the training environment, as well as aggregated success rates across 25 different test environments with randomized camera pose, colors, lighting, and background. Mean of 5 seeds.
Robotic manipulation
Arch.
reach (train)
reach (test)
mv.tgt.
mv.
push (train)
push (lest)
(encoder)
(train)
(test)
DrQ
CNN
1.00
0.60
1.00
0.69
0.76
0.26
DrQ + conV
CNN
0.59
0.77
0.60
0.89
0.13
0.12
SVEA w/ CONV
CNN
1.00
0.89
1.00
0.96
0.72
0.48
DrQ
VIT
0.93
0.14
1.00
0.16
0.73
0.05
DrQ + conV
VIT
0.26
0.67
0.48
0.82
0.08
0.07
SVEA w/ conv
VIT
0.98
0.71
1.00
0.81
0.82
0.17
6.3 Robotic Manipulation
training
randomized endrronments
We additionally consider set of goal-conditioned robotic manipula- tion tasks using simulated Kinova Gen3 arm: (i) reach, a task in which the robot needs to position its gripper above goal indicated by a red mark; (ii) reach moving target, a task sim- ilar to (i) but where the robot needs to follow a red mark moving contin- uously in a zig-zag pattern at ran- dom velocity; and (iii) push, a task in which the robot needs to push cube red mark. The initial config- uration of gripper, object, and goal is randomized, the agent uses 2D posi- tional control, and policies are trained using dense rewards. Observations stacks of RGB frames with ac- cess to state information. Training and test environments
test
Figure 8. Robotic manipulation. Agents are trained in fixed environment and evaluated on challenging environ- ments with randomized colors, lighting, background, and camera pose.
Robot, ttrining!
Robot, reach my, tge. Orriningg
Robot, push (taining)
tpaeece return
-20
18
30 Number fromes (K10P)
10
20
1:00
200
300
Numoor
flames (x307)
Numbor framess (N11P)
DrQ
DrO + canv
SVEA wl cony
Figure 9. Stability with a CNN encoder. Training per- formance (episode return) of SVEA and DrQ in 3 robotic manipulation tasks. Mean and std. deviation of 5 seeds. Success rates are shown in Table 2.
shown in Figure 8. See Appendix G for further details and environment samples.
50
Robot, reach (traiting)
Robot, reach my. tgt. (traningg
Robot, push (raining)
Results are shown in Figure and Figure 10. For both CNN and ViT encoders, SVEA trained with conv augmentation has similar sample ef- ficiency and training performance as DrQ trained without augmentation, while DrQ + conv exhibits poor sam- ple efficiency and fails to solve the push task. Generalization results are shown in Table 2. We find that naïve application of data augmentation has higher success rate in test environ- ments than DrQ, despite being less successful in the training environment, which we conjecture is because it is optimized only from augmented data. Conversely, SVEA achieves high success rates during both training and testing.
Eplinde return
-10
10
20
30
40 107)
50
10 23
30
40
50
1.0
200
300
Number of frames 1x
Number of frames (x10))
Number of frames (x100)
DrQ
DrD conv
SVEA W/ cony
Figure 10. Stability with a VIT encoder. Training per- formance (episode return) of SVEA and DrQ in 3 robotic manipulation tasks. Mean and std. deviation of 5 seeds. Suc- cess rates are shown in Table 2. DrQ is especially unstable under augmentation when using ViT encoder.
Conclusion. SVEA is found to greatly improve both stability and sample efficiency under augmenta- tion, while achieving competitive generalization results. Our experiments indicate that our method scales to ViT-based architectures, and it may therefore be promising technique for large-scale RL experiments where data augmentation is expected to play an increasingly important role.
Broader Impact. While our contribution aims to reduce computational cost of image-based RL, we remain concerned about the growing ecological and economical footprint of deep learning and RI. in particular- with increasingly large models such as VIT; see Appendix for further discussion.
10

---

Acknowledgments and Funding Transparency. This work was supported by grants from DARPA LwLL, NSF CCF-2112665 (TILOS). NSF 1730158 CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), NSF ACI-1541349 CC*DNI Pacific Research Platform, NSF grant I18-1763278, NSF CCF-2112665 (TIL.O), as well as gifts from Qualcomm, TuSimple and Picsart.
References
[1] Rishabh Agarwal, Marlos C. Machado, P. S. Castro, and Mare Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. ArXiv, abs/2101.05265. 2021.
[2] Richard Bellman. markovian decision process. Journal of Mathematics and Mechanics, 6(5):679 684, 1957.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models too hig? Proceedings of the 202/ ACM Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, 2021.
[4] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Den- nison, David Farhi, Quirin Fischer, et al. Dota 2 with large scale deep reinforcement learning. ArXiv, abs/1912.06680, 2019.
[5] T. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neclakan- tan, Pranav Shyam, et al. Language models are few-shot learners. arXiv, abs/2005. 14422005.1415,,, 2020.
[6] Yevgen Chebotar, A. Handa, Viktor Makoviychuk, M. Macklin, J. Issac, Nathan D. Ratliff, and Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. 2019 International Conference on Robotics and Automation (ICRA), pages 8973 8979, 2019.
[7] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
[8] K. Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman, Quantifying generalization in reinforcement learning. In lcml, 2019.
[9] Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. 2018.
[10] A. Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Dehghani, Matthias Minderer, et al. An image worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11992 2020.
[11] Frederik Ebert. Chelsea Finn. Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control, 2018.
[12] Lasse Espeholt, Hubert Soyer, R. Munos, K. Simonyan, V. Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, et al. Impala: Scalable distributed deep-rl with importance weighted actor-Icamer architectures. ArXiv, abs/1802.01561 2018.
[13] Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and regularization in dqn. ArXiv, abs/1810000122 2018.
[14] Meire Fortunato, M. G. Azar, Bilal Piot, Jacob Menick. Ian Osband, A. Graves, Vlad Mnih, R. Munos. et al. Noisy networks for exploration. ArXiv, abs/1 777.70.......... 2018.
[15] Scott Fujimoto, V. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. ArXiv, hths/1802.09477, 2018.
[16] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, 2018.
[17] R. Givan, Dean, and M. Greig. Equivalence notions and model minimization in markov decision processes. Artificial Intelligence, 147:163 147:163-223, 2003.
[18] Tuomas Haarnoja. Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-pooicy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:/801.01200 2018.
[19] Danijar Hafner, T. Lillicrap, Ian S. Fischer, R. Villegas, David R Ha, H. Lee, and James Davidson. Learning latent dynamics for planning from pixels. ArXiv, 1. 2019.
[20] Danijar Hafner, T. Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. ArXiv, abs/1912.01603, 2020.
[21] Nicklas Hansen and Xiaolong Wang, Generalization in reinforcement learning by soft data augmentation. In International Conference Robotics and Automation, 2021.
11

---

[22] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto. and Xiaolong Wang. Self-supervised policy adaptation during deployment. In International Conference on Learning Reprexentations 2021.
[23] H. V. Hasselt, A. Guez, Matteo Hessel, V. Mnih, and D. Silver. Learning values across many orders of magnitude. In Nips, 2016.
[24] H. V. Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-leamnngg In Aaai, 2016.
[25] M. Hausknecht and P. Stone. Deep recurrent q-learning for partially observable mdps. In AAAI Fall Symposia, 2015.
[26] Kaiming He, X. Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016
IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 770-777 77007777777 2016.
[27] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsu- pervised visual representation learning. 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735, 2020.
[28] Matteo Hessel, Joseph Modayil, H. V. Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, et al. Rainbow: Combining improvements in deep reinforcement learning. In Acai, 2018.
[29] Max Jaderberg. Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks, 2016.
[30] Leslie Pack Kaelbling, Michael L. Littman. and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains, Artificial Intelligence, 1998.
[31] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Eric Jang. Deirdre Quillen, Ethan Holly, al. Qi- opt: Scalable deep reinforcement learning for vision-based robotic manipulation. ArXiv, abs/1 htss/80...um 10293, 2018.
[32] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6990. 2015.
[33] llya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. International Conference Learning Representations, 2020.
[34] K. G. Larsen and A. Skou. Bisimulation through probabilistic testing (preliminary report). In Proceedings of the 16th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, 1989.
[35] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforce- ment learning with augmented data. arXiv preprint arXi::2004. 14990, 2020.
[36] Kimin Lee. Kibok Lee, Jinwoo Shin, and Honglak Lee. simple randomization technique for generaliza- tion in deep reinforcement learning. ArXiv, abs/ 1910.5539, 2019.
[37] Sergey Levine, Chelsea Finn, Trevor Darrell, and Picter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334 1373, 2016.
[38] T. Lillicrap. J. Hunt, A. Pritzel, N. Heess. T. Erez, Y. Tassa, D. Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509902971. 2016.
[39] L. J. Lin. Self-improving reactive agents based on reinforcement learning. planning and teaching. Muchine Learning, 8:293-321, 2004.
[40] Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in neural information processing systems, 32, 2019.
[41] Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney. On the effect of auxiliary tasks on representation dynamics. In Aistats, 2021.
[42] P. Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, R. Goroshin, al. Learning navigate in complex environments. ArXiv, abs//6ll.../// 1.03673, 2017.
[43] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1312v:311..500, 2013.
[44] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl. Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pages 9191-9200, 2018.
[45] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69 84. Springer, 2016.
[46] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536 2544, 2016.
12

---

[47] Deepak Pathak, Dhiraj Gandhi, and A. Gupta. Self-supervised exploration via disagreement. ArXiv, abs/1906.04166 2019.
[48] Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamies randomization. 2018 IEEE International Conference on Robotics and Automation (ICRA), May 2018.
[49] Lerrel Pinto and Abhinay Gupta. Supersizing supf-superiision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3406-3413. leee, 2016.
[50] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv: orXiv:/710.060 171000647.. 2017.
[51] Roberta Raileanu, M. Goldstein, Denis Yarats, Ilya Kostrikov, and R. Fergus. Automatic data augmentation for generalization in deep reinforcement learning. ArXiv, abs/2006. 12862, 2020.
[52] Fabio Ramos, Rafael Possas, and Dieter Fox. Bayessim: Adaptive domain randomization via probabilistic inference for robotics simulators. Robotics: Science and Systems XV, Jun 2019.
[53] Tom Schaul, John Quan, Ioannis Antonoglou, and D. Silver. Prioritized experience replay. CoRR, abs/1511.05955 2016.
[54] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self self-predietive representations. arXiv preprint arX1v:2007.05992) 2020.
[55] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models, 2020.
[56] Evan Shelhamer, Parsa Mahmoudich, Max Argus, and Trevor Darrell. Loss is its own reward: Self- supervision for reinforcement learning. ArXiv, 111.00077,7 2017.
[57] Connor Shorten and T. Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6:1-48, 2019.
[58] Xingyou Song. Yiding Jiang. Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. ArXiv, abs/1912.02975, 2020.
[59] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.0411 2020.
[60] Austin Stone, Oscar Ramirez, K. Konolige, and Rico Jonschkowski. The distracting control suite a challenging benchmark for reinforcement learning from pixels. ArXiv, abs221000222,, 2021.
[61] Adam Stooke, Kimin Lee, Picter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement lcarning. ArXiv, abs/2004, 14999, 2020.
[62] R. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9 44, 2005.
[63] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
[64] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li. Diego de Las Casas. David Budden, Abbas Abdolmaleki, et al. Deepmind control suite. Technical report, DeepMind, 2018.
[65] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint mXXvv1906.00544 2019.
[66] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Sep 2017.
[67] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012. doi: 10. 1199/rro.. 20w.oos..012..6 2012.636619.
[68] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018
[69] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv, abw//176.037666 2017.
[70] Oriol Vinyals, I. Babuschkin, Wojciech Czarnecki, Micha
"el Mathieu, Andrew Dudzik, J. Chung, D. Choi, Richard Powell, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, pages 1-5, 2019.
[71] K. Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. ArXiv, abs/2010.10814. 2020.
13

---

[72] Xudong Wang, Long Lian, and Stella X. Yu. Unsupervised visual attention and invariance for reinforcement learning. ArXiv, abs/2104.02921, 2021.
[73] Ziyu Wang, Tom Schaul, Matteo Hessel, H. V. Hasselt, Mare Lanctot, and N. D. Freitas. Dueling network architectures for deep reinforcement learning. ArXiv,
20051122116 2016.
[74] Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
[75] Zhenlin Xu, Deyi Liu, Junlin Yang, and Niethammer. Robust and generalizable visual representation learning via random convolutions. ArXiv, abs/2007. 13003, 2020.
[76] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images, 2019.
[77] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. arXiv preprint arXiv:2102. /271, 2021.
[78] Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, Lerrel Pinto. Visual imitation made easy. CoRL. 2020.
[79] Tianhe Yu, Saurabh Kumar, Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, ArXiv, abs/2001.061 0.67822, 2020.
[80] A. Zhang, Rowan McAllister, R. Calandra, Y. Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. ArXiv, abs/2006. 10742, 2020.
[81] C. Zhang, Oriol Vinyals, R. Munos, and S. Bengio. A study on overfitting in deep reinforcement learning. ArXiv, abs/1804.06893, 2018.
[82] Richard Zhang, Phillip Isola, Alexei Efros. Colorful image colorization. Earopean conference on computer vision, pages 649-666. Springer, 2016.
[83] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In Icra, pages 3357- 3364. Icee, 2017.
[84] Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. Proceedings of the 23rd National Conference on Artificial Intellignce, volume 2008.
14


Reviews:
[{"review_id": "kH_8QIb005l", "reviewer": "Reviewer_UP5u", "summary": "The paper presents a way of improving training stability of DrQ (an image-based RL algorithm that uses data augmentation) by applying image augmentation more carefully. The authors propose to input a mix of augmented and un-augmented observations into the critic, and stop augmenting the actor's inputs. On the considered benchmarks the proposed method demonstrates better generalization results. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper identifies an issue with the increased variance of the target estimation in DrQ (image-based RL with data augmentation). The authors conjecture two root causes for the increased variance: non-deterministic Q-targets and over-regularization.\nTo remedy these issues they propose to apply data augmentation more carefully. Specifically, the authors suggest to mix augmented and un-augmented inputs to the critic and stop augmenting of the actor's inputs. This method is then empirically justified in a series of experiment on tasks from the DeepMind Control Suite.\n\nPros:\n* The author investigate an interesting question of an effect of data augmentation on generalization in image-based RL. \n* The method is thoroughly verified in different settings. \n* The paper is clearly written and easy to follow.\n* The exploration of alternative architecture, such as ViT.\n\nCons:\n* The paper proposes a small trick that lacks in novelty. The impact of data-augmentation on the estimation variance was also studied in the original DrQ paper (Kostrikov et. al), where a very similar technique was proposed -- the target estimation was done over 2 different augmentations. This work introduces a tiny modification of the DrQ's variance reduction approach, that just combines one augmented and one un-augmented observation instead of combining two augmented observations. Furthermore, the authors fail to mention this prior result in their paper and compare to it. \n* From the paper it is not clear what hyper-parameters are used for the baselines, specifically for DrQ. If the authors use the original DrQ's parameters, then I see several major issues with the empirical results: 1) SVEA uses encoder arch that consists of 11 conv layers, while DrQ only uses 4. 2) The size of replay buffer of SVEA is 500K, while DrQ's 100K. 3) SVEA virtually increases the batch size by factor of 2, which usually leads to better performance on the DeepMind Control Suite tasks, it is not clear if the authors increase batch size for DrQ or not. These discrepancies alone could result in a drastic difference in performance and can render the empirical study to be invalid. \n* There is no experiments that compare individual components of SVEA to verify is that the improved performance comes from the proposed method or better/different hyper-parameters. \n* The authors claim in line 30 that they \"theoretically ground\" their findings, but this is not the case as there is no theoretical results presented. In general, the authors don't do a good job justifying that the two pitfalls are real. Their argument is hand-wavy and is based on a series of indirect comparisons to the baseline (DrQ). Unfortunately, there are too many moving parts in those comparisons that make it virtually impossible to validate the claim (see also the point above about the hyper-parameter discrepancies).\n\nGiven the very limited scope of novelty and technical contribution, the failure to mention the prior work that is almost identical, and the inconclusive empirical study I recommend to reject this paper from a high-profile venue of NeurIPS. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "U0pN4EETJhb", "reviewer": "Reviewer_MseA", "summary": "This paper studies the problem of high-variance Q-targets under data augmentation, and proposes a method to reduce the variance of the targets called SVEA. The key idea contains three parts: (1) Apply augmentation to the Q-value estimation of the current state and not the next state Q-targets for bootstrapping, and (2) modify the objective to include both the targets using augmented and original observations and (3) optimize the actor using only the unaugmented data for actor-critic algorithms, with parameter sharing. \n\nThe paper performed experiments in DeepMind control suite and its variants (with raw pixel inputs). Compared to other data augmentation methods (e.g. DrQ) their proposed SVEA method is robust to the type of data augmentation and is more sample efficient with a better final performance. Some ablations were also presented among the above components to indicate the usefulness of all components. Generalization performance was demonstrated in the DistractingCS and robotic manipulation environments. The method also appears to help when using a Vision Transformers architecture. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The proposed framework can be seen as a generalization of existing data augmentation approaches, by considering combinations of both augmented and unaugmented data in the loss function. The related work section is adequately touching upon the several areas which this framework can be viewed from. \n\n**Quality**: There is no theoretical analysis for the proposed framework, but the paper presents several empirical results to back up the claims, including ablations on the various components. However, I would suggest backing up the claim on Pitfall 1: Non-deterministic Q-target with some empirical data as well. How bad is the variance with non-deterministic Q-targets in practice? With SVEA, even though we have deterministic Q-targets, we still have noisy predictions due to the data augmentation. So there can still be comparison of variance in the case of combination of augmented and unaugmented current state and next state. \n\nFor the alpha and beta hyperparameter: it was mentioned that 0.5/0.5 works well for the experiments. Have the authors tried varying this parameter throughout the training (i.e. some sort of annealing schedule), and whether that hinders or helps the performance even more?\n\n**Clarity**: The paper is overall fairly clear to read. \n\n**Significance**: Given the recent popularity of applying data augmentation in vision-based reinforcement learning agents, I believe that the proposed technique, which is simple to implement, can be easily incorporated as a toolbox of RL practitioners/researchers. Demonstrating that this framework still scales to ViT architectures is also promising. It would be nice to have some more theoretical justification for the framework by the original authors, but perhaps that could be a follow up work (potentially by others). \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues with this paper. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "TAJ3pE4xjjQ", "reviewer": "Reviewer_xUWy", "summary": "(Auxiliary Review only -- the authors need not respond to this).\n\nThis work proposes issues with data augmentation in RL and proposes a couple of techniques to address those issues. It applies augmentation only on the current state and not future state to address erroneous bootstrapping. It also proposes a Q objective that uses both the augmented and un-augmented data. Also, for actor critic experiments, the actor is optimised only with un-augmented data and learn a  policy that is general via sharing parameters between the networks. These are simple techniques that help the paper demonstrate good results in a range of tasks.\n\nThis is a clear paper with good set of experiments. The problem has been described clearly and the work proposes simple refinements to the data augmentation strategies used in RL. While I am convinced of the quality of the method itself and the experiments, this work could have benefitted from more rigour with the baselines to make a stronger case for the method. Hence, I am offering a score of 6 noting that the paper is marginally above acceptance threshold.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Note : * This is an AUXILIARY REVIEW only. The authors are not expected to respond to this review or any of the concerns presented here. * *\n\nThe paper is well-written, clear and provides a good description of the experiments. It clearly sets up the problem that standard data augmentation methods suffer from. It also provides empirical solutions to these problems and backs up these claims with experimental results. Even though the paper justifies the claims only empirically based on experimentation, the ideas proposed here have made simple and easily extendable modifications, which can be very useful in complex state action spaces when RL+Data augmentation is employed. The ideas proposed here are pretty simple and straightforward, but I am not convinced these methods alone provide us a full-fledged framework for data augmentation as claimed. Also, any theoretical justification provided with this work is not rigorous enough to demonstrate that these methods are universal for all data augmentation problems.  On the novelty front, the paper's key contribution seems to be the idea of using un-augmented inputs to the targets. There are two ways to look at this -- just as a simple innovation being not novel enough on its own or trying to understand how the work arrives at this simple innovation and whether the innovation helps improve the algorithm performance over a wide variety of tasks. I am more inclined towards the latter. I believe this is still a useful enough contribution to practitioners and researchers alike, where a few simple modifications help greatly stabilize performance.\n\nMy major concern with the paper was the hyper-parameters and architectures used for the baselines. It looks like the adapted baselines were handicapped a bit by architectural changes and these baselines were not tuned to optimality, or at least it was not very evident that this was done thoroughly. This takes a bit of the shine off the impressive results presented by the paper and I am not sure if tuning the parameters would change any relative comparison between those baselines and this method.\n\nThe paper's proposed methods are good and the results are useful on their own to the community. Having said that, because of the other minor issues mentioned earlier, I am not able to give a higher score than 6 for this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have addressed the limitations.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns that I am aware of.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "SsnCSOo29O8", "reviewer": "Reviewer_xDk5", "summary": "This work proposes a simple technique to improve the robustness and sample-efficiency of reinforcement learning. The technique can be applied to any algorithm via a simple modification of the input pipeline to said algorithm's Q-learning subroutine. The claim is that this modification reduces the variance of Q-targets and therefore stabilizes learning.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Disclaimer: Emergency review with all the short-comings that comes with.\n\n# Originality\n\nWhile the idea of data-augmentation has been extensively used in both supervised and reinforcement learning, this paper proposes an approach that in some ways generalizes the DrQ approach, and in others simplifies it. In my view the modifications, though subtle, are novel.\n\n# Quality\n\nThe work produces an extensive array of empirical results showing reobust generalization and impressive sample-efficiency. The comparisons to DrQ lack specification of the hyperparamters K and M, which can easily be added. If, as I suspect, the comparison is with DrQ[K=1, M=1], it would be nice to see a comparison to the default DrQ[K=2, M=2] and/or at the very least DrQ[K=2, M=1], which matches the number of forward passes through the Q-network.\n\nSimilarly, in the empirical demonstration of Pitfall 2 (Figure 1) it would be nice to see the equivalent plot for various values of K.\n\nFinally, I would slightly quibble with the claim of \"theoretically grounded\" on line 30, as this paper does not feature any rigorous theoretical statements.\n\n# Clarity\n\nOverall the paper is very clearly and well written. The only major point of contention is that DrQ is parameterized by a K and M which control how many samples the target Q-values and TD loss, respectively, are averaged over. It would be good to specify which is being used here. It sounds like [K=1, M=1] (?) but I would think the fairest comparison would be [K=2, M=1].\n\n\n## Minor points\n\n- The targets on line 215 and line 9 of Algorithm 1 presumably look different in continuous control cases, which is the main focus of the experimental section.\n- Very minor point: would the expressions not be equivalent but simpler if the trade-off was simply alpha and (1 - alpha), instead of beta?\n- The experimental setup is slightly unclear in the text, copying the caption of Figure 3 into the main body would help. In particular, the sentence \"all methods are trained for 500k frames and evaluated on the full set of tasks\".\n- Figure 6 (left) needs error-bars.\n\n# Significance\n\nData-augmentation seems to be a very simple technique for improving the sample-efficiency and generalizability of RL agents. This is particularly important for real-world applications where interactions with the environment can be expensive and the sim-to-real gap can be prohibitive.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I did not see any mention of limitations. The authors mentioned the growing footprint of large models and how this work could help negotiate that.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Gb8m7TXP42Q", "reviewer": "Reviewer_7gSX", "summary": "The paper addresses the question of zero-shot or fast generalization of value-based deep reinforcement learning agents using visual inputs to image transformation which do not change the (conceptual) state of the environment. The long term goal is therefore training agents which learn efficient, high return control policies both which are robust to irrelevant variations in high dimensional input observations. The ultimate goal seems to be generalization of learned skills to novel environments, but as far as I can tell this particular evaluation philosophy is implemented in experiments.\n\nThe paper reviews previous attempts to learn robust state representations to visual variations and formulates clear hypotheses for explaining the causes of difficulties which prior methods faced and were hindered by. The said hypotheses suggest a simplified approach (SVEA) which is empirically developed and evaluated in the remainder of the paper. Control tasks from widely used benchmarks are used to evaluate the method, somewhat less usual is the focus on vision-based high-dimensional versions of such tasks, instead of standard domains such as Atari games and benchmarks for generalization such as Natural Atari [1], CoinRun/ProcGen [2].\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am generally happy with the paper, I have a few suggestions:\n\n* The goal of reinforcement learning is discovering an optimal control policy, not necessarily an action-value function. Strictly speaking, value estimation is just a crutch. Hence, why bother with robust value estimation? Why not learn value functions without augmentations, use them to discover an optimal policy, and then use supervised learning [3, 4] to encode a robust policy? Please clarify in the text! The proposed method is attempting to solve a problem (RL under observation augmentation) which may not be relevant if all that is needed is good control under diverse observation augmentations. How am I getting this wrong?\n* Indeed, for all these environments it is not clear that value prediction from vision is needed at all, since a policy could be derived much faster from full state representations in terms of sample efficiency and computational speed, and then have that policy supervise a vision to action mapping, circumventing the robust value estimation problem altogether.\n* There are many possible definitions of generalization in reinforcement learning; one is accurate value prediction in unseen states, as realistic problems have enormous state spaces; another is robust value prediction from high-dimensional observations, where states may have been seem before but there is extraneous variance in the observations which needs to be dealt with. Finally, there are progressively stronger definitions of generalization, namely across domain gaps which may or may not introduce variability relevant to the control policy itself. Which definition have you tested for in your experiments?\n* Could you please clarify in the main text which experiments use some form of state information, e.g. \"proprioceptive\" data common in control tasks, as well as target object information, and which do not? If such information is used, how does it reflect on the ability to extrapolate the presented results to problems where such information is not available.\n* Could you please specify for each set of results whether the set of initial states was randomized and by how much? One surefire way to get robustness to high variability in observations is open-loop control, which can also be implemented by non-recurrent architectures, e.g. by relying heavily on proprioceptive inputs or general statistics of observations. \n\n\nReferences:\n1. (Natural Atari) Amy Zhang et al. 2018. Natural Environment Benchmarks for Reinforcement Learning. https://arxiv.org/abs/1811.06032\n2. (CoinRun/ProcGen) Karl Cobbe et al. 2019. Leveraging Procedural Generation to Benchmark Reinforcement Learning. https://arxiv.org/abs/1912.01588\n3. Andrei Rusu et al. 2015. Policy Distillation. https://arxiv.org/abs/1511.06295\n4. Wojciech Czarnecki et al. 2019. Distilling Policy Distillation. https://arxiv.org/abs/1902.02186\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I understand why every paper claims the maximal general applicability possible which cannot be refuted by reviewers without experiments; please note that proving negatives is hard, and much rests on your honesty and expertise alone. I am not comfortable with text which talks about aspirational goals as if they were achieved in this paper. The abstract talks about generalization to novel environments and great improvements in stability and sample efficiency, it seems to me that not enough attention to detail has been expended in quantifying the limitations of this approach and the wider family of algorithms to which it belongs. For example, a claim of \"greatly improved sample efficiency\" sounds outstanding, but then it is not clear to a non-RL specialist how many order of magnitude we are talking about, and how augmentations themselves make the problem worse. While the paper deserves credit for exploring this subject in detail, such claims are vague at best and may not age well; sample efficiency may mean very different amounts of experience in 5 years time.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the problems related to the high variance in the targets that arise in TD-learning when the network is trained with data augmentation. Thee paper proposes a simple and effective way to address this problem by applying the data augmentation only on the online Q-network not on the target Qs and a modified objective function. The paper presents results on a range of tasks.

The paper is well-written and it is well-motivated. The authors investigate the proposed method on a range of interesting tasks and provide a very simple but seemingly effective solution.

The authors have provided an extensive rebuttal during the discussion period. However, some of the reviewers complaints are not completely addressed and I would appreciate if they can be addressed in the camera-ready version of this paper:

1) The theoretical justification provided in this paper is not rigorous enough. There is no guarantee that the proposed approach would work on any type of data augmentations. It would be nice if the authors are more careful about this and highlight that in the paper in order to avoid over-claiming.
2) The Reviewer MseA, xDk5 and UP5u asked several questions about the hyperparameters of the model. It would be nice if the authors can provide more analysis on the impact of the different hyperparameters used for different baselines. For example Reviewer UP5u pointed the different hyperparameters of DrQ and suggested more experiments to compare against.
3) More clear descriptions of the experimental setup along with a table of all hyperparameters used for all models (potentially in the appendix).
4) Adding error bars to the figures where applicable.
 
In addition to those points, I would recommend the authors to address all the other concerns by the reviewers as much as possible.
Decision: Accept (Poster)