Measuring Visual Generalization in Continuous Control from Pixels
Authors: Jake Grigsby
Yanjun Jane Qi
Venue:  (2021)
Keywords: Continuous Control
Reinforcement Learning

Abstract: Self-supervised learning and data augmentation have significantly reduced the performance gap between state and image-based reinforcement learning agents in continuous control tasks. However, it is still unclear whether current techniques can face the variety of visual conditions required by real-world environments. We propose a challenging benchmark that tests agents' visual generalization by adding graphical variety to existing continuous control domains. Our empirical analysis shows that current methods struggle to generalize across a diverse set of visual changes, and we examine the specific factors of variation that make these tasks difficult. We find that data augmentation techniques outperform self-supervised learning approaches and that more significant image transformations provide better visual generalization.

Under review as a conference paper at ICLR 2021
MEASURING VISUAL GENERALIZATION IN CONTINUOUS CONTROL FROM PIXELS
Anonymous authors Paper under double-blind review
ABSTRACT
Sulf-super learning and data augmentation have significantly reduced the performance gap between state and image-based reinforcement learn- ing agents in continuous control tasks. However, is still unclear whether current techniques can face the variety of visual conditions required by real-world environments. We propose challenging benchmark that tests agents' visual generalization by adding graphical variety to existing contin- nous control domains. Our empirical analysis shows that current methods struggle to generalize across diverse set of visual changes, and exam- ine the specific factors of ion that make these tasks difficult. We find that data augmentation techniques outperform self-supervised learning ap- proaches, and that more significant image transformations provide better visual generalization.
INTRODUCTION
Reinforcement Learning has successfully learned to control complex physical systems when presented with real-time sensor data (Gu et al., 2017) (Kalashnikov et al., 2018). However, much of the field's core algorithmic work happens in simulation (Lillicrap et al., 2015) (Haarnoja et al., 2018a), where all of the environmental conditions are known. In the real world, gaining access to precise sensory state information can be expensive or impossible. Camera-based observations are practical solution, but create representation learning problem in which important control information needs to be recovered from images of the environment.
Significant progress has been made towards solution to this challenge using auxiliary loss functions (Yarals et al., 2019) (Zhang et al., 2020) (Srinivas et al., 2020) and data augmen- tation (Laskin al., 2020) (Kostrikov et al., 2020). These strategies often match or exceed the performance of state- based approaches in simulated benchmarks. However, current con- tinuous control environments include very little visual diversity. Tf existing techniques are going to be successful in the real world, they will need to operate in variety of visual con- ditions. Small differences in lighting, camera position, or the surrounding environment can dramatically alter the raw pixel values presented to an agent, without affecting the under- lying state. Ideally, agents would learn representations that are invariant to task-irrelevant visual changes.
In this paper, we investigale the extent to which current methods meet these requirements. We propose a challenging benchmark to measure agents' ability to generalize across a diverse of visual conditions, including changes in camera position, lighting, color, and scenery, by extending the graphical variety of existing continuous control domains. Our benchmark provides a platform for examining the visual generalization challenge that imago- con- trol systems may face in the real world while preserving the advantages of sinulation-basel training. We evaluate several recent approaches and find that while they can adapt to subtle changes in camera position and lighting, they struggle to generalize across the full range of visual conditions and are particularly distracted by changes in texture and scenery. A comparison across multiple control domains shows that data augmentation significantly outperforms other approaches, and that visual generalization benefits from more complex, color-altering image transformations.

---

Under review as a conference paper at ICLR 2021
2 BACKGROUND
Reinforcement Learning: We deal with the Partially Observed Markov Decision Process (POMDP) setting defined by a tuple (8, O, o, A, R,T,T) S is the set of states, which are usually low-dimensional representations of all the environment information needed to choose an appropriate action. In contrast, O is typically a higher-dimenssonal observation space that is visible to the agent. R the reward function $ A. - R, T is the function representing transition probabilities S x $ + [0, 1]. Φ is the emittion funct ion S - O that determines how the observations in O are generated by the true states in S. A is the set of focus available actions that can be taken at each state; in the continuous control setting on in this paper, A is a bounded subset of IR", where n is the dimension of the action space. An agent is defined by stochastic policy л that maps observations to a distribution over actions in A. The goal of Reinforcement Learning (RL) is to find a policy that maximizes the discounted sum of rewards over trajectories of experience T, collected from a POMDP
7 R(s,, az)l. where E 10, is a discount factor that M, denoted T) (T) = E
1=0
determines the agent's 's emphasis on long-term rewards.
Continuous Control: Many of the challenging benchmarks and applications of RL involve continuous action spaces. These include classic problems like Cartpole, Mountain Car and Inverted Pendulum (Moore, 1990) (Barto et al., 1990). However, recent work in Deep RL has focused on a collection of 3D locomotion tasks in which a robot rewarded for a specific type of movement in physics simulator. At each timestep, the state is vector representing key information about the robot's current position and motion (e.g. the location, rotation or velocity of joints and limbs). The action space is continuous subset of RR" where each element controls some aspect of the robot's movement. (e.g- the torque of a joint motor). There have been several standardized implementations of these environments (Schulman et al., 2015) (Brockman et al., 2016). This paper will use modified versions of those provided in the DeepMind Control Suite (DMC) (Tassa et al., 2018), built using the MuJoCo physics simulator (Todorov et al., 2012).
Soft Actor Critic (SAC) (Haarnoja et al., 2018a) is an off-policy actor- critic method that achieves impressive performance many continuous control tasks. For each training step, a SAC agent collects experience from the environment by sampling from a high-entropy policy (a ~ (o)))) paramaterized by neural network with weights 0. It then adds experience to a large buffer D in the form of a tuple (o, ao.a,r,of o SAC samples a batch of previous experience from the buffer, and updates the actor to encourage higher-value actions, as determined by the critic network (Q)(o,a)):
Soft Actor Critic:
Lactor = =- =-B E min (o) a log x (ā(o)) ā To()) i=1,2
(1)
oND
The critic networks are updated by minimizing the mean squared error of their predictions relative a bootstrapped estimate of the action-value function:
(Rates.)) - (r+ +-mimm Qsr,(o', log no rā'oo''))) ā' To(o')
Leritic
E
(o
E12
(2)
The log Th terms are maximum-entron component that encourages exploration subject to a parameter a, which can either be fixed or learned by gradient descent to approach a target entropy level (Haarnoja et al., 2018b). The min operation makes use of two critic networks that are trained separately, and helps to reduce overestimation bias (Fujimoto et al., 2018). of refers target networks, which are moving averages of the critie networks' weights and help to stabilize learning a common trick in Deep RL (Lillicrap el al., 2015) (Mnih et al., 2015).
Generalization in Reinforcement Learning: Consider a set of POMDPs M {SS, Φo, Dg, Ao. Ro, To, T0), ..., (Sn, On, O,, An, Ra, Ta, %.)). Given access to one or several training tasks Mirain ~ M, we would like to learn a policy that can generalize to the entire
We leave out the terminal boolean d common to most implementations both for notational simplicity and because many of the DMC tasks reset after a fixed amount of time.
2

---

Under review as a conference paper at ICLR 2021
set. This is analogous to the way we train supervised models to generalize across entire distribution despite only having labels for a finite number of imputs. In supervised learn- ing, we would partition our labeled data into training and testing sets, and measure the difference in our model's performance between them. We can create Merain and MMest in a similar way (Zhang et al., 2018a). This paper focuses on the zero-shot generalization setting ([Marain) = 1), and will normalize by the training return in an effort to compare the general- ization of policies with different baseline levels of performance. A policy's generalization er- ElMcoel "MMuse (TMMra (T)).
ror, (T)) is defined as: (T(T) = (1)Meram (T)
3 A BENCHMARK FOR MEASURING VISUAL GENERALIZATION IN CONTINUOUS CONTROL FROM PIXELS
3.1 CONTINUOUS CONTROL FROM PIXELS
While SAC performs well on many of the DMC tasks, has access to state information that would be difficult to replicate in the context of real-world robolics (O = S). Ideally, we could learn a policy from imperfect or high-dimensional observations. This had led to a more challenging benchmark where the same set of tasks are learned from image renderings of the environment. Many successful approaches build around the addition of a convolutional encoder to the SAC agent architecture. The encoder attempts to process stack of consec- utive frames into a low-dimensional reprosentation that is then treated as a surrogate state vector for the actor and critic(s). However, the RL setting's sparse gradient signal makes it difficult to train a high-capacily encoder with a limited number of samples, and baseline im- plementat ions typically fail to make progress (Tassa et al., 2018). Several recent. works have addressed this problem by adding additional objectives to the encoder's gradient update. SAC+AE adds an image reconstruction term that trains a decoder network to recover the original image from the encoder's 's compressed representation, as in a standard autoencoder (Yarats et al., 2019). CURL uses a contrastive loss (Srinivas et al., 2020) that encourages similar representations across random crops of its input images. DBC uses a bisimulation metric to discard observation information that isn't relevant to control (Zhang et al., 2020). All three result in performance at near the levels of state-based SAC and experiments show that the representation learned by the encoder recovers key state information.
Data Augmentation in Deep RL: Another recent line of work applies standard data augmentations from image processing to the training loops of RL algorithms. This simple technique can outperform more complicated approaches like SAC+AE and CURL. There are some subtletics in both the types of data augmentations that are used and how they are applied within the training update. RAD (Laskin et al., 2020) renders the environment at a higher resolution and randomly crops each observation as it sampled from the replay buffer. Data Regularized Q-Learning (DrQ) (Kostrikov et al., 2020) is a similar approach that includes the option to augment the of images separately within each timestep in hopes of computing a lower-variance target for the critic updates. Instead of upscaling, DrQ pads each image by replicating border pixels before randomly cropping back to the original size. Data Regularized Actor-Critic (DrAC) (Raileanu et al., 2020) more explicitly regularizes the outputs of the actor and critic networks to be invariant under each augmentation. It also attempts to learn which transformations to apply automatically; experiments in all of these works show this to be a surprisingly high-stakes problem, many augmentation choices have little to no effect on learning (cf. Laskin et al. (2020) Figure 2a, Kostrikov el al. (2020) Appendix E Figure 6, and Appendix A.3.3). The search finds random crop to be the most helpful in many of the Procgen benchmark games (Cobbe et al., 2019a) and shows that it a strong default even when outperformed by other methods. Most of the successful augmentations aside from Random Crop are color-based transformations. These include Network Randomization (NR) (Lee al., 2019) which uses randomly initialized convolutional layers to create visually distinct filters of the same image and Color Jitter (CJ). which shuffles the color palette.
Instead of benchmarking NR, DrQ and RAD separately, we implement a custom version of Pixel SAC that lets us combine aspects of all three algorithms. The overall structure is sim- ilar to the originals: we use a large convolutional encoder that is only updated by the critic's gradients and copy many of the baselines' hyperparameters for the sake of comparison. Data.
3

---

Under review as a conference paper at ICLR 2021
4o(50)
(210(5)
Ass2s(50l
2 (5x)
Figure 1: Example "Walker, Walk" visual seeds. The random aesthetic changes gen- erated by DMCR al- low for significant vi- sual diversity. @o uses the default DMC as- scts.
(0577(50)
4 (5p)
46627(5g)
1000l5
augmentation is applied in way that is most similar RAD (Laskin et al., 2020). We introduce new hyperparaaeter 3 E [0, 1] which controls how mix the augmented and original batches for the actor and critic updates. = uses augmented data only; B uses the original. We set 3 .9. This setup looks to alleviate the need to perform test time augmentation, even when the augmentation significantly alters the image, as in (Lee et al., 2019) (see more analysis in Appendix A.3.1). In addition, we encourage the output of the encoder, eg to be invariant. to our augmentation pipeline 2, by adding an explicit. regularization term to the encoder loss:
Lencoder Leritic X E [|leç(0) - =c(2(0))]11)
(3)
oND
We set À = 1e-5, and only let gradients flow through the eo(z(o)) pass, for stability. Ap- pendix A.3.2 discusses this idea more detail.
After a comparison (Appendix A.3.3), we settle on the pad/crop transformation from DrQ (Kostrikov et al., 2020) our primary augmentation. We will refer to this algorithm as SAC+AUG. Any other transformations that are used will be added to the acronym in results and figures. For example, SAC+AUG that applics Color Jitter before the DrQ crop will be denoted SAC+CJ+AUG.
3.2 OBSERVATIONAL OVERFITTING AND VISUAL GENERALIZATION
While algorithms like CURL, SAC+AE and DrQ can learn successful policies from pixels, we are interested in measuring the extent to which they overfit to the visual conditions of their training environments. Observational overfitting occurs when an agent becomes dependent features that spuriously correlated with high reward (Song generalization is succinetly described as low EG across a set of POMDPs that varies primarily (or even exclusively) in Φ. We aim evaluate this by creating a set of environments that vary only in their appearance, mcaning that any generalization error is the result of overfitting to visual fcatures that are not relevant the task.
al., 2019). Visual
3.3 EXPANDING THE VISUAL DIVERSITY OF DMC TASKS
Towards this goal, we introduce a modified version of the DeepMind Control Suite with an emphasis on visual diversity. Environments are given a new "visual seed", which significantly modifies the graphical appearance of the MuJoCo renderings while leaving the transition dynamics untouched. This allows the same sequence of states to be rendered in millions of distinct ways (see Figure 9). When initialized, our environments make a sequence of pseudo-randon graphical choices, deterministically seeded by the visual seed, k, resulting in a unique appearance Φx:
1. Floor. The floor's pattorn is sampled from a game design pack that includes hundreds of rock, grass, soil, sand, ice and concrete textures.
2. Background. The background is sampled from collection of natural landscape photos.
3. Body Color. The color of the robot's body is sampled from the full RGB space. Tf there is another MuJoCo body in the scene (e.g., the glowing target location in "Reacher, Reach"), it is given second random color.
4. Camera and Lighting. The camera's position and rotation are sampled uniformly within a relatively small range that varies across each domain. Lighting position is chosen similarly, along with other properties like ambience and diffusion. The limits of these distributions were determined by generating a large number seeds across each domain and ensuring that there were sufficient variations without hiding important task information.

---

Under review as a conference paper at ICLR 2021
These simple alterations are enough to create an enormous amount of variety. Figure shows a sample of 10 seeds in the "Walker, Walk" task, and Appendix A.5 contains many more examples. We also allow each type of adjustment to be toggled on off independently, and deterministic seeding makes it easy to train or evaluate multiple agents in the same environments. Importantly, we decouple the visual seed from the random seed that is already present in DMC tasks, determining the robot's 's initial position. During training, each reset generates a new starting position, usual. We br-nd the original random seed as the "dynamics seed" for clarity.
The initial release includes 10 of the DMC domains for a total of 24 tasks. A full list of the randomized elements that are relevant to each domain provided in Table 5. We will refer to this version of DMC with expanded graphics "DeepMind Control Remastered" (DMCR).
EXPERIMENTAL RESULTS AND ANALYSIS
We evaluate several recent works in pixel-based continuous control on DMC Remastered. The experimental results are organized as follows: in Section 4.1, we test the basclines' performance on the more complicated graphics assets of DMCR under the classic setup of a single training and lesting seed. In Section 4.2, we evaluate each method's ability to generalize to the entire distribution of levels. In Section 4.3, we test each factor of variation in isolation to get an understanding of the specific visual characteristics current methods struggle with the most. Then in Section 4.4, we highlight the advantages of data augmenta- tion methods and investigate the performance of more powerful image transformations like Color Jitter and Network Randomization.
4.1 TRAINING PIXEL-BASED AGENTS ON COMPLEX GRAPHICS TEXTURES
We first look at whether agents can be successfully trained from scratch inside environments with DMCR's more distracting visuals, including changes in lighting and camera position. We train CURL, SAC+AE, and SAC+AUG on the original DMC visuals (oo) and three representative samples from the DMCR version of "Walker, Walk" one of the more dif- ficult. tasks in the benchmark. The CURL and SAC+AF runs use their authors' original implementations, while SAC+AUG is mix of data augmentation methods, as discussed in Section 3.1. Results are shown in Figure SAC+AE and CURL perform well across each visual seed, but SAC+AUG collapses early in training in one environment. Follow- ing this, we collect data using the original DrQ implementation and notice a similar effect. We investigate this issue further by comparing the performance of SAC+AUG and CURL in the original environment against more than 60 visual seeds from the DMCR version of "Ball in Cup, Catch." " Results (shown in the bottom right of Figure 2) suggest that DMCR environment visuals are slightly more difficult than the high contrast default appearance of DMC.
4.2 GENERALIZING TO THE FULL & DISTRIBUTION
Next, we measure the ability of agents trained in one environment to adapt to the full vi- sual diversity of the DMCR benchmark. We train CURL, SAC+AE, and SAC+AUG agents in six tasks, using the original (øo) visuals. The pixel-based MuJoCo literature makes important distinction between 'training steps' and 'environment steps.' training step is defined by the agent taking an action in the environment and storing the resulting experi- ence in its replay bufler. The simulator repeats the action several times before returning the next observation; this 'Trame skip' 'action repeat'' hyperparameler creates discrep- ancy between the training step count and the total number of environment timesteps. Our experiments use the settings listed in Table 3 (see Appendix). After training, the agent is evaluated over 100 dynamics seeds in its original training environment and tested in 3 dy- namics soods across 100 different visual soeds from DMCR. Table 1 reports the mean return over each set, as well as the normalized generalization error. The results show that these methods do not learn features capable of generalizing across significant visual changes.
5

---

Under review as a conference paper at ICLR 2021
6985
400
Seed
Seed
1785
5285
625720
739644
DNCR Ball
Cotch
Agont
SAC-AUG
CURL
Benchimare
7inuct
726Get
DRFCR
DMC
252
1005 10005
150ml
25400
1805 19008
300m
2506 4000
6000 8000 18000
12003 14000 14000
Figure Pixel SAC variants trained on different visual seeds from the DMCR version of "Walker, Walk". Final performance is relatively consistent, though the pure data angmentation methods are prone to occasional collapses early in training that are difficult to recover from. We also provide comparison of the DMC and DMCR. versions of Ball in Cup, Catch" (in bottom right).
Method:
SAC+AE
CURL
SAC+AUG
Metrics:
Train
Test
EG
Train
Test
EG
Train
Test
EG
Walker, Walk
643.6
25.0
96. 96.1%
622.7
25.3
95.9%
919.3
28.0
97.0%
Cheetah, Run
387.9
4.2
98.9%
202.3
2.4
98.8%
712.8
19.8
97.2%
Hopper, Stand
631.8
1.6
99.8%
425.5
1.9
99.5%
878.1
3.3
99.6%
Finger, Spin
622.1
0.1
100.0%
731.3
1.6
99.8%
916.7
7.4
99.2%
Ball in Cup, Catch
647.0
99.2
84.6%
815.6
106.3
87.0%
934.5
103.0
89.0%
Cartpole, Balance
942.8
212.3
77.3%
885.7
233.3
73.7%
987.4
218.8
77.8%
Table 1: Training and Testing returns across 6 tasks. Training scores are averaged over 100 random initializations with no visual changes. Testing scores random initializations on 100 visual seeds. EG indicales the agent's 's decline in performance relative to it's training environment. Results are averaged over five runs.
4.3 IDENTIFYING THE MOST DIFFICULT FACTORS OF VARIATION
While generalizing to the entire distribution of visuals is desirable goal, the high level of difficulty can make it hard to differentiate between algorithms and measure gradual progress. It would be helpful to know the specific factors of variation that are most challenging. We investigate this by allowing cach category of visual changes to be toggled on or off independently, Each agent is re-cvaluated seeds, across 7 variations of the core environment. Results for all tasks are shown in Table 2. In addition the CURL, SAC+AE and SAC-AUG agents from Section 4.2, we test the impact of Color Jitter and Network Randonization augmentations. SAC+AUG has the highest returns across every task, adding more evidence recent findings (Laskin al,, 2020) (Kalashnikov et al., 2018) that simple augmentation techniques can outperform the more complicated auxiliary losses of SAC+AF and CURL. We find that all methods adapt well to changes in lighting conditions. Aside from SAC+AE, many agents can also handle the camera shifts and rotations. This may be because SAC+AE is the only method that does not involve random crop, which can have similar effect to subtle camera translations. SAG+NR+AUG and SAC+CI+AUG are capable of ignoring changes in floor texture, and identifying the agent as it shifts color. The large improvement in generalization between SAC+AUG and and it's cearier-augmentationn variants is investigated further Section 4.4.
100 visual seeds, cach with 3 dynamics

---

Under review as a conference paper at ICLR 2021
None
Light
Caméra
Body
Floor
Background
None
Light
Camera
Body Color
Floor
Background
All
SAC+AE
647.0
610.5
489.9
334.5
105.3
107.5
99.2
Cup. Catch
CURL
822.5
707.0
766.9
342.3
169.4
106.5
109.2
Ball
SACFAUG
931.5
918.9
873.1
261.0
147.1
118.6
103.0
+AUGGJ+AUG
955.6
954.9
889.6
928.2
949.3
509.4
322.I
SAG
948.5
947.8
864.9
937.6
932.6
582.3
411.2
Walker,
SAC+AE
643.6
538.0
145.9
316.9
31.7
53.8
25.0
Walk
CURL
622.7
172.8
532.6
231.7
25.8
16.3
25.3
SAC+AUG
919.3
830.1
518.3
550.7
31.1
106.2
28.0
SAC+CJ+AUG
925.1
905.4
459.6
852.6
535.3
240.3
71.9
SAC+AE
622.1
615.7
76.4
347.4
8.1
38.8
0.1
Finger; Spin
CURL
731.3
716.9
521.1
433.7
29.6
53.6
1.6
SAC AUG
916.7
873.2
441.7
525.3
170.3
98.1
7.1
SAC+CI+AUG
798.5
796.4
388.8
760.6
795.7
274.6
82.8
BAC+NR-AUG
797.6
788.0
420.6
779.5
795.5
332.4
154.7
Cheetahy Run
SAC+AE
387.9
252.9
36.4
252.6
65.8
75.9
4.2
CURL
202.3
142.6
137.9
109.3
18.5
32.1
2.4
SAC+AUG
712.7
321.8
259.3
171.3
127.9
310.6
19.8
SACFCJ+AUG
669.2
406.3
278.9
620.6
144.3
359.7
47.4
Cartpole, Balance
SAC+AE
942.8
810.8
377.2
492.6
313.8
254.2
212.3
CURL
885.7
789.9
831.7
521.2
403.9
261.4
233.3
SACTAUG
987.4
963.0
878.1
655.2
565.1
422.2
218.8
FAC
981.2
920.9
876.5
865.5
978.3
457.1
358.6
SAC+NR-AUG
946.0
859.8
835.6
860.9
923.2
462.5
377.4
Hopper, Stand
SAC+AE
631.8
616.4
7.7
260.4
6.3
1.4
1.6
CURL
425.5
311.9
202.7.4
72.1
12.4
4.7
1.9
SACFAUG
878.1
764.
195.2
403.4
46.0
39.5
3.3
SAC+CJ+AUG
848.3
778.6
188.2
693.4
144.5
50.5
Table Return pixel-based SAC variants in tasks from DMCR. Agents are trained on the default visuals ("None"), ), and evaluated in versions of the environment where one aspect of the scene ("Light", "Floor", ...) is randomly adjusted. Testing lasts for 300 episodes across 100 distinct visual seeds. Reported scores are the average of five experiments.
4.4 ANALYZING THE SUCCESS OF DATA AUGMENTATION METHODS
To understand the effect that augmentations like Color Jitter and Network Randomization have on generalization, we take closer look their performance relative to the default DrQ crop. While SAC+AUG generalizes poorly to changes in floor texture in "Ball in Cup, Catch" (Table 2) (Eg 84.2%), SAC+NR+ generalizes nearly perfectly (Eg 1.7%). We speculate that the random color changes introduced by the NR augmentation reduce the variance of the encoder's state representation w.r.t changes in floor texture and color. We can measure this by using DMCR to generate batch of observations of the exact same state, rendered with hundreds of different floor patterns. We pass these observations through the encoder of trained SAC+AUG and SAC+NR+AUG agents, and collect the resulting state representations. An encoder with perfect visual generalization would have zero output state vector for every observation. We plot the
variance, meaning it predicts the standard deviation of the agents' representations in Figure 3. Because the encoder's indices sort them by increasing variance. We average the results of this experiment over five trained agents. Network Randomization learns much more consistent representation of this state than the DrQ crop augmentation alone. This process is repeated for the SAC+CJ+AUG agents in "Cartpole, Balance" with very similar results.
have no meaningful order,
Finally, compare the spatial activation maps of SAC+AUG and SAC+CJ+AUG encoders in "Ball in Cup, Catch", and find that the CJ agent much less distracted by a change in floor texture. Results can be found in Appendix A.1 Figure/ 4. We argue that the zero-shot generalization benchmark provided by DMCR is a better use-case for significant.

---

Under review as a conference paper at ICLR 2021
Ball in Cup, Catch
Cartpole, Balance
SAC+AUG (6gg 84.2%) SAC+NR+AME (Eg 1.7%
SAC+AUG (Eg 42.75%
SAC+C]+AUG (Es -66%)
53
2.4
crerted)
Figure 3: Standard deviation the encoder network's output across renderings of the same state with different floor textures and colors. Indices sorted by increasing variance. Results averaged over five agents.
image alterations like CJ and NR than the typical single-environmenn setting, where simple random crops have become the default (Laskin et al., 2020) (Kostrikov et al., 2020). By randomizing color during training, they act a kind of self- domain randomization (Tobin et al., 2017) (Mchta et al., 2019) (OpenAI et al., 2019) and regularize learning by increasing generalization across tasks at the expense of performance This is also supported by the success of NR and CJ on Proegen (Raileanu et al., 2020) (Cobbe et al., 2019a), where success involves similar amount of visual generalization.
the training task.
5 RELATED WORK
The generalization of Deep RL methods an active area of research. Agents have been shown to overfit to deterministic environment elements (Zhang et al., 2018a), despite efforts to add stochasticity during evaluation (Zhang et al., 2018c) (Machado et al., 2017). Other work focuses generalization across environments with different transition dynamics, par- ticularly in the continuous control setting (Packer et al., 2019) (Zhao et al., 2019) (James et al., 2019) (Plappert et al., 2018), and recommends evaluation protocol where agents' generalization is assessed based on their performance in a held out set of tasks (Cobbe et al., 2019b). This has led to the development of benchmarks with a large or infinite (procedurally generated) number of similar environments (Nichol et al., 2018) (Juliani et al., 2019) (Juste- sen al., 2018). Notable among these is Procgen (Cobbe al., 2019a) meant to imitate the strengths of the Atari 57 benchmark (Bellemare et al., 2013) but with procedurally generated level layouts and visuals. Unlike the DMCR environments discussed in this paper, tasks in the Procgen benchmark have different state spaces and transition dynamics, and demand a more universal form of generalization; DMCR tests perception and observational overfitting in isolation by keeping the underlying task fixed. Procgen's discrete action space also lends itself to a different family of algorithms than are often used in the contimuous control setting.
set of games
Other work has investigated visual generalization by adding spurious visual features to popular RL environments (Gamrian & Goldberg, 2019) (Roy & Konidaris, 2020) (Sonar et al., 2020). This paper, and the DMCR benchmark, most directly inspired by (Zhang et al., 2018b), as well as experiments in (Zhang et al., 2020) and (Yarais et al., 2019), where the observations of similar continuous control tasks are processed to remove all of the floor and background pixels and replace them with natural images video. This is usually accomplished by masking out the color of the background after the observation has been rendered. DMCR takes this idea step further, and directly changes the graphical asscts of the environment. This design gives it more control over the tasks' appearance and opens up the opportunity for changes in lighting and camera position. It is also the first time this concept has been made into reproducible benchmark for future research.
CONCLUSION
In this work, propose challenging benchmark for measuring the visual generalization of continuous control agents by expanding the graphical variety of the DeepMind Control Suite. Our experiments show that recent advances in representation learning in RL not enough to achieve a high level of performance across a wide range of conditions, but suggest that data augmentation may be promising path forward.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems, pp. 81 93. IEEE Press, 1990. ISBN 0818620153.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253 279, 2013.
Greg Brockman, Vicki Choung, Ludwig Pettersson, Jonas Schncider, John Schulman, Jic Tang, and Wojeicch Zaremba. Openai gym, 2016.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint 1rXiv:1912.01588, 2019 19.9.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying gencralization reinforcement learning, 2019b.
Terrance Devries and Graham W. Taylor Improved regularization of convolutional neural networks with cutout. ArXit, abs/ hoss1708.0055,, 2017.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018.
Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to-image translation, 2019.
S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic ma- nipulation with asynchronous off-policy updates. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3389 3396, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications, 2018b.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment, 2019.
Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng. Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vision, control, and planning, 2019.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmnd Khalifa, Julian To- gelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation, 2018
Dmitry Kalashnikov, Alex Trpan, Peter Pastor, Julian Tbarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manip- ulation, 2018.
llya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels, 2020.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data, 2020.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization in deep reinforcement. learning, 2019.
9

---

Under review as a conference paper at ICLR 2021
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2015.
Zhichao Lu, Ian Whalen, V. Boddcti, Yashesh D. Dhebar, K. Deb, E. Goodman, and W. Banzhaf. Nsga-net: ncural architecture search using multi-objective genetic algo- rithm. Proceedings of the Genetic and Evolutionary Computation Conference, 2019.
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents, 2017.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J. Pal, and Liam Paull. Active domain randomization, 2019.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533-3 2015.
A. W. Moore. Efficient memory-based learning for robot control. 1990.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization rl, 2018.
OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob Mc- Grew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with robot hand, 2019.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning, 2019.
Hicu Pham, Mclody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient noural architecture search via parameter sharing. In ICML, 2018.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.
Roberta Raileamu, Max Goldstein, Denis Yarats, Tlya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning, 2020.
Josh Roy and George Konidaris. Visual transfer for reinforcement learning via wasserstein domain confusion, 2020.
John Schulman, Philipp Moritz, Sergey Levino, Michael Jordan, and Picter Abbecl. High- dimensional continuous control using generalized advantage estimation, 2015.
Anoopkumar Sonar, Vincent Pacelli, and Anirudha Majumdar. Invariant policy optimiza- tion: Towards stronger generalization in reinforcement learning, 2020.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement. learning, 2019.
Aravind Srinivas, Michael Laskin, and Picter Abbeel. Curl: Contrastive unsupervised rep- resentations for reinforcement learning, 2020.
Ryo Takahashi, Takashi Matsubara, and Kuniaki Uehara. Data augmentation using random image cropping and patching for deep cmns. ArXiu, abs/181 1111111111000,,,.1 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018
10

---

Under review as a conference paper at ICLR 2021
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robols and Systems, pp. 5026 5033, 2012.
Qizhe Xie, Zihang Dai, E. Hovy, Minh- Sahh-Thang Luong, and Quoc V. Lc. Unsupervised data augmentation. ArXiv, 198/1190..2248, 2019.
Denis Yarats, Amy Zhang, Nya. Kostrikov, Brandon Amos, Joelle Pincau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images, 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer, 2017.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. dissection overfitting and generalization in continuous reinforcement learning, 2018a.
Amy Zhang, Yuxin Wu, and Joelle Pineau. Natural environment benchmarks for reinforce- ment learning, 2018b.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction, 2020.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning, 2018c.
Hongyi Zhang, M. Cissé, Yann Dauphin, and David Lopez-Pas Paz. mixup: Beyond empirical risk minimization. ArXiv, abs/ abs/1710.00941. 2018d.
Chenyang Zhao, Olivier Sigaud, Freek Stulp, and Timothy M. Hospedales. Investigating generalisation in continuous deep reinforcement learning, 2019.
Z. Zhong, L. Zheng, Guoliang Kang. Shaozi Li, and Y. Yang. Random crasing data aug- mentation. In AAAI, 2020.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning, ArXie, abs/1611.00157, 2017.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 SPATIAL ATTENTION MAP RESULTS
Using a method described in (Zagoruyko & Komodakis, 2017), we analyze the spatial atten- tion maps of SAC+AUG and SAC+CJ+AUG encoder networks in "Ball in Cup, Catch." The activations of intermediate convolutional layers are resized and superimposed over the original observation, giving us a sense of what. the agent. paying attention to. In the training environment, both agents are fixated the path of the ball", as well the closest edge the cup. However, SAC+AUG becomes distracted by a change in floor texture, while the Color Jitter variant remains focused on the task. This provides some insight into why AAG performs so much better than the default SAC+AUG when testing over random floor changes (see Table 2).
SAC+AUG, Train Seed
SAC+AUG, Test Seed
SAC+C] Train Seed SAC+C]+AUG, Test Seed
Figure 4: Spatial attention maps of trained SAC+AUG and SAC+CJ+AUG agents. Com- puted by taking the channel-wise average of encoder layer activations, and overlaying them on the original observation. Green and red heatmap colors indicate high levels of atten- tion. Both agents trained with the default chockerboard floor, but the CJ agent is less distracted by a change to a concrete texture.
A.2 CONNECTING TO GENERAL DATA AUGMENTATION
Data augmentation synthesizes new training data from an available set of samples and pro- vides a practical and common way achieve better generalization. Basic data augmentation strategies like random affine and projective transformations, and color jittering, have shown significance in computer vision applications.
We can group existing data augmentation methods roughly into four groups: (1) spatial or color transformation, (2) information deletion, (3) mixing up samples, and (4) meta- augmentation.
(1) Spatial transformation and color distortion revise some channels of original image information (Takahashi al., 2018). This type helps the training set better simulate the real-world.
(2) Information dropping type is more recent, including methods like cutout (Devries & Taylor, 2017) and random erasing (Devries Taylor, 2017; Zhong al., 2020). This kind tries to enforce the representation learning to focus on less sensitive signals, aiming for more robust models.
(3) Mixup Zhang et al. (2018d) proposed the idea of interpolating two images and their ground truth labels to augment the training data. Mixup and its variants aim for stable model training and incrcased model robustness adversarial examples.
(4) More recently, researchers have proposed methods to automatically search for data augmentation policies with Reinforcement Learning (RL) or Evolutionary Algo- rithms (Zoph & Le, 2017; Lu et al., 2019; Pham et al., 2018; Xie et al., 2019).
2The high-attention zones trailing the hall's current position are result of past timesteps in the framestack that make up this observation, which not pictured here.
12

---

Under review as a conference paper at ICLR 2021
Data augmentation investigated in this paper fall into the first first types. We foresce great potentials for benchmarking the latter two groups to reinforcement learning and will investigate shortly.
A.3 AUGMENTED SOFT ACTOR CRITIC: DETAILS AND ABLATIONS
SAC+AUG is custom implementation meant to combine several of the ideas from recent work on data augmentation in RL. It uses the augmentation process described in RAD (Laskin al., 2020), which was chosen for its simplicity: observations they cxit the replay buffer, and then used in a standard SAC learning update. We make parameter control how the augmented data is mixed into the update batch, and a fealure-matching loss term.
augmented as
two additions:
A.3.1 3 PARAMETER TO ALLEVIATE TEST-TIME AUGMENTATION
Several methods apply their augmentations observations during both training and eval- uation. In Lee et al. (2019), the agent's actions are average across several different augmentations of the current observation. CURL and RAD render the environment in a higher resolution in order to efficiently perform their random crop augmentations, which requires a center crop during testing. We experiment with skipping this step by mixing un-augmented images back into the gradient update, controlled by 3 E [0, 1]. However, results in Figure 5 show this to be unimportant, at least over short training runs. Our main experiments do not use test-time augmentation, and fix β = .9.
8 with RAD in Walker, Walk
B with NR in Cartpole, Balance
600
9=
.95
800
300
200
500
20000
40000
60000
50000
100000
20003
40000
60000
50000 100000 120000 140000 150000 Envrrmmment Steps
Envinnmment Shepss
Figure 5: Testing a range of 3 values on "Walker, Walk" and "Cartpole, Balance." " We find there to be little difference over short training runs.
A.3.2 STATE REGULARIZATION ABLATION
The encoder's goal is distill the high-dimensional image observations into a low- dimensional representation that, ideally, would recover the information present in the state- based versions of these tasks. The observational changes caused by data augmentation have no effect the true state, so would hope that they would be equally irrelevant to the encoder's representation. Therefore, we regularize the output of the encodor éç to be invari- ant to augmentation pipcline by adding adding norm-based penalty that is mixed into the encoder's loss with a hyperparameter A:
Lencoder = -Loritic + A E oND
(4)
We compare several choices of X in Figure 6.
A.3.3 AUGMENTATION COMPARISON
We consider 11 augmentations to serve as the SAC+AUG default. Any randomness (e.g. where to crop, how far to move) is kept consistent between the same batch index in the o and of tensors, meaning the o and o' pairs that go into the critic update have identical transformations. Observations consist of stack of 3 frames. Random elements also con-
13

---

Under review as a conference paper at ICLR 2021
Parameter Search Walker Walk
3e-3
250
1e-3
1e-5
Retum
150
50
20000
40000 Environment Steps
60000
80000
100000
Figure 6: Testing a range of X values on "Walker, Walk" with 3 = averaged over trials.
sistent across frames in this stack. This requires augmentations that can be deterministically scoded.
brief summary of cach transformation:
1. RAD imitates the augmentation in (Laskin et al., 2020) by resizing the 84x84 observation to 100x100, and then randomly cropping back to the original size.
2. Cutout Color selects rectangular patch of the image occlude with a random color mask.
3. DrQ imitates the augmentation in (Kostrikov et al., 2020). The image padded by pixels by replicating the colors at the border, and then randomly cropped back to 84x84. Kostrikov et al. (2020)'s 's implementation uses resampling method that turns each pixel into a non-integer float. The need for deterministic seeding prevents from using that version, so we mimic the effect by adding small amounts of noise.
DrQ No Noise uses the same pixel pad/crop as DrQ, but without the additional noise.
5. Large DrQ pads and crops by 12 pixels instead of 4. Figure 8 uses Large DrQ for visual effect.
6. Large DrQ No Noise pads and crops by 12 pixels and does not add noise.
7. Translate pads the image with 4 pixels of random color, and then shifts the image within this expanded area before center-cropping.
Large Translate pads and shifts by 12 pixels instead of Figure 8 uses Large Translate for visual effect.
9. Rotate rotates the image by one of {0",90 90", 180%, 270"}
10. Vertical Flip rotates portion of each batch by 180
11. Window selects rectangular viewing lens to see the original image and sets all other pixels to 0.
An implementation of each transformation (many based on the RAD Procgen (Laskin et al., 2020) experiments) is available in our code release. See Figure 8 for examples.
Figure shows the learning curves of all augmentations over the first 100k steps of the "Walker, Walk" task, averaged over 5 trials with 8 = .9 and X = .0015. In general, all pad + crop augmentations perform well; the additional noise does not appear to be an important detail. Rotations and flips slow down training significantly. Window and Cutout Color also perform poorly. This somewhat counter intuitive, as these transformations hide information from the image much like Translate and DrQ. However, they tend to mask
14

---

Under review as a conference paper at ICLR 2021
Augmentation Comparison on Walker Walk 100k
350
Augmentation
300
Cutout Color
DrQ
DrQ Noise
250
DrO
Large DrO No Noise
Return
200
Large Translate
Rotate
Translate
Vertical Flip
Window
50
20000
40000
60000
80000
100000
Environment Steps
Figure 7: All the minor augmentations tested for 5 trials on "Walker, Walk" with $ and À = .0015. Figure best viewed in color.
large portions of the input, and might be making the lask too difficult. This is supported by the reduced performance DrQ and Translate when their crop factor raised to even 12 pixels.
The limited sample size makes difficult to distinguish between DrQ and Translate. We decide to use DrQ based on the intuition that the color-replicating padding strategy will fare better in DMCR, where becomes more important to disregard the background.
Alterations
Augmentations
or jitter
Flip
Network Randomization
Intrabatch Style Tranfer
Cutout Color
Rotate
Window
Random Style Transfer
Grayscale
Translate
Horzzntall Flip
Cutout
Figure 8: Examples of our augmentation implement We calegorize the (ransfor- mations into two groups. "Alterations" make significant enough visual changes that the result could conceivably be from another emission function. "Augmentations" reduce over- fitring by occluding or moving the image. Our code release includes several augmentations not mentioned in the main results, which were either less effective too computationally expensive to allow for multiple training russ.
15

---

Under review as a conference paper at ICLR 2021
A.4 ADDITIONAL BASELINE DETAILS AND HYPERPARAMETERS
We use the publicly available implementations of CURL, SAC+AE, and DrQ. Frame skip settings are chosen based on (Srinivas et al., 2020), and listed in Table 3 for convenience. See Table 4 for a list of other hyperparameters.
Training Steps
Frame Skip (Action Repeat)
Walker, Walk
250k
2
Cheetah, Run
250k
4
Hopper, Stand
250k
4
Finger, Spin
100k
2
Ball in Cup, Catch
75k
4
Cartpole, Balance
50k
8
Table 3: The frame skip settings used in all of our experiments, along with the length of training runs used to compute the results in Tables and 2.
Hyperparameter
Value
Frame Stack
3
Image Size
84
Replay Buffer Capacity
100,000
Warmup Steps
1,000
Batch Size
256, 128 (SAC+AE)
Actor
2x ReLU(FC(1024)) Tanb(FC(outt)
Crities
2x ReL.U(FC(1024) + FC(out)
Encoder
ReLU(Conv(32 filters, stride 2)) - 3x ReLU(Conv(33 fil- ters, stride 1)) - FC(50) + Tann(LayerNorm
Critic Learning Rate
1e-3, 2c-4 (CURL Cheetah)
Actor Learning Rate
le-3, 2c-4 (CURL Cheetah)
Encoder Learning Rate
1o-3, 2c-4 (CURL Choetah)
Initial Alpha
.1
Target Entropy
-|A|
Alpha Learning Rate
le44
Action Log Std Range
(-10, 2)
Encoder Tau
.05
Actor Tan
.01
Critic Tau
.01
Gamma
.99
Gradient Updates per Step
1
Update Delay
2
Table 4: Hyperparameter settings used in all experiments.
A.5 DMC REMASTERED: DETAILS AND EXAMPLES
This initial version of DMC Remastered includes 10 of the DeepMind Control Suite domains for a total of 24 tasks. There are 300+ floor textures and 100+ backgrounds. Early results showed this to be more than enough of challenge, but there is nothing to prevent. (many) more images from being added in the future.
Each environment's 's visuals are generated by sequence of pseudo-random decisions, deter- ministically seeded by the 'visual seed.' The seeds are not guaranteed to generate the same visuals across domains, although they are consistent across tasks within the same domain (e.g. seed 11 in Fish, Swim will look identical to seed 11 of Fish, Upright). Real-valued parameters like the camera's x, and coordinates or the light diffusion are sampled uni- formly from a preset range of values that varies by domain. These ranges were determined by generating large number of images and ensuring that there was sufficient variation without making the task unsolvable by hiding important information. Several of these dis- tributions far conservative than they could be, in an effort to make this useful
16

---

Under review as a conference paper at ICLR 2021
t=0
t=1
t=2
t=3
t=4
t=5
(()
(s)
16:08
106880 (S)
Figure 9: DMCR allows the same state trajectory to be rendered in millions of distinct. visual styles. Here show 4 examples from the "Hopper, Hop" task. Our benchmark can always recover the original DMC environment by setting the visual seed to O.
benchmark for the abilities of current methods. In particular, the camera is restricted to minor translations and tilts, although it would be interesting to see if agents could maintain control across significantly different perspectives. Slight movements in camera position al- ter the raw pixel values of the observations without meaningfully changing the information within them. Moving the camera to angles may create more difficult challenge where the agent needs to adapt to an entirely different observation. Not every factor variation applicable every domain; default DMC assels, however, some tasks alter the default camera angle in an elfort. to fit. more of the background floor in the frame.
Table 5 for complete list. The 0 seed is reserved for the
The DMCR benchmark also includes a version of the environment that samples from set of visual seeds after every reset convenient way to train and evaluate few-shot generalizers. Training seeds are sampled from fixed range while testing seeds are drawn up to a large positive constant. Initial experiments suggested that successful few-shot learning requires high-throughput training setup. We extended each of the basclines described in See 3.1 to collect experience from 8 actors in parallel, each interacting with different random seed. We found that this was not enough to see positive results on all but the most straight.orward domains. Due to concerns the few-shot environment remains in the code release, and we hope it can be helpful in future work.
cost, we shifted focus to the zero-shot setting. However,
17

---

Under review as a conference paper at ICLR 2021
Camera
Light
Body Color
Target Color
Background
Floor
Reflectance
Cheetah
Yes
Yes
Yes
No
Yes
Yes
Yes
Walker
Yes
Yes
Yes
No
Yes
Yes
Limited Effect
Hopper
Yes
Yes
Yes
No
Yes
Yes
Limited Effect
Cartpole
Yes
Yes
Yes
No
Yes
Yes
No
Finger
Yes
Yes
Yes
No
Yes
Yes
No
Pendulum
Yes
Yes
Yes
No
Yes
Yes
No
Fish
Yes
Yes
Yes
Yes
No
Yes
No
Humanoid
Yes
Yes
Yes
No
No
Yes
No
Ball in Cup
Yes
Yes
Yes
No
Yes
Yes
No
Reacher
Yes
Yes
Yes
Yes
No
Yes
No
Table 5: Factors of variation that are applicable to each environment. The cam-
era position, lighting conditions, body color and floor texture randomizations are always available. Some tasks use birds-eye-view default cameras that do not show the background. "Reflectance' adjusts the visibility of the agent's reflection on the floor, a very small or non-existent change in most environments that is included as way to screen for extreme overfitting. 'Adding camera movement to the Reacher environment will show some of the background (see Figure 12).
45(5g)
(1716(56)
(52))(5))
(22)
(5y(5p)
c5 (5p
3599
C sp (5p)
@2300
(50)
Laaso (5g)
P57921
(50g
(52
20 (5g)
(242
Poo (5))
(53
(573
dree9 (5)
1o1 (5p)
(54
17215 (5g)
Figure 10: Example seeds from the DMCR version of "Cartpole, Swingup."
18

---

Under review as a conference paper at ICLR 2021
4b15g)
#21e15g1
(
Panoa (5g)
(5195150)
(1uve(Fo)
Pes4s18gg
P
@nono (Sa)
P #r1)
rnall
(ress(So)
(pars(5n)
$1.2
(54
Figure 11: Example seeds from the DMCR version of "Walker, Walk." 17
(u(s))
Asssa(Sa)
(S1)s(5g)
(So))
@200(5))
(111)(5g)
Qa4rs Sig)
@baga Sp)
As105
40o12 (Sa)
#200a Sil)
P41as
(5o)s(5o)
Paass
Prese So)
(Sa
(Sa)
(S.)
Prass (5a)
(54)
(S)
17191
Figure 12: Example seeds from the DMCR version "Reacher, Easy." The birds-eye view reduces visual complexity considerably. Note that the target position is given its own domized color.
19

---

Under review as a conference paper at ICLR 2021
(5
(591
(5689(5o)
(5337(Sp)
(1159(5o)
(595
(5
(4333(50)
Peeee
(4979
44230(50)
P1775(50)
(1696(50)
2600 (Sp)
06547(50)
(5644
(9539
66304 (5)
5557 (5))
3335
@2607 (5g)
(5032(50)
(51
Pas5 (5g)
0
Figure 13: Example seeds from the DMCR version of Finger, Spin." "
20


Reviews:
[{"review_id": "vmPm1AySLGX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### **Summary and Contributions of Paper** \nThis paper proposes a new RL Generalization benchmark based on the DM Control Suite, where there are multiple changing backgrounds but fixed dynamics. Multiple baselines, most of which involve data augmentation for RL, are tested and shown to possess poor performance on this benchmark, which suggests that it is challenging and meaningful enough to be used for future research.\n\n### **Strengths**\n- This benchmark appears to be easy to use, and highly accessible once released.\n- The baselines used are comprehensive, and contain multiple recent and competitive methods, especially in data augmentation.\n- This benchmark is highly motivated, as it is important to cleanly separate the visual and dynamics factors of RL generalization, and provides good background/related work.\n- Paper is written well and is easily understandable.\n\n### **Weaknesses**\n- Unfortunately, there have already been multiple benchmarks proposed with the \"changing background\" setting, such as in (Zhang et al 2018b, Sonar 2020). Nearly identical DeepMind Control/background benchmarks have been proposed in (Zhang et al 2020). Therefore, this paper will need to emphasize its novelty component. Some ways to provide novelty are described in the next points.\n- One unfortunate trap that many RL generalization benchmark papers fall into, is the pattern of \"Propose benchmark, test algorithms/hyperparameters\", without providing revelations/ablations as to why such methods work conceptually, or why certain phenomena exist. Many times it is difficult to tell if problems in generalization occur specifically with the certain benchmark, or actually occur at a broad level from some deep reason. Differentiating from the \"N+1-th changing background benchmark\" would allow this paper to provide much more impact.\n- It would be helpful if the authors could provide more insights (other than only showing reward numbers) into why overfitting occurs in this benchmark, such as e.g. 1. Saliency plots on what the RL agent focuses on, 2. Properties of the policy/SAC algorithm, such as how different data augmentations affect its optimization process.\n\nOverall, I currently give a score of 6 (marginally above acceptance) because the usefulness/assumed accessibility of this benchmark outweighs the weaknesses, but I would be happy to increase my score if the authors address some of my concerns.\n\n### **Clarity Questions**\n- One of the most important details I was looking for was how/if the background moves compared to the agent, i.e. what is the precise definition of \\phi? This is very important, as from experience in running these types of benchmarks, overfitting does not usually occur when the background moves independently to the agent (e.g. when the agent is stopped, the background can still keep moving). I assume from Fig. 8 that the background moves along with the agent, and thus forward progress (via reward function) can be spuriously correlated with the background, but could you please clarify this point?\n\n\n### **References**\n- https://arxiv.org/pdf/1811.06032.pdf (Zhang et al 2018b)\n- https://arxiv.org/abs/2006.10742 (Zhang et al 2020)\n- https://arxiv.org/pdf/2006.01096.pdf (Sonar 2020)\n\n### **Rebuttal Update**\nI thank the authors for updating their draft. I think this paper is worth accepting due to its ease of use and many different features, but I will keep my current score at a 6.\n\n**Here's why:**\n\nMy experience in these types of benchmarks have usually shown that **completely static** backgrounds (when cleanly used - there can be very subtle things that can still correlate with progress) do not actually affect generalization, but rather, anything correlated with progress will cause overfitting. The difference between this benchmark's background and e.g. a ProcGen/CoinRun's background is that CoinRun's background still moves when the character moves (which implies a slight correlation with forward progress).\n\nFrom the authors' responses to Q 3.2 and Q 3.3, I believe that the main visual overfitting is occurring with the floor tile, as it is the only spurious object that is correlated with progress. This is because, as the authors have stated, the actual background image does not actually change when the agent is moving as the background is too far away. The authors also reinforce this aspect when they added the saliency/spatial attention maps in Appendix A.1, which shows that most of the attention is focused on the floor tiles, especially on test environments.\n\nThis means that I suspect that the randomized static background portion of the benchmark does not affect generalization, and I urge the authors to rethink this portion of the benchmark. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "nv3IM_jTTO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper extends the DeepMind Control Suite benchmark by adding a series of visual variations across different tasks (e.g. lighting, color, background, textures, camera angles etc.). They also contrast a few recent self-supervised learning and data augmentation RL methods and measure these agents’ ability to generalize and transfer across a variety of visual variations provided by their benchmark. They also investigate different aspects of the variation in the scene that these agents seem to be most affected by.\n\nOverall, this paper is clearly written and the analysis is very thorough and well motivated. However the results in my view are not that surprising: the observation that agents that have been trained only on a 'single' visual variation of an environment overfit to the visual features and fail to generalize outside of this distribution is not particularly novel. While I appreciate the amount of work that has been done here and the helpful analysis, contrasting different state of the art methods on self supervised learning, I have some questions/concerns:\n\n1. There are a variety of existing benchmarks that allow for variation of the environment visual properties as well as variation in task distribution (e.g. [RLBench](https://arxiv.org/abs/1909.12271) & [OpenAI Fetch environments](https://arxiv.org/abs/1802.09464)). While Control Suite is one of the most widely used benchmarks for contrasting continuous control methods, I’m not convinced that it is on its own the best setting for studying visual generalization of agents as the majority of the tasks involve only a single agent/object and the appearance of agent/object in the environment is often not relevant to the task (making it a rather extreme example). I believe in order to better understand which methods allow for better generalization, one needs to combine tasks with different properties.\n2. To my understanding the agents are trained only on a single instance of visual properties of the environment while tested on a large range of variations in the visual properties of the environment at test time. Maybe a more realistic test would be to allow a small variation of the environment during training time and then assess the generalisation gap?  \n3. Related to my first point, I would like to perhaps better understand what is the desired outcome for this benchmark. I am concerned that in order for some method to avoid overfitting to spurious correlations in such tasks, this might result in some trivial task-specific heuristics for augmenting the data, which would perhaps solve this benchmark while still failing to generalize to more interesting/complex settings involving many objects where some visual properties of the environment are indeed task relevant. Could you comment on that? \n\nIn summary, I enjoyed reading this paper and I believe the authors have done a nice job of contrasting different self-supervised learning algorithms in a challenging benchmark. I’m not fully convinced of the impact of the benchmark on future research but I think there are still interesting takeaways that would be valuable for the community.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "l1EYKS4p_yp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new dataset for continuous control tasks based on the DeemMind control suite, but with varying backgrounds, colors and lighting/camera conditions. Instead of the fixed appearance in the original suite the frames from this dataset contain a lot of varying structure. The purpose of the dataset is to measure the generalization and transfer abilities of pixel-based continuous control agents across different visual variations.\nThe paper presents a comparison between several different representation learning method for CC models and concludes that augmentation plays a big role in making such models generalize well.\n\nAll in all I think this is an interesting contribution and the dataset may be useful for the purpose it was created, but I have several reservations.\n\nFirst, I would say that this is still a rather simplistic notion of visual variability - especially the choice for not changing the camera positions in a substantial manner - this makes the task of the visual representation much easier and by \"ignoring\" color (as would be the result of color augmentation) most of the generalization problem really does go away. Demonstrating the ability of methods to be robust to more severe changes in the viewpoint would be a harder and more convincing test case (as it affects, directly, some task-relevant properties which can be extracted from the scene). Also - maybe changing the actual *appearance* of the limbed model (say, by skinning it) would have been more interesting - the underlying dynamics would stay the same but the appearance would change greatly while still be a deterministic function of the underlying state.\nFinally, in terms of the baselines used and the results shown - I think it would have been good to show another RL method other than SAC - just to make sure results (in terms of augmentation and generalizablity) to not depend greatly on the specific choice of underlying RL algorithm.\n\nTo conclude - I think this is a nice contribution with a potential to be even nicer.\n\n*Post Rebuttal*\n\nThe authors have mostly addressed my concerns - I still think SAC should not only be the only RL method in the paper, and though I'm still not convinced more drastic camera variations wouldn't make this more interesting I think all in all this is a decent contribution and probably should be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IHCeoJcDP1M", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a \"remastered\" version of the Deep Mind Control suite that allows for testing of visual generalization to a number of environmental changes (e.g., pattern/texture of the floor, background, and lighting).  \n\nI will begin my review by stating that I have somewhat limited knowledge of soft actor critic, Deep Mind Control and some of the most recent related work in this space and my review should be taken with that caveat.  However, the proposed forms of data/simulation augmentation that are proposed seem quite useful as a way of moving the Deep Mind Control suite from towards a more realistic and richer set of visual data. While this paper seems positioned as a more systematic baselining and data set paper and the novelty from an methodological perspective is somewhat low.  I think these are helpful in increasing how challenging the baseline is and providing more sight about how well the current methods perform.  \n\nHaving said that I think there are some ways the paper could be improved.  The link to the anonymous page wasn't working for me, I tried several PDF viewers but couldn't access it.  Also the image assets did not seem to be included in the supplementary material.  The main reason I was looking for these is that the details in section 3.3. are quite limited.  If possible I would like to be able to gain more insight about the diversity of the types of augmentation.  I appreciate that spaces is limited in the paper but showing an image with small thumbnails seems like it could be an option.   I link to the assets would be very helpful.\n\nAnother element that is a little unclear to me is how the backgrounds change as the characters move.  Are these full 3D environments or 2D backgrounds?  I am referring mainly to the \"Background\" elements. By the description it seems that they are photo graphs and therefore they don't capture 3D elements. \n\nThe literature review around the use of simulation and/or graphics to created augmented data for interrogating models is a little bit limited.  I understand that these augmentations might be novel in the domain of RL, however, I think it is worth noting how adding conceptually this is something that has been done in past in several computer vision domains/tasks.  Not that that necessarily diminishes the utility of the proposed dataset.\n\nTables 1 and 2 don't have clearly labeled units.  This might seem minor but is a constant frustration for readers.  \nFor Figure 2. The it isn't clearly indicated what the error regions represent or the units of the axes.\n\nTo summarize, I think that this work is a helpful contribution, albeit with somewhat limited innovation.  I think more information about the data augmentation and positioning in the space of other uses of graphics/simulation for testing and augmenting vision data might be helpful. I would argue the paper as it stands is borderline and needs some improvements, but those could be achieved with revisions making it a good contribution.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a modification to the DeepMind Control Suite to measure generalization with respect to visual variation. The authors run baseline experiments against their new benchmark and discover, unsurprisingly, that RL agents learning from visual observations overfit to spurious details of the observations.

Reviewers generally found the work to be clearly written, and the experimental analysis to be thorough and well done, though concerns about the rather simple nature of the visual augmentations persisted even after updates and author rebuttals. There were also concerns that by focusing only on Soft Actor Critic in the experiments.

3 of 4 reviewers felt the work met the acceptance bar, albeit only marginally. The dissenting reviewer's concerns centered on clarity (many specific issues appear to have been remedied), the relatively limited nature of the augmentations, and the fact that reviewers were not given access to the code. 

While the submission has potential, improvements needed are not minor, and given the short process, we can only accept papers as is, rather than expecting certain changes. Please take the reviewers' comments into consideration as you revise and resubmit to a future venue.
Decision: Reject