Robust Domain Randomization for Reinforcement Learning
Authors: Reda Bahi Slaoui
William R. Clements
Jakob N. Foerster
Sébastien Toth
Venue:  (2020)
Keywords: reinforcement learning
domain randomization
domain adaptation

Abstract: Producing agents that can generalize to a wide range of environments is a significant challenge in reinforcement learning. One method for overcoming this issue is domain randomization, whereby at the start of each training episode some parameters of the environment are randomized so that the agent is exposed to many possible variations. However, domain randomization is highly inefficient and may lead to policies with high variance across domains. In this work, we formalize the domain randomization problem, and show that minimizing the policy's Lipschitz constant with respect to the randomization parameters leads to low variance in the learned policies. We propose a method where the agent only needs to be trained on one variation of the environment, and its learned state representations are regularized during training to minimize this constant. We conduct experiments that demonstrate that our technique leads to more efficient and robust learning than standard domain randomization, while achieving equal generalization scores.

Under review as a conference paper at ICLR 2020
ROBUST VISUAL DOMAIN RANDOMIZATION FOR RE- INFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Producing agents that can generalize to a wide range of visually different envi- ronments is significant challenge reinforcement learning. One method for overcoming this issue is visual domain randomization, whereby at the start of each training episode some visual aspects of the environment are randomized so that the agent is exposed to many possible variations. However, domain random- ization is highly inefficient and may lead to policies with high variance across do- mains. Instead, we formalize the visual domain randomization problem, and show that minimizing the policy's s Lipschitz constant with respect to the randomization parameters leads to low variance in the learned policies. We propose regulariza- tion method where the agent is only trained on one variation of the environment, and its learned state representations are regularized during training minimize this constant. We conduct experiments that demonstrate that our technique leads more efficient and robust learning than standard domain randomization, while achieving equal generalization scores.
INTRODUCTION
Deep Reinforcement Learning (RL) has proven very successful on complex high-dimensional prob- lems ranging from games like Go (Silver al. 2017) and Atari games (Mnih et al.] 2015) robot control tasks (Levine et al. 2016). However, one prominent issue is that of overtitting, illustrated in figure 1: agents trained on one domain fail generalize to other domains that differ only in small ways from the original domain (Sutton. 1996:
Cobbe et al. 2018 Zhang et al.. 2018b Packer et al.) 2018 Zhang et al., 2018 Witty et al., 2018 Farebrother al. 2018). Good generalization es- sential for problems such as robotics and autonomous vehicles, where the agent is often trained in a simulator and is then deployed in the real world where novel conditions will certainly be encoun- tered. Transfer from such simulated training environments to the real world is known crossing the reality gap in robotics, and is well known to be difficult, thus providing an important motivation for studying generalization.
We focus on the problem of generalizing between environments that visually differ from each other, for example in color or texture, but where the underlying dynamics the same. In reinforcement learning, prior work to address this topic has studied both domain adaptation and domain random- ization. Domain adaptation techniques aim update the data distribution in simulation to match the real distribution through some form of canonical mapping or using regularization methods (James) al. 2018 [oussmlii et al.) 2017 Gamrian & Goldberg 2018). Alternatively, domain randomiza- tion, in which the visual and physical properties of the training domains are randomized at the start of each episode during training, has also been shown lead to improved generalization and trans- fer to the real world with little or no real world data (Tobin et al.) 2017 (Sadeghi & Levine 2016 Antonova et al. 2017 7 Peng et al., 2017 Mordatch et al.] 2015) Rajeswaran et al. 2016]] OpenAl, 2018). However, domain randomization has been empirically shown to often lead to suboptimal policies with high variance in performance over different randomizations (Mehta et al. 2019). This issue can cause the learned policy to underperform in any given target domain.
We propose a regularization method for learning policies that are robust to irrelevant visual changes in the environment. Our work combines aspects from both domain adaptation and domain random- ization, in that we maintain the notion randomized environments but use a regularization method to achieve good generalization the randomization space. Our contributions the following:

---

Under review as a conference paper at ICLR 2020
We formalize the visual domain randomization problem, and show that the Lipschitz stant the agent's policy over visual variations provides an upper bound on the agent's robustness to these variations.
We propose an algorithm whereby the agent is only trained on one variation of the en- vironment but its learned representations are regularized so that the Lipschitz constant is minimized.
We experimentally show that our method is more efficient and leads to lower-variance policies than standard domain randomization, while achieving equal better returns and generalization ability.
This paper is structured follows. We first review related work, formalize the visual generalization problem, and present our theory contributions. We then describe our regularization method, and illustrate its application domain randomization and other regularization techniques in complex visual environments.
toy gridworld problem. Finally, compare method with standard
Figure 1:
Illustration of the visual generalization challenge in reinforcement learning. this cart- pole domain, the agent must learn to keep the pole upright. However, changes in the background color can completely throw off trained agent.
RELATED WORK
2.1 GENERALIZATION DEEP REINFORCEMEN LEARNING
Generalization to novel samples is well studied in supervised learning, where evaluating generaliza- tion through train/test splits is ubiquitous. However, evaluating for generalization novel conditions through such train/test splits is not common practice in Deep RL. Zhang et al.) (2018b) show that Deep RI. algorithms are shown to suffer from overfitting to training configurations and memorize training scenarios in discrete maze tasks. Packer et al. (2018) study performance under train-test domain shift by modifying environmental parameters such as robot mass and length to generate new domains. Farebrother al. (2018) propose using different game modes of Atari games measure generalization. They turn to supervised learning for inspiration, finding that both L2 regularization and dropout can help agents learn more generalizable features. These works all show that standard Deep RL algorithms tend to overfit to the environment used during training, hence the urgent need for designing agents that can generalize better.
2.2 DOMAIN RANDOMIZATION
We distinguish between two types of domain randomization: visual randomization, in which the variability between domains should not affect the agent's policy, and dynamics randomization, in which the agent should learn to adjust its behavior to achieve its goal. Visual domain randomization, which we focus on this work, has been successfully used directly transfer RL agents from sim- ulation to the real world without requiring any real images (Tobin et al. 2017 Sadeghi & Levine 2016 Kang al. 2019). These approaches used low fidelity rendering and randomized scene prop- erties such as lighting. textures, camera position, and colors, which led improved generalization.
Other work has also combined domain randomization and domain adaptation techniques (James) et al. 2018 Chebotar al. 2018 Gamrian & Goldberg 2018). These approaches both randomize the simulated environment and penalize the gap between the trajectories in the simulations and the real world, either by adding simulation and the real world. However, these methods require a large number of samples of real world trajectories, which be expensive collect.
to the loss. learning mapping between the states of the
2

---

Under review as a conference paper at ICLR 2020
Prior work has, however, noted the inefficiency of domain randomization. Mehta al. (2019) show that domain randomization may lead to suboptimal policies that vary a lot between domains, and propose to train the most informative environment variations within the given randomization ranges. Zakharov et al. (2019) also guide the domain randomization procedure by training a Decep- tionNet, that learns which randomizations actually useful to bridge the domain gap for image classification tasks.
2.3 LEARNING MRRAIIN-VVAAIANT FEATURES AND DOMAIN ADAPTATION
Learning domain-invariiat features has emerged as promising approach for taking advantage of the commonalities between domains. For instance, in the semi-supervised context, Bachman et al.) (0144); (Sajjadi al. (2016); Coors al. (2018); MMyato et al. (2018); ;); Xie et al. (2019) enforce that predictions of their networks be similar for original and augmented data points, with the objective of reducing the required amount of labelled data for training. Our work extends such methods to reinforcement learning.
In the reinforcement learning context, several other papers have also explored this topic. Tzeng al. (015) and Gupta et al. (2017) add constraints to encourage networks to learn similar embeddings samples from both a simulated and a target domain. Daftry et al. (006) apply similar approach to transfer policies for controlling aerial vehicles to different environments. Boousmalis et al. (2017) compare different domain adaptation methods in a robot grasping task, and show that they improve generalization. Wulfmeier al. (2017) use adversarial loss to train RL agents in such way that similar policies are learned in both simulated domain and the target domain. While promising, these methods are designed for cases when simulated and target domains both known, and cannot straightforwardly be applied when the target domain is only known to be within a distribution of domains.
Concurrently and independently of our work, Aractingi et al. (2019) also propose a regularization scheme to learn policies that are invariant to randomized visual changes in the environment without any real world data. Our work differs from theirs in that we propose a theoretical justification for this regularization and an analysis of the effects of this regularization on the learned representations. Crucially, whereas Aractingi et al. (2019) propose regularizing the network outputs, we regularize intermediate layers instead. In the appendix, we experimentally compare their regularization to ours and show that regularizing the network outputs leads to an undesirable trade-off between agent performance and generalization.
PROBLEM FORMULATION
We consider Markov decision processes (MDP) defined by S,A,RR,,, where S is the state space, the action space, R x A -> the reward function, T A Pr(S) the transition dynamics, and the discount factor. In reinforcement learning, an agent's objective is to find policy ı that maps states to distributions over actions such that the cumulative discounted reward yielded by its interactions with the environment is maximized.
3.1 VISUAL DOMAIN RANDOMIZATION
We consider a framework in which we are given a set of N parameters that can be changed to visually modify the environment, defined within randomization space E C N These parameters can for example control textures, colors, or lighting. Denoting J J {) the cumulative returns of a policy æ, the goal is to solve the optimization problem defined by J(m") max, [x(
Standard domain randomization, in which parameters & are randomly sampled at the start of each training episode, empirically produces policies with strongly varying performance over different regions of the randomization space, as demonstrated by Mehta et al. (2019). This high variance cause the learned policy underperform in any given target domain. To yield insight into the robustness of policies learned by domain randomization, we start by formalizing the notion of visually randomized MDP.
Definition Let M = (S, R,T,T be an MDP. A randomizer function of M is a mapping à S + S' where S' is new of states. The randomized MDP M4 (Sp, As: Rø, Tø, To)
3

---

Under review as a conference paper at ICLR 2020
defined as, for 8 E S, a
7o Given a policy т on MDP and randomization MR, we also define the agent's policy on Mo as T (-|s) = (-1((s)).
S = =((), =
= A, =) = T() R = R(s, a),
Despite all randomized MDPs sharing the same underlying rewards and transitions, the agent's s policy can vary between domains. For example, in policy-based algorithms (Williams 1992), if there are several optimal policies then the agent may adopt different policies for different Φ. Furthermore, for malue-based algorithms such as DQN (Mnih al. 2015), two scenarios can lead to there being different policies for different Φ. First, the (unique) optimal Q-function may correspond to several possible policies. Second, imperfect function approximation can lead to different value estimates for different randomizations and thus to different policies. To compare the ways in which policies can differ between randomized domains, introduce the notion of Lipschitz continuity of a policy over set of randomizations.
We assume the state space is equipped with a distance metric. A policy a is Lipschitz continuous over set of randomizations {6} for all randomizations Φn and Φ2 in {Ø),
Definition 2
(+)( 101(+) - 02(s))
K sup sup (L.) sES
is finite. Here, (PV()))Q) is the total variation distance between distributions (given by EneA - when the action space is discrete).
The following inequality shows that this Lipschitz constant is crucial in quantifying the robustness of RL agents affected by different randomization parameters. Informally, if policy is Lipschitz continuous over randomized MDPs, then small changes in the background color in environment will have small impact on the policy.
randomization space. The smaller the Lipschitz constant, the less policy is
Proposition 1 We consider an MDP M and set of randomizations {Ø} of this MDP. Let be K K-Lipschitz policy over {Ø). Suppose the rewards are bounded by Tmax such that Va A, S, (r(s, a)l I Tmax- Then for all Φ1 and oz in {Ø), the following inequalities hold:
2mmax Kr |1bb - 22
Im - -1/- S 2T max Σn min(1, (1 1)Kall - +221m)) S (1 7)-
(1)
Where ni is the expected cumulative return of policy The on MDP Ms for i {1,2}, and (0
#2 blloo sup.es (s) (2(s)).
Proof. See appendix.
These inequalities shows that the smaller the Lipschitz constant, the smaller the maximum variations of the policy over the randomization space can be. In the following, we present a regularization tech- nique that produces low- wwwvaaiance policies over the randomization space by minimizing the I. Lipschitz constant of the policy.
PROPOSED REGULARIZATION
We propose simple regularization method to produce an agent with policies that vary little over randomized environments, despite being trained on only one environment. We start by choosing one variation environment on which to train an agent with policy マ parameterized by and during training we minimize the loss
(()) = (R) + (If(ss) - fe(o
(2)
where À is a regularization parameter, LRF is the loss corresponding to the chosen reinforcement learning algorithm, the first expectation is taken over the distribution of states visited by the currett
4

---

Under review as a conference paper at ICLR 2020
policy which assume to be fixed when optimizing this loss, and fe is feature extractor used by the agent's policy. In our experiments, we choose the output the last hidden layer of the value or policy network as feature extractor. Minimizing the second term in this loss function minimizes the Lipschitz constant as defined above over the states visited by the agent, and causes the agent to learn representations of states that ignore variations caused by the randomization.
Our method can be applied to many RL algorithms, since it involves simply adding an additional term the learning loss. In the following, we experimentally demonstrate applications both value-based and policy-based reinforcement learning algorithms. Implementation details can be found in the appendix, and the code will be made available online.
EXPERIMENTS
5.1 ILLUSTRATION ON A GRIDWORLD
We first conduct experiments on simple gridworld to illustrate the theory described above.
Experimental Ina nxl
Generalization bound
Path
10"
10
Agent
Same path probability 86% 100% %
10a
Fire -1
4
Randomized Regularized (ours)
10
Agent
Path 1
-
10-1
10-2
10-1
10P
10a
Bound
I01
n2l
Figure 2: Left: a simple gridworld, in which the agent must make its way the goal while avoiding the fire. Center: empirical differences between regularized agents' policies on two randomizations of the gridworld compared corresponds to one agent, and 20 training seeds per value of À are shown here. Right: probability that different agents choose the same path for two randomizations of this domain. Our regularization method leads to more consistent behavior.
our theoretical bound in equation [ (the dashed line). Each point
The environment we use is the x gridworld shown in figure 2 which two optimal policies exist. The agent starts in the bottom left of the grid and must reach the goal while avoiding the fire. The agent can move either up or right, and in addition to the rewards shown in figure receives -1 reward for invalid actions that would case it to leave the grid. We set a time limit of TO steps and 1. We introduce randomization into this environment by describing the state observed by the agent tuple (I. y. y. {). where (x, y) is the agent's position and is randomization parameter with no impact on the underlying MDP. For this toy problem, we consider only two possible values for &: +5 and -5. The agents consider the REINFORCE algorithm (Sutton et al. 2000) with a baseline (see appendix), and multi-layer perceptron as the policy network.
First, we observe that even simple environment such this one, randomized agent regularly learns different paths for different randomizations (figure (2). An agent trained only on E and regularized with our technique, however, consistently learns the same path regardless of &. Although both agents easily solve the problem, the variance of the randomized agent's 's policy can be problematic in more complex environments in which identifying similarities between domains and ignoring irrelevant differences is important.
Next, we compare the measured difference between the policies learned by regularized agents on the two domains to the smallest of our theoretical bounds in equation [ which in this simple environ- ment can directly calculated. For a given value of we train regularized agent on the reference domain. We then measure the difference in returns obtained by this agent on the reference and on the regularized domain, and this return determines the agent's 's position along the x axis. We then numerically calculate the Lipschitz constant from the agent's action distribution over all states, and use this constant to calculate the bound proposition 1. This bound determines the agent's position along the y axis. Our results for different random seeds and values of À shown in figure 2 We
5

---

Under review as a conference paper at ICLR 2020
observe that increasing does lead to decreases in both the empirical difference in returns and in the theoretical bound.
5.2 VISUAL CARTPOLE WITH DQN
We compare standard visual domain randomization to our regularization method on more chal- lenging visual environment, in terms of 1) training stability, 2) returns and variance of the learned policies, and 3) state representations learned by the agents.
5.2.1 EXPERIMENTAL SETTING
To run domain randomization experiments, we use visual Cartpole environment shown in figure where the states consist raw pixels the images. The agent must keep a pole upright as long as possible on cart that can move left or right. The episode terminates either after 200 time steps, the cart leaves the track, or if the pole falls over. The randomization consists of changing the color of the background. Each randomized domain & E E corresponds to color (r, g. 6). where this environment is based on the OpenAl Gym (Brockman the DQN algorithm with CNN architecture similar to that used by Mnih et al. 2015). In principle, such a value-based algorithm should learn a unique value function independently of the randomization parameters consider. However, we will show function approximation errors cause different value functions to be learned for different background colors.
0 < r. b < 1. Our implementation
et al. 2016). For training.
We compare the performance of three agents. The Normal agent trained on only one domain (with a white background). The Randomized agent is trained on a chosen randomization space E. The Regularized agent is trained a white background using our regularization method with respect to randomization space E. The training of all three agents is done using the same hyperparameters, and over the same number of steps.
5.2.2
PERFORMANCE DURING TRAINING
Training curves, Small Domain
Training curves, Big Domain
175
150
150-
125
125
g
200
75
Nonnal Reqularized Randomized
Roqularized Randamized
50
Figure 3: Training curves over randomization spaces Esmall (left) and Emig (right). Shaded areas indicate the 95% confidence interval the mean, obtained over 10 training seeds.
We first compare the performance of our agents during training. We train all three agents over two randomization spaces (environments with different background colors), having the following sizes:
small = ((r, g, b), 0.5 S r.g, b.) j0.5, 1|s: of the unit cube.
Ebg = [0, 1) [0.5, 1] [0, 1] half the unit cube.
We find that the normal and regularized agents have similar training curves and are not affected by the size of the randomization space. However, the randomized agent learns more slowly on the small randomization space Esmall (left), and also achieves worse performance on the bigger randomization space Ebig (right). In high-dimensional problems, we would like to pick the randomization space E to be as large as possible to increase the chances of transferring the target domain. We find that standard domain randomization scales poorly with the size of the randomization space E, whereas our regularization method is more robust to larger randomization space.
We obtain the training curves shown in figure 3

---

Under review as a conference paper at ICLR 2020
Scores Reqularized
Scores Randomized
Scores Normal
1.0
0.8
150
0.6
Blue
100
0.2
0.0.0
0.5
1.0 0.0
0.5 Red
1.0 0.0
1.0
Red
Red
Figure 4: Comparison the average scores of different agents over different domains. The scores are calculated over a plane of the (r.g.b) cube in Ebig. where = 1 is fixed, averaged over 1000 steps. The training domain for both the regularized and normal agents located at the top right. The regularized agent learns more stable policies than the randomized agent over these domains.
5.2.3 GENERALIZATION AND VARIANCE
We compare the returns of the policies learned by the agents in different domains within the ran- domization space. We select plane within Ebig obtained by varying only the R and B channels but keeping G fixed. We plot the scores obtained on this plane in figure 4 We see that despite having only been trained on one domain, the regularized agent achieves consistently high scores on the other domains. On the other hand, the randomized agent's s policy exhibits returns with high variance between domains, which indicates that different policies were learned for different domains.
5.2.4 REPRESENTATION LEARNED BY THE AGENTS
Regularized
Randomized
15
Agent
Standard Deviations
0
-5
Normal
10.1
-10
Randomized
6.2
Regularized (ours) 3.7
-15
-15
-10 -5
-15 10
10
Figure 5: Left: Visualization of the representations learned by the agents for pink and green back- ground colors and for the same set of states. We observe that the randomized agent learns different representations for the two domains. Right: Standard deviation of estimated value functions over randomized domains, averaged over 10 training seeds.
To understand what causes this difference in behavior between the two agents, we study the represen- tations learned by the agents by analyzing the activations of the final hidden layer. We consider the agents trained on Ebig. and a sample of states obtained by performing a greedy rollout on a white background (which is included in Eeig). For each of these states, we calculate the representation corresponding to that state for another background color in Ebig- We then visualize these represen- domain. A representative example of such a plot is shown in figure 5 We see that the regularized agent learns similar representation for both backgrounds, whereas the randomized agent clearly separates them. This result indicates that the regularized agent learns to ignore the background color, whereas the randomized agent is likely to learn different policy for different background color. Further experiments comparing the representations of both agents can be found in the appendix.
tations using t-SNE plots, where each color corresponds
To further study the effect of our regularization method on the representations learned by the agents, we compare the variations in the estimated value function for both agents over Esig Figure shows the standard deviation of the estimated value function over different background colors, averaged over 10 training seeds and sample of states obtained by the same procedure described above. We observe that our regularization technique successfully reduces the variance of the value function over the randomization domain.
7

---

Under review as a conference paper at ICLR 2020
5.3
CAR RACING WITH PPO
Treining curves
902
Nermal
Regulariced
200
0025
CO2S
03
200
400
100
8000
1000
1200
3900
tpisade
Figure 6: Left: frames from the reference and randomized CarRacing environment. Right: training curves of our agents, averaged over seeds. Shaded areas indicate the 95% confidence interval of the mean.
To demonstrate the applicability of our regularization method to other domains and algorithms, we also perform experiments with the PPO algorithm (Schulman et al. 2017) on the CarRacing environment (Brockman et al.) (016), in which an agent must drive a car around racetrack. An example state from this environment and randomized version in which part of the background changes color are shown in figure 6 We start by training 3 agents this domain: normal agent the original background, a randomized agent, and a regularized agent with À = 50. Randomization in this experiment occurs over the entire RGB cube, which is larger than for the cartpole experiments. Training curves are shown in figure 6 policy this large randomization space, whereas the other agents successfully learn.
We see that the randomized agent fails to learn successful
Agent
Return (original)
Return (all colors)
Normal
554 + 68
60 53
Randomized
-17 + 33
-35 9
Regularized (ours) 10
622 + 81
324 = 51
Regularized (ours) 50
640 + 40
553 = 80
Dropout 0.1
696 107
154 86
Weight decay 10-4
692 + 55
61 + 15
EPO
- 14 28
-7
Table 1: Average returns on the original environment and its randomizations over all colors, with 95% confidence intervals calculated from 5 training seeds.
We also compare the generalization ability of these agents to other agents trained with different randomization and regularization methods. On the reference domain, we train regularized agent with smaller value of À 10, and two agents respectively with dropout 0.1 and 12 weight decay of 10-4, as in Cobbe al. (018)) and Aractingi al. (2019). On the randomized domain, we train an agent with the EPOpt- algorithm Rajeswaran et al. (2016), where in implementation the agent only trains on the randomized domains on which its score is worse than average. Scores on both the reference domain and its randomizations regularization leads to agents that are both successful in training and successfully generalize to a wide range of backgrounds. Moreover, larger value of À yields higher generalization scores. Of the other regularization schemes that we tested, we find that although they do improve learning on the randomization
shown in [ These results confirm that
the reference domain, only dropout leads to improvement in generalization space compared to our baseline.
CONCLUSION
In this paper studied generalization visually diverse environments in deep reinforcement learn- ing. We formalized the problem, illustrated the inefficiencies of standard domain randomization, and proposed theoretically grounded method that leads to robust, low-variance policies that generalize well. We conducted several experiments different environments of differing complexities using both on-policy and off-policy algorithms to support claims.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Anthony and Peter Bartlett. Neural network learning: Theoretical foemratical cambridge university press, 2009.
Rika Antonova, Silvia Cruciani, Christian Smith, and Danica Kragic. Reinforcement learning for pivoting task. CoRR, (ass/170330472, 2017. BRL.tttp://arx.. wwtp/oo//////ooo.oo... 00472
Michel Aractingi, Christopher Dance, Julien Perez, and Tomi Silander. Improving the generalization of visual navigation policies using invariance regularization. ICML 20/9 Workshop RLARealLife, 2019.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pscudo-ensembles. In Advances in Neural Information Processing Systems, pp. 3365-3373, 2014.
Peter L Bartlett. For valid generalization the size of the weights is more important than the size of the network. In Advances in neural information processing systems, pp. 134-140, 1997.
Peter Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalizeed margin bounds for neural networks. CoRR, nos.1700608498 2017. URL htt.:/::/aaax...oo/obbbb////////
08498
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakr- ishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, and Vincent Van- houcke. Using simulation and domain adaptation to improve efficiencc of deep robotic grasping. CoRR, abs/1709.07857, 2017. http:::/aaa....og///bb//////////
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan D. Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. CoRR, abs/1810.05687, 2018. /BRL/ttpe//araav.orr///a////
05687
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning, arXiv preprint arXiv: 18/2.0341 2018.
Benjamin Coors, Alexandru Condurache, Alfred Mertins, and Andreas Geiger. Learning transfor- mation invariant representations with weak supervision. In VISIGRAPP (5: VISAPP), pp. 64-72, 2018.
Shreyansh Daftry, J. Andrew Bagnell, and Martial Hebert. Learning transferable policies for monoc- ular reactive MAV control. CoRR, abs/1608.0062. 2016. URL httt::://rxxv..rr///aa
1668800627
Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and regularization in DQN. CoRR, abs/1810.00133 2018. Utt.bt:p://axxxv..rrgg/////
htw.rrg/ab////11 00123
Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to-image translation. CoRR, abs/1806.07377, 2018. URL wtR.:tttpp//axxi..rrr/////
18C6.07377
Abhishek Gupta, Coline Devin. Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. CoRR, abs/l703.0094// 2017. URL
http://an awwiv org/ abs / 703. 02949 949
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to- sim: Data-efficient robotic grasping via randomized-to-camoniee adaptation networks. CoRR, abs/1812.07252, 2018. httl/htpp://axxllorrg//////
Katie Kang, Suneel Belkhale, Gregory Kahn, Pieter Abbeel, and Sergey Levine. Generalization through simulation: Integrating simulated and real data into deep reinforcement learning for vision-based autonomous flight. CoRR, abs/1900.03701, 2019. URL attt:t/p//// //tp:///aavvv..////e/ee.oaia
abs 1902. 03701

---

Under review as a conference paper at ICLR 2020
Michat Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojeiech Jaškowski. Viz- doom: A doom-based research platform for visual reinforcement learning. In 2016 IEEE Conference Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17 133--1333, 2016.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J. Pal, and Liam Paull. Active do- main randomization. CoRR, abs/1904.04762, 2019. URL URL.tttpp://ax...o//////aa
04762
Takeru Miyato, Shin-ichi Macda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-199) 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Igor Mordatch, Kendall Lowrey, and Emanuel Todorov. Ensemble-CHS CIO: Full-body dynamic mo- tion planning that transfers to physical humanoids. 2015 TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5307-5314, 2015.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015.
Adam M. Oberman and Jeff Calder. Lipschitz regularized deep neural networks converge and gen- eralize. CoRR, abs/ 805.00540, 2018. /tt/:ttt:://rr///.rrr wwtp://////.///....000000... 09540
OpenAl. Learning dexterous in-hand manipulation. CoRR, abs/1808.00177, 2018. URL http:
00177
Charles Packer, Katelyn Gao, Jernj Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song.
Assessing generalization in deep reinforcement learning. CoRR, 0bss/8800.220 2018. URL
http::/arxiv. wttp://v.rrgbbbb/bboaoooo http:://arxxi..orrrab/b/// 111111110111122222
Xue Bin Peng. Marcin Andrychowicz, Wojciceh Zaremba, and Picter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. CoRR, abs/1710006523 2017. URL http:
//
/1710. 06537
Aravind Rajeswaran, Sarvjeet Ghotra, Sergey Levine, and Balaraman Ravindran. Epopt: Learning robust neural network policies using model ensembles. CoRR, abs/1610.01283, 2016. URL
http::/ /ttp:///aaxvv../ wwtp:b////aabbi/oo
01283
Fereshteh Sadeghi and Sergey Levine. (CAD) Real single-image flight without a single real image. CoRR, abs/1611 abs/1611.04201, 2016. URL httt::///ttxiiv
http:///aaiw..or//aa//////// wttg/oo/////..../oooo 04201
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems, pp. 1163-1177 1171, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (xXiv:1707006347 2017.
David Silver, Julian Schritwieserr, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354, 2017.
Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in neural information processing systems, pp. 1038-1044, 1996.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- fion processing systems, pp. 1057-1063, 2000.
10

---

Under review as a conference paper at ICLR 2020
Joshua Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. CoRR, abs/1703.06900 2017.
06907
Eric Tzeng, Coline Devin, Judy Hoffman, Chelsea Finn, Xingchao Peng, Sergey Levine, Kate Saenko, and Trevor Darrell. Towards adapting deep visuomotor representations from simulated to real environments. CoRR, abs/1511.077 2015. UtLptpp.//aaxiv..or///////11111
07111
Ronald J Williams. Simple statistical frolwint algorithms for connectionist reinforcementt learning. Machine learning. 8(3-4):229-255) 1992.
Sam Witty, Jun Ki Lee, Emma Tosch, Akanksha Atrey, Michael L. Littman, and David Jensen. Mea- suring and characterizing gemeralizaaion in deep reinforcement learning. CoRR, abs/1812.02868, 2018. Bttttttp///xxxttt wttp://w.gggbbbb//aa//boaaa 1812. C2868
Markus Wulfmeeer, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. CoRR, abs/1707.07907/ 2017. URL [ttt:://aaxxi htwp://wwwrrr.r.a..aerraaaaoo ab///bbs/11// 079977
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc Le. Unsupervised data augmentation. arXiv preprint arXiv:1904.12845 2019.
Sergey Zakharov, Wadim Kehl, and Slobodan Ilie. Deceptionnet: Network-driven domain random- ization. CoRR,
2019. URL
wttp://///abbb//////// 02750
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. CoRR, tbss/800.00937, 2018a. URL lattp://axii..
org/abs/1805, 07937
Chiyuan Zhang, Oriol Vinyals, Rémi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. CoRR, 800.18934,,,9,, 2018b. URL http: http///raxxv..ooaba....a. 1804. 06893

---

Under review as a conference paper at ICLR 2020
A PROOF OF PROPOSITION 1
The proof presented in the following applies to MDPs with a discrete action space. However, it can straightforwardly be generalized to continuous action spaces by replacing sums over actions with integrals over actions.
The proof uses the following lemma:
Lemma For two distributions p(a, y) = p(z) p(z)p(ylz) and ((, v) = (xx)q(y(2)) we can bound the total variation distance of the joint distribution
Drv(p(:)))ee & Drv (rv()())l(()))
max DrV (rr(p(1aa))laaa)))
Proof of the Lemma.
We have that
1 Σ () - <(x,,))
Drv(p(:-))la(-))
2 (o(x)p(ulx) (acplll)))(())((laal (x)))())))
Σ
2 Σ (((x)(ula)) -(())())))
+ (p(x) (x))))(11=1
1 Σ p(=))p()l)) - [(yl=)
$ 2
F.y
+ lp(x) (xx))a()1z)
I max Drv (p)
Dry (p(-)ll9())
Proof of the proposition.
Let p'o, (s, a) be the probability of being in state (,(s) at time t, and executing action for
i == 1,2. Since both MDPs have the same reward function, we have by definition that
E.a E. 2'p,, (s, a),()))ss, a). so we can write:
Im 721 S Σ
Σ
r I'v'' (s, (s)))
ΣΣ > Vl''s (s) pbs (s,aa)
< T Tiax
2r max
Σ 2" Drv (p's (1 (.:))
(3)
But p/, (s, (5) (als) and plan (s,a) p/ (s) d2 (als), Thus (Lemma 1):
Drv (.)) (3))) s Drv (P',, (.) (1pp), (-)) + max Drv (1 (-1s)]1
Drv (p', (. (1%))
k2ll01 Ø2llo0
12

---

Under review as a conference paper at ICLR 2020
We still have to bound Drv (P/, ()) (-)). For S we have that
1n (+) p/z2 (+) - Σ Pos (8t = 8/5 "/o (s')
Pez (st =sl') (s')
Σ Pos (st s/s '' (s')
Pes (8t 1/ (s')
+ P8z s|s') els))p=T (s')
- Pea (8: 8/5 2
$ Σ 1-1 (5'))s))p, (o|5') - Poo (s|s')]
+ Poo (s') 2
Summing over s we have that
Σ B (1pos (5|s¹)
(s
+ Drv(p's (-)1lp' (-)
But by marginalizing over actions Pe, (s|s') En Thh (al s po, (s aa s'), and using the fact that
Pe. (s|a, s') = To, (sla, s') = (s|a, 5 = Pd. (s/a, 8') = p(s|a, s'), ) we have that
ppo, (sls') - Poz (s|s') = > p(sla, (IT (als')
(a|s'))
(a|s')
And using p(sla, 8') = 1 we have that
E Bernph [lppr (s|s') - Poz (8|s')]
2
E Bo
(')l x (als')) (1))
< max Drv ( ( (+1s)](=(2+115))
< Ku 61 02
Thus, by induction, and assuming DTV (v%, (lll,, ()) = =0:
DTV (r', ('Il'', ()) S 1K=llør d2l10c
Plugging this into inequality 4 we get We also note that the total variation distance takes values between 0 and 1, so we have
Drv (p', (. (..)) < (t + 1) - 222l
DTV (r', (, "lp'' (s))) S min(1, (1 + 1) - -22llx))
Plugging this into inequality Bleads to our first bound,
Im m2 S 2r max Σn gd min(1, (t + 1) 1| @21lo0)
13

---

Under review as a conference paper at ICLR 2020
Our second, looser bound now be achieved as follows,
Im 7/2 S mamax Er'( =1 02|100
W/ 72l 5 mux
(1 7)2 K 221100
B COMPARISON OF OUR WORK WITH ARACTINGI ET AL.. (019))
Training curves, Big Domain, Regularized Outputs
175
0.1
==
150
= 10
125
Score
100
100
75
50
25
20000
40000
60000
80000
100000
Step
Lambda 0.1
Lambda 1
Lambda 10
Lambda 100
1.0
0.8
150
g 0.6
0.4
100
0.2
0.8.0
0.5
1.00.0
0.5 Red
1.00.0
0.5
1.00.0
0.5
1.0
50
Red
Figure 7: Top: training curves of agents with different regularization strengths when following the scheme of [Aractingi al. (2019). Shaded errors correspond to 95% confidence intervals of the mean, calculated from 10 training seeds. Bottom: scores obtained by trained agents for different regularization strengths on plane within the RGB cube.
Concurrently and independently of our work, work, Aractingi et al. (2019) propose similar regularization scheme randomized visual domains, which they experimentally demonstrate with the PPO algo- rithm on the VizDoom environment Kempka et al.) (2016) with randomized textures. As opposed to the regularization scheme proposed in our work in which regularize the final hidden layer of the network, they propose regularizing the output of the policy network. Regularizing the last hidden layer as in our scheme more clearly separates representation learning and policy learning, since the final layer of the network is only affected by the RL loss.
We hypothesized that regularizing the output of the network directly could lead to the regularization loss and the RL loss competing against each other, such that tradeoff between policy performance and generalization would be necessary. To test this hypothesis, we performed experiments on the visual cartpole domain with output regularization with different values of regularization parameter X. Our results are shown in figure 7 We find that increasing the regularization strength adversely affects training, However, agents trained with higher values of do achieve more consistent results over the randomization space. This shows that there indeed a tradeoff between generalization and policy performance when regularizing the network output in Aractingi al. (2019). In our experiments, however, we have found that changing the value of À only affects generalization ability and not agent performance the reference domain.
14

---

Under review as a conference paper at ICLR 2020
C EXPERIMENTAL DETAILS
All code used for experiments will be made available online.
C.I STATE PREPROCESSING
For our implementation of the visual cartpole environment, each image consists of 84 x 84 pixels with RGB channels. To include momentum information in our state description, stack k frames, so the shape of the state that is sent to the agent is 84 x 84 x 9.
We note that because of this preprocessing, agents trained until convergence achieve average returns of about 175 instead of the maximum achievable score of 200. Since the raw pixels do not contain momentum information, we stack three frames as input to the network. When the environment is reset, two random actions are thus taken before the agent is allowed to make decision. For some initializations, this causes the agent to start in a situation it cannot recover from. Moreover, due to the low image resolution the agent may sometimes struggle to correctly identify momentum and thus may make mistakes.
In CarRacing, each state consists of 96 x 96 pixels with RGB channels. We introduce frame skipping as often done for Atari games (Mnih et al. (015)), with a skip parameter 5. This restricts the length of episode 200 action choices. We then stack 2 frames to include momentum information into the state description. The shape of the state that is sent to the agent is thus 96 x 96 6. We note that although this preprocessing makes training agents faster, it also causes trained agents to not attain the maximum achievable score on this environment.
C.2 VISUAL. CARTPOLE
C.2.1 EXTRAPOLATION
Generalization score
200
Normal
Regularized
150
Randomized
Score
100
50
1
2 3
8
Distance to the reference domain
Figure 8: Generalization scores, with 95% confidence intervals obtained over 10 training seeds. The normal agent is trained on white (,, 1). corresponding to a distance to train= 0. The rest of the domains correspond (z, x, x), for 0.9, 0.8.....0.
Given that regularized agents are stronger in interpolation over their training domain, is natural to wonder what the performance of these agents is in extrapolation to colors not within the range of colors sampled within training. For this purpose, we consider randomized and regularized agents trained on Ebigs and test them the set {Z,,,,r, < < 1}. None of these agents ever exposed to z < 0.5 during training.
Our results are shown in figure We find that although the regularized agent consistently outper- forms the randomized agent in interpolation, both agents fail to extrapolate well outside the train domain. Since only regularize with respect to the training space, there is indeed guarantee that our regularization method can produce an agent that extrapolates well. Since the objective of domain randomization often is achieve good transfer
priori unknown target domain, this
15

---

Under review as a conference paper at ICLR 2020
result suggests that it is important that the target domain lie within the randomization space, and that the randomization space be made as large as possible during training.
C.2.2 FURTHER STUDY OF THE REPRESENTATIONS LEARNED BY DIFFERENT AGENTS
We perform further experiments to demonstrate that the randomized agent learns different repre- sentations for different domains, whereas the regularized agent learns similar representations. We consider agents trained on Explit = [0, U.2]$ 1] the union of darker, and lighter backgrounds. We then rollout each agent on a single episode of the domain with white background and, for each state this episode, calculate the representations learned by the agent for other background colors. We visualize these representations using the t-SNE plot shown in figure 9 randomized agent clearly separates the two training domains, whereas the regularized agent learns similar representations for both domains.
We observe that the
Figure 9: t-SNE of the representations over Expta of the Regularized (Left) and Randomized (Right) agents. Each color corresponds to a domain. The randomized agent learns very different represen- tations for [0, 0.2]$ and [0.8, 1)3.
We are interested in how robust our agents are to unseen values & E Explit- To visualize this, we rollout both agents in domains having different background colors {3r, ()), S 1}, i.e ranging from black to white, and collect their features over an episode. We then plot the t-SNEs of these features for both agents in figure IO, where each color corresponds domain.
Figure 10: t-SNE of the features of the Regularized (Left) and Randomized (Right) agents. Each color corresponds domain.
We observe once again that the regularized agent has much lower variance over unseen domains, whereas the randomized agent learns different features for different domains. This shows that the regularized agent more robust to domain shifts than the randomized agent.
D FURTHER RELATED WORK: LIPSCHITZ CONTINUITY AND GENERALIZATION IN DEEP LEARNING
Lipschitz-sensitive bounds on the generalization abilities of neural networks have long history Bartlett (1997): |Anthony & Bartlett (2009;; Neyshabur et al. (0115)). Recently, Bartlett et al. 2017) proved generalization bound in terms of the norms of each layer, which is proportional to the Lipschitz constant of the network. Oberman & Calder (2018) studied generalization through a general empirical risk minimization procedure with Lipschitz regularizatio,, and provides general- ization bounds. Similarly, we show that the Lipschitz constant of the network with respect to the randomization parameters plays important role in achieving zero-shot transfer target domain.
16

---

Under review as a conference paper at ICLR 2020
E ALGORITHMS
Algorithm Deep Q-learning with our regularization method
Initialize replay memory to capacity N
Initialize action-valie function Q with random weights 0
Initialize the randomization space E, and a reference MDP Marer to train on. Initialize regularization parameter Define a feature extractor fo for episode 1, .M do
Sample a randomizer function grampled uniformly from E.
for = 1, do
With probability select a random action d;
otherwise select a, arg max, Q( 8) Execute action in Mares, observe reward the reference state, and the corresponding randomized state with the chosen visual settings: oref (s1+1), gsampled(s14)
Store transition (6 (st), (raampedd ae; rt, (.ee( @sammled ($t+1) in D
gref (amp) Sample random minibatch of transitions (øjef, .0, _ (aampled(old) ag j+11 pj+1
from D. where o sampleddoold) is the randomization that had been selected when the transi- tion had been observed.
ryfor terminal oTef
Set yj
+ y m Q66)+1 j+1+ a',
ref
otherwise.
Perform a gradient descent step on (y3 Q(6 100000000,00000 (N;;0))) end for end for
)-olo,, (ampelmmmll
17

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Policy Gradient with baseline using our regularization method
Initialize policy network function za with random weights 0, baseline by Initialize the randomization space E, and a reference MDP Mares to train on. Initialize regularization parameter Define a feature extractor fe for episode = 1, .M do
Collect a set of trajectories (80, "0,.1TT-., 0r-1; rr) by executing He Maret.
For each trajectory. sample a randomizer function @rempled uniformly from E. for t = 1, in each trajectory do
Compute the return Rg TT'1 ? 'T,
Estimate the advantage A, R₁ - (o()re((s))))
end for
Perform a gradient descent step on
T
Σ Ag log no(a no(arlore
+ R4 bo bo(or
+ * fo (drampled (s))) (s))
end for
Note that algorithm 2 can be straightforwardly adapted to several state of the art policy gradient algorithms such as PPO.
F A DYNAMICS RANDOMIZATION EXPERIMENT
Training curves
200
175
150
125
100
Normal Regularized Randomized
25
10000 20000 30000 40000
50000 60000
Step
Figure 11: Training curves of the agents on the Cartpole domain, averaged over 100 seeds.
We also perform an experiment to demonstrate that learning domain-invariant features can be helpful only for visual randomization, but also for some instances of dynamics randomization. We consider once again the Cartpole domain, where this time we randomize some physical dynamics of the environment. Specifically, we choose to randomize the pole length and the gravity g. The state for our reinforcement learning agent the 6-dimensional vector (ar, i. 0, 0, 1, where x the position of the cart, its velocity, the angle between the pole and the horizontal plane, and the angular velocity. We train all three agents (Normal, Regularized, Randomized) using only two randomized domains : = 0.5. g 9.8 and 1.g = 50. We use the DQN algorithm with Multi-Layer Perceptron architecture for this experiment.
18

---

Under review as a conference paper at ICLR 2020
We first compare the training speed of the agents. The training curves averaged over 100 seeds plotted figure I1 We observe once again that the randomized agent significantly slower than the regularized one, and is more unstable.
Randomized
Regularized
75
75
200
Gravity
50
50
150
100
25
25
0
0
2
Pole Length
Pole Length
Figure 12: Generalization scores averaged over 5 training seeds and 4 test episodes per seed. Red dots correspond to training environments
Next, we examine the agents' generalization ability. We test the agents on environments having values of pole length I and gravity unseen during training. We plot their scores in figure T2] The randomized agent clearly specializes on the two different training domains, corresponding to the two clearly distinguishable regions where high scores are achieved, whereas the regularized agents achieves more consistent scores across domain. This result can be understood as follows. Although the different dynamics between the two domains lead to there being different sets of optimal policies, our regularization method forces the agent to only learn policies that do not depend on the specific values of the randomized dynamics parameters. These policies are therefore more likely to also work when those dynamics are different.
F.1 REPRESENTATION LEARNED BY THE AGENT
We analyze the representations learned by each agent in our dynamics randomization experiment. Once the agents trained, we rollout their policies in both randomized environments with greedy strategy, where we use 0.2 to reach a larger number of states of the MDP, over 10000 steps. We collect the representations (the activations of the last hidden layer) corresponding the visited states. These features are 100-dimensional, so in order to visualize them. we use the t-SNE plots shown in figure [3] We emphasize that although this figure corresponds to a single training seed, the general aspect of these results is repeatable.
Randomized
Regularized
Env
Env
Env
Figure 13: t-sne of the representations learned by the regularized and randomized agents on the two training environments.
The randomized agent learns completely different representations for the two randomized environ- ments. This explains its high variance during the training, since it tries learn different strategy for each domain. On the other hand, our regularized agent has the same representation for both domains, which allows learn much faster, and learn policies that are robust to changes in the environment's 's dynamics.
19


Reviews:
[{"review_id": "S1gzAdmp_r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces the high variance policies challenge in domain randomization for reinforcement learning. The paper gives a new bound for the expected return of the policy when the policy is Lipschitz continuous. Then the paper proposes a new method to minimize the Lipschitz constant for policies of all randomization. Experiment results prove the efficacy of the proposed domain randomization method for various reinforcement learning approaches.\n\nThe paper mainly focuses on the problem of visual randomization, where the different randomized domains differ only in state space and the underlying rewards and dynamics are the same. The paper also assumes that there is a mapping from the states in one domain to another domain. Are there any constraints on the mapping? Will some randomization introduces a larger state space than others?\n\nThe paper demonstrates that the expected return of the policy is bounded by the largest difference in state space and the Lipschitz constant of the policies, which is a new perspective of domain randomization for reinforcement learning. \n\nThe proposed method minimizes the expected variations between states of two randomizations but the Lipschitz constant is by the largest difference of policy outputs of a state pair between domains. Should minimizing the maximum difference be more proper?\n\nThe center part of Figure 2 is confusing, could the authors clarify it?\n\nIn the Grid World environment, how does the random parameter influence the states?\n\nThe baselines are a little weak. The paper only compares the proposed with training reinforcement learning algorithm on randomized environments. Could the authors compare with other domain randomization methods in reinforcement learning or naively adapt domain randomization methods from other areas to reinforcement learning?\n\nOverall, the paper is well-written and the ideas are novel. However, some parts are not clearly clarified and the experiments are a little weak with too weak baselines. I will consider raising my score according to the rebuttal.\n\nPost-feedback:\nI have read the rebuttal. The authors have addressed some of my concerns but why minimizing the expected difference is not convincing. I think the paper should receive a borderline score between 3 and 6. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gGDk4uKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nTo improve the generalization ability of deep RL agents across the tasks with different visual patterns, this paper proposed a simple regularization technique for domain randomization. By regularizing the outputs from normal and randomized states, the trained agents are forced to learn invariant representations. The authors showed that the proposed method can be useful to improve the generalization ability using CartPole and Car Racing environments. \n\nDetailed comments:\n\nI'd like to recommend \"weak reject\" due to the following reasons:\n\n1. Lack of novelty: The main idea of this paper (i.e. regularizing the outputs from normal and randomized states) is not really new because it has been explored before [Aractingi' 19]. Even though this paper provides more justification and analysis for this part (Proposition 1 in the draft), the contributions are not enough as the ICLR publications.\n\n2. As shown in [Cobbe' 19], various regularization and data augmentation techniques have been studied for improving the generalization ability of deep RL agents. Therefore, the comparisons with such baselines are required to verify the effectiveness of the proposed methods.\n\n3. For domain randomization, it has been observed that finding a good distribution of simulation parameters is a key component [Ramos' 19, Mozifian' 19, Chebotar' 19], but the authors did not consider training the distribution of simulation parameters in the paper. \n\n[Ramos' 19] Ramos, F., Possas, R.C. and Fox, D., BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators. In RSS, 2019.\n\n[Cobbe' 19] Cobbe, K., Klimov, O., Hesse, C., Kim, T. and Schulman, J., Quantifying generalization in reinforcement learning. In ICML, 2019.\n\n[Mozifian' 19] Mozifian, M., Higuera, J.C.G., Meger, D. and Dudek, G., Learning Domain Randomization Distributions for Transfer of Locomotion Policies. arXiv preprint arXiv:1906.00410, 2019. \n\n[Chebotar' 19] Chebotar, Y., Handa, A., Makoviychuk, V., Macklin, M., Issac, J., Ratliff, N. and Fox, D., May. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 8973-8979). 2019\n\n[Aractingi' 19] Michel Aractingi, Christopher Dance,  Julien Perez, Tomi Silander,  Improving the Generalization of Visual Navigation Policies using Invariance Regularization, ICML workshop 2019.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bklrq4s0KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a regularization scheme for training vision-based control policies that are robust to variations of the visual input, which the paper classifies as \"visual randomization\". Visual randomization, as the paper notes, is one of the state-of-the-art techniques proposed from simulation to real robot transfer of vision-based policies trained on synthetic images. The regularization proposed in the paper aims to produce policies that rely on features that are invariant to the visual randomization, so that the resulting behaviour of the agent is consistent across different variations of its visual input for the same states. \n\nThe paper proposes that forcing a policy trained under randomization to have a Lipschitz constant of K, over the randomization parameters, causes the optimal policy to be similar across randomization parameters, with the difference in expected returns of two randomized environments being bounded by K times the maximum difference in their randomization parameters.\n\nI recommend this paper to not be accepted until the following issues are addressed. \n\n* There are missing details from the experimental setup, which makes the results hard to interpret (see, below). \n\n* There are  missing details on how vanilla domain randomization was implemented. Domain randomization aims to maximize the expected performance over the environment distribution. This can be implemented properly by computing the expected gradients with data from more than one environment. From the algorithm descriptions in the appendix, it is not clear that this is how vanilla domain randomization was implemented. \n\n* The title, introduction and conclusions do not reflect the scope of the paper. The paper only addresses situations where the same behaviour is achievable on all environments, an assumption (Mehta et al, 2019) also makes, and its proposed regularization is based on the assumption that the optimal behaviour is achievable with the same policy on all environments. But this is not true in general: for dynamics randomization, different environments may require different policies (e.g. driving a car on a road vs driving off-road). The regularization method may result in overly conservative policies i such situations.  \n\nQuestions about experimental details: \n\nWhat are the maximum returns  for Cartpole when trained until convergence without randomization? (175? 200? 1000?) If the maximum returns are higher than 175, how does Figure 4 look with more data? This is crucial to understand, for example, the results in Figure 11. That figure shows the proposed regularization slightly hinders the performance for the environments near l=1, g=50 (that region is a darker shade of green on the left subfigure). How do we know if the task has been successfully solved in the green vs purple regions? In all experiments, are the training curves showing the performance of the policies over the same environments (same seeds)? If not, how are the training curves comparable?\n\nOther things to improve:\n\nThe conclusions of this paper can be made stronger by adding a comparison with EpOpt-PPO (i.e. optimizing the worst case performance over a set of trajectories sampled from multiple environments)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a technique for learning RL agents to generalize well to unseen environments.

All reviewers and AC think that the paper has some potential but is a bit below the bar to be accepted due to the following facts:

(a) Limited experiments, i.e., consider more appealing baselines/scenarios and provide more experimental details.
(b) The proposed method/idea is simple/reasonable, but not super novel, i.e., not enough considering the ICLR high standard (potentially enough for a workshop paper).

Hence, I think this is a borderline paper toward rejection.


Decision: Reject