Title: ROBUST VISUAL DOMAIN RANDOMIZATION FOR RE-INFORCEMENT LEARNING

Abstract: Producing agents that can generalize to a wide range of visually different environments is a significant challenge in reinforcement learning. One method for overcoming this issue is visual domain randomization, whereby at the start of each training episode some visual aspects of the environment are randomized so that the agent is exposed to many possible variations. However, domain randomization is highly inefficient and may lead to policies with high variance across domains. Instead, we formalize the visual domain randomization problem, and show that minimizing the policy's Lipschitz constant with respect to the randomization parameters leads to low variance in the learned policies. We propose a regularization method where the agent is only trained on one variation of the environment, and its learned state representations are regularized during training to minimize this constant. We conduct experiments that demonstrate that our technique leads to more efficient and robust learning than standard domain randomization, while achieving equal generalization scores.

1. INTRODUCTION:

Deep Reinforcement Learning (RL) has proven very successful on complex high-dimensional problems ranging from games like Go (Silver et al., 2017) and Atari games (Mnih et al., 2015) to robot control tasks (Levine et al., 2016) . However, one prominent issue is that of overfitting, illustrated in figure 1 : agents trained on one domain fail to generalize to other domains that differ only in small ways from the original domain (Sutton, 1996; Cobbe et al., 2018; Zhang et al., 2018b; Packer et al., 2018; Zhang et al., 2018a; Witty et al., 2018; Farebrother et al., 2018) . Good generalization is essential for problems such as robotics and autonomous vehicles, where the agent is often trained in a simulator and is then deployed in the real world where novel conditions will certainly be encountered. Transfer from such simulated training environments to the real world is known as crossing the reality gap in robotics, and is well known to be difficult, thus providing an important motivation for studying generalization.

We focus on the problem of generalizing between environments that visually differ from each other, for example in color or texture, but where the underlying dynamics are the same. In reinforcement learning, prior work to address this topic has studied both domain adaptation and domain randomization. Domain adaptation techniques aim to update the data distribution in simulation to match the real distribution through some form of canonical mapping or using regularization methods (James et al., 2018; Bousmalis et al., 2017; Gamrian & Goldberg, 2018) . Alternatively, domain randomization, in which the visual and physical properties of the training domains are randomized at the start of each episode during training, has also been shown to lead to improved generalization and transfer to the real world with little or no real world data (Tobin et al., 2017; Sadeghi & Levine, 2016; Antonova et al., 2017; Peng et al., 2017; Mordatch et al., 2015; Rajeswaran et al., 2016; OpenAI, 2018) . However, domain randomization has been empirically shown to often lead to suboptimal policies with high variance in performance over different randomizations (Mehta et al., 2019) . This issue can cause the learned policy to underperform in any given target domain.

We propose a regularization method for learning policies that are robust to irrelevant visual changes in the environment. Our work combines aspects from both domain adaptation and domain randomization, in that we maintain the notion of randomized environments but use a regularization method to achieve good generalization over the randomization space. Our contributions are the following:

• We formalize the visual domain randomization problem, and show that the Lipschitz constant of the agent's policy over visual variations provides an upper bound on the agent's robustness to these variations. • We propose an algorithm whereby the agent is only trained on one variation of the environment but its learned representations are regularized so that the Lipschitz constant is minimized.

• We experimentally show that our method is more efficient and leads to lower-variance policies than standard domain randomization, while achieving equal or better returns and generalization ability.

This paper is structured as follows. We first review related work, formalize the visual generalization problem, and present our theory contributions. We then describe our regularization method, and illustrate its application to a toy gridworld problem. Finally, we compare our method with standard domain randomization and other regularization techniques in complex visual environments.

Figure 1 : Illustration of the visual generalization challenge in reinforcement learning. In this cartpole domain, the agent must learn to keep the pole upright. However, changes in the background color can completely throw off a trained agent.

2. RELATED WORK:



2.1. GENERALIZATION IN DEEP REINFORCEMENT LEARNING:

Generalization to novel samples is well studied in supervised learning, where evaluating generalization through train/test splits is ubiquitous. However, evaluating for generalization to novel conditions through such train/test splits is not common practice in Deep RL. Zhang et al. (2018b) show that Deep RL algorithms are shown to suffer from overfitting to training configurations and to memorize training scenarios in discrete maze tasks. Packer et al. (2018) study performance under train-test domain shift by modifying environmental parameters such as robot mass and length to generate new domains. Farebrother et al. (2018) propose using different game modes of Atari games to measure generalization. They turn to supervised learning for inspiration, finding that both L2 regularization and dropout can help agents learn more generalizable features. These works all show that standard Deep RL algorithms tend to overfit to the environment used during training, hence the urgent need for designing agents that can generalize better.

2.2. DOMAIN RANDOMIZATION:

We distinguish between two types of domain randomization: visual randomization, in which the variability between domains should not affect the agent's policy, and dynamics randomization, in which the agent should learn to adjust its behavior to achieve its goal. Visual domain randomization, which we focus on in this work, has been successfully used to directly transfer RL agents from simulation to the real world without requiring any real images (Tobin et al., 2017; Sadeghi & Levine, 2016; Kang et al., 2019) . These approaches used low fidelity rendering and randomized scene properties such as lighting, textures, camera position, and colors, which led to improved generalization.

Other work has also combined domain randomization and domain adaptation techniques (James et al., 2018; Chebotar et al., 2018; Gamrian & Goldberg, 2018) . These approaches both randomize the simulated environment and penalize the gap between the trajectories in the simulations and the real world, either by adding a term to the loss, or learning a mapping between the states of the simulation and the real world. However, these methods require a large number of samples of real world trajectories, which can be expensive to collect.

Prior work has, however, noted the inefficiency of domain randomization. Mehta et al. (2019) show that domain randomization may lead to suboptimal policies that vary a lot between domains, and propose to train on the most informative environment variations within the given randomization ranges. Zakharov et al. (2019) 2017) use an adversarial loss to train RL agents in such a way that similar policies are learned in both a simulated domain and the target domain. While promising, these methods are designed for cases when simulated and target domains are both known, and cannot straightforwardly be applied when the target domain is only known to be within a distribution of domains.

Concurrently and independently of our work, Aractingi et al. (2019) also propose a regularization scheme to learn policies that are invariant to randomized visual changes in the environment without any real world data. Our work differs from theirs in that we propose a theoretical justification for this regularization and an analysis of the effects of this regularization on the learned representations. Crucially, whereas Aractingi et al. (2019) propose regularizing the network outputs, we regularize intermediate layers instead. In the appendix, we experimentally compare their regularization to ours and show that regularizing the network outputs leads to an undesirable trade-off between agent performance and generalization.

3. PROBLEM FORMULATION:

We consider Markov decision processes (MDP) defined by (S, A, R, T, γ), where S is the state space, A the action space, R : S × A → R the reward function, T : S × A → P r(S) the transition dynamics, and γ the discount factor. In reinforcement learning, an agent's objective is to find a policy π that maps states to distributions over actions such that the cumulative discounted reward yielded by its interactions with the environment is maximized.

3.1. VISUAL DOMAIN RANDOMIZATION:

We consider a framework in which we are given a set of N parameters that can be changed to visually modify the environment, defined within a randomization space Ξ ⊂ R N . These parameters can for example control textures, colors, or lighting. Denoting J(π, ξ) the cumulative returns of a policy π, the goal is to solve the optimization problem defined by

J(π * ) = max π E ξ [J(π, ξ)].

Standard domain randomization, in which parameters ξ are randomly sampled at the start of each training episode, empirically produces policies with strongly varying performance over different regions of the randomization space, as demonstrated by Mehta et al. (2019) . This high variance can cause the learned policy to underperform in any given target domain. To yield insight into the robustness of policies learned by domain randomization, we start by formalizing the notion of a visually randomized MDP.

Definition 1 Let M = (S, A, R, T, γ) be an MDP. A randomizer function of M is a mapping φ : S → S where S is a new set of states. The randomized MDP M φ = (S φ , A φ , R φ , T φ , γ φ ) is defined as, for s, s ∈ S, a ∈ A :

S φ = φ(S), A φ = A, T φ (φ(s )|φ(s), a) = T (s |s, a), R φ (φ(s), a) = R(s, a), γ φ = γ

Given a policy π on MDP M and a randomization M φ , we also define the agent's policy on M φ as

π φ (•|s) = π(•|φ(s)).

Despite all randomized MDPs sharing the same underlying rewards and transitions, the agent's policy can vary between domains. For example, in policy-based algorithms (Williams, 1992) , if there are several optimal policies then the agent may adopt different policies for different φ. Furthermore, for value-based algorithms such as DQN (Mnih et al., 2015) , two scenarios can lead to there being different policies for different φ. First, the (unique) optimal Q-function may correspond to several possible policies. Second, imperfect function approximation can lead to different value estimates for different randomizations and thus to different policies. To compare the ways in which policies can differ between randomized domains, we introduce the notion of Lipschitz continuity of a policy over a set of randomizations.

Definition 2 We assume the state space is equipped with a distance metric. A policy π is Lipschitz continuous over a set of randomizations {φ} if for all randomizations φ 1 and φ 2 in {φ},

K π = sup φ1,φ2∈{φ} sup s∈S D T V (π(•|φ 1 (s)) π(•|φ 2 (s))) |φ 1 (s) -φ 2 (s)| is finite.

Here, D T V (P Q) is the total variation distance between distributions (given by 1 2

a∈A |P (a) -Q(a)| when the action space is discrete).

The following inequality shows that this Lipschitz constant is crucial in quantifying the robustness of RL agents over a randomization space. The smaller the Lipschitz constant, the less a policy is affected by different randomization parameters. Informally, if a policy is Lipschitz continuous over randomized MDPs, then small changes in the background color in an environment will have a small impact on the policy.

Proposition 1 We consider an MDP M and a set of randomizations {φ} of this MDP. Let π be a K-Lipschitz policy over {φ}. Suppose the rewards are bounded by r max such that ∀a ∈ A, s ∈ S, |r(s, a)| ≤ r max . Then for all φ 1 and φ 2 in {φ}, the following inequalities hold :

|η 1 -η 2 | ≤ 2r max ∞ t γ t min(1, (t + 1)K π φ 1 -φ 2 ∞ ) ≤ 2r max K π (1 -γ) 2 φ 1 -φ 2 ∞ (1)

Where η i is the expected cumulative return of policy π φi on MDP M φi , for i ∈ {1, 2}, and |φ 1 -

φ 2 ∞ = sup s∈S |φ 1 (s) -φ 2 (s)|.

Proof. See appendix.

These inequalities shows that the smaller the Lipschitz constant, the smaller the maximum variations of the policy over the randomization space can be. In the following, we present a regularization technique that produces low-variance policies over the randomization space by minimizing the Lipschitz constant of the policy.

4. PROPOSED REGULARIZATION:

We propose a simple regularization method to produce an agent with policies that vary little over randomized environments, despite being trained on only one environment. We start by choosing one variation of the environment on which to train an agent with a policy π parameterized by θ, and during training we minimize the loss

L(θ) = L RL (θ) + λ E s∼π E φ f θ (s) -f θ (φ(s)) 2 2 (2)

where λ is a regularization parameter, L RL is the loss corresponding to the chosen reinforcement learning algorithm, the first expectation is taken over the distribution of states visited by the current policy which we assume to be fixed when optimizing this loss, and f θ is a feature extractor used by the agent's policy. In our experiments, we choose the output of the last hidden layer of the value or policy network as our feature extractor. Minimizing the second term in this loss function minimizes the Lipschitz constant as defined above over the states visited by the agent, and causes the agent to learn representations of states that ignore variations caused by the randomization.

Our method can be applied to many RL algorithms, since it involves simply adding an additional term to the learning loss. In the following, we experimentally demonstrate applications to both value-based and policy-based reinforcement learning algorithms. Implementation details can be found in the appendix, and the code will be made available online.

5. EXPERIMENTS:



5.1. ILLUSTRATION ON A GRIDWORLD:

We first conduct experiments on a simple gridworld to illustrate the theory described above.

Agent Same path probability Randomized 86% Regularized 100% (ours) Figure 2 : Left: a simple gridworld, in which the agent must make its way to the goal while avoiding the fire. Center: empirical differences between regularized agents' policies on two randomizations of the gridworld compared to our theoretical bound in equation 1 (the dashed line). Each point corresponds to one agent, and 20 training seeds per value of λ are shown here. Right: probability that different agents choose the same path for two randomizations of this domain. Our regularization method leads to more consistent behavior.

The environment we use is the 3 × 3 gridworld shown in figure 2 , in which two optimal policies exist. The agent starts in the bottom left of the grid and must reach the goal while avoiding the fire. The agent can move either up or right, and in addition to the rewards shown in figure 2 receives -1 reward for invalid actions that would case it to leave the grid. We set a time limit of 10 steps and γ = 1. We introduce randomization into this environment by describing the state observed by the agent as a tuple (x, y, ξ), where (x, y) is the agent's position and ξ is a randomization parameter with no impact on the underlying MDP. For this toy problem, we consider only two possible values for ξ: +5 and -5. The agents we consider use the REINFORCE algorithm (Sutton et al., 2000) with a baseline (see appendix), and a multi-layer perceptron as the policy network.

First, we observe that even in a simple environment such as this one, a randomized agent regularly learns different paths for different randomizations (figure 2 ). An agent trained only on ξ = 5 and regularized with our technique, however, consistently learns the same path regardless of ξ. Although both agents easily solve the problem, the variance of the randomized agent's policy can be problematic in more complex environments in which identifying similarities between domains and ignoring irrelevant differences is important.

Next, we compare the measured difference between the policies learned by regularized agents on the two domains to the smallest of our theoretical bounds in equation 1, which in this simple environment can be directly calculated. For a given value of λ, we train a regularized agent on the reference domain. We then measure the difference in returns obtained by this agent on the reference and on the regularized domain, and this return determines the agent's position along the x axis. We then numerically calculate the Lipschitz constant from the agent's action distribution over all states, and use this constant to calculate the bound in proposition 1. This bound determines the agent's position along the y axis. Our results for different random seeds and values of λ are shown in figure 2 . We observe that increasing λ does lead to decreases in both the empirical difference in returns and in the theoretical bound.

5.2. VISUAL CARTPOLE WITH DQN:

We compare standard visual domain randomization to our regularization method on a more challenging visual environment, in terms of 1) training stability, 2) returns and variance of the learned policies, and 3) state representations learned by the agents.

5.2.1. EXPERIMENTAL SETTING:

To run domain randomization experiments, we use a visual Cartpole environment shown in figure 1 , where the states consist of raw pixels of the images. The agent must keep a pole upright as long as possible on a cart that can move left or right. The episode terminates either after 200 time steps, if the cart leaves the track, or if the pole falls over. The randomization consists of changing the color of the background. Each randomized domain ξ ∈ Ξ corresponds to a color (r, g, b), where 0 ≤ r, g, b ≤ 1. Our implementation of this environment is based on the OpenAI Gym (Brockman et al., 2016) . For training, we use the DQN algorithm with a CNN architecture similar to that used by Mnih et al. (2015) . In principle, such a value-based algorithm should learn a unique value function independently of the randomization parameters we consider. However, as we will show function approximation errors cause different value functions to be learned for different background colors.

We compare the performance of three agents. The Normal agent is trained on only one domain (with a white background). The Randomized agent is trained on a chosen randomization space Ξ. The Regularized agent is trained on a white background using our regularization method with respect to randomization space Ξ. The training of all three agents is done using the same hyperparameters, and over the same number of steps. We first compare the performance of our agents during training. We train all three agents over two randomization spaces (environments with different background colors), having the following sizes :

5.2.2. PERFORMANCE DURING TRAINING:

• Ξ small = {(r, g, b), 0.5 ≤ r, g, b ≤ 1.} = [0.5, 1] 3 : 1 8 of the unit cube. • Ξ big = [0, 1] × [0.5, 1] × [0, 1] : half the unit cube.

We obtain the training curves shown in figure 3 . We find that the normal and regularized agents have similar training curves and are not affected by the size of the randomization space. However, the randomized agent learns more slowly on the small randomization space Ξ small (left), and also achieves worse performance on the bigger randomization space Ξ big (right). In high-dimensional problems, we would like to pick the randomization space Ξ to be as large as possible to increase the chances of transferring to the target domain. We find that standard domain randomization scales poorly with the size of the randomization space Ξ, whereas our regularization method is more robust to a larger randomization space. The regularized agent learns more stable policies than the randomized agent over these domains.

5.2.3. GENERALIZATION AND VARIANCE:

We compare the returns of the policies learned by the agents in different domains within the randomization space. We select a plane within Ξ big obtained by varying only the R and B channels but keeping G fixed. We plot the scores obtained on this plane in figure 4 . We see that despite having only been trained on one domain, the regularized agent achieves consistently high scores on the other domains. On the other hand, the randomized agent's policy exhibits returns with high variance between domains, which indicates that different policies were learned for different domains. To understand what causes this difference in behavior between the two agents, we study the representations learned by the agents by analyzing the activations of the final hidden layer. We consider the agents trained on Ξ big , and a sample of states obtained by performing a greedy rollout on a white background (which is included in Ξ big ). For each of these states, we calculate the representation corresponding to that state for another background color in Ξ big . We then visualize these representations using t-SNE plots, where each color corresponds to a domain. A representative example of such a plot is shown in figure 5 . We see that the regularized agent learns a similar representation for both backgrounds, whereas the randomized agent clearly separates them. This result indicates that the regularized agent learns to ignore the background color, whereas the randomized agent is likely to learn a different policy for a different background color. Further experiments comparing the representations of both agents can be found in the appendix.

To further study the effect of our regularization method on the representations learned by the agents, we compare the variations in the estimated value function for both agents over Ξ big . Figure 5 shows the standard deviation of the estimated value function over different background colors, averaged over 10 training seeds and a sample of states obtained by the same procedure as described above.

We observe that our regularization technique successfully reduces the variance of the value function over the randomization domain. To demonstrate the applicability of our regularization method to other domains and algorithms, we also perform experiments with the PPO algorithm (Schulman et al., 2017) on the CarRacing environment (Brockman et al., 2016) , in which an agent must drive a car around a racetrack. An example state from this environment and a randomized version in which part of the background changes color are shown in figure 6 . We start by training 3 agents on this domain: a normal agent on the original background, a randomized agent, and a regularized agent with λ = 50. Randomization in this experiment occurs over the entire RGB cube, which is larger than for the cartpole experiments.

Training curves are shown in figure 6 . We see that the randomized agent fails to learn a successful policy on this large randomization space, whereas the other agents successfully learn. 1 : Average returns on the original environment and its randomizations over all colors, with 95% confidence intervals calculated from 5 training seeds.

We also compare the generalization ability of these agents to other agents trained with different randomization and regularization methods. On the reference domain, we train a regularized agent with a smaller value of λ = 10, and two agents respectively with dropout 0.1 and l2 weight decay of 10 -4 , as in Cobbe et al. (2018) and Aractingi et al. (2019) . On the randomized domain, we train an agent with the EPOpt-PPO algorithm Rajeswaran et al. (2016) , where in our implementation the agent only trains on the randomized domains on which its score is worse than average. Scores on both the reference domain and its randomizations are shown in 1. These results confirm that our regularization leads to agents that are both successful in training and successfully generalize to a wide range of backgrounds. Moreover, a larger value of λ yields higher generalization scores. Of the other regularization schemes that we tested, we find that although they do improve learning on the reference domain, only dropout leads to improvement in generalization over the randomization space compared to our baseline.

6. CONCLUSION:

In this paper we studied generalization to visually diverse environments in deep reinforcement learning. We formalized the problem, illustrated the inefficiencies of standard domain randomization, and proposed a theoretically grounded method that leads to robust, low-variance policies that generalize well. We conducted several experiments in different environments of differing complexities using both on-policy and off-policy algorithms to support our claims.

A PROOF OF PROPOSITION 1:

The proof presented in the following applies to MDPs with a discrete action space. However, it can straightforwardly be generalized to continuous action spaces by replacing sums over actions with integrals over actions.

The proof uses the following lemma :

Lemma 1 For two distributions p(x, y) = p(x)p(y|x) and q(x, y) = q(x)q(y|x), we can bound the total variation distance of the joint distribution :

D T V (p(•, •) q(•, •)) ≤ D T V (p(•) q(•)) + max x D T V (p(•|x) q(•|x))

Proof of the Lemma.

We have that :

D T V (p(•, •) q(•, •)) = 1 2 x,y |p(x, y) -q(x, y)| = 1 2 x,y |p(x)p(y|x) -q(x)q(y|x)| = 1 2 x,y |p(x)p(y|x) -p(x)q(y|x) + (p(x) -q(x))q(y|x)| ≤ 1 2 x,y p(x)|p(y|x) -q(y|x)| + |p(x) -q(x)|q(y|x) ≤ max x D T V (p(•|x) q(•|x)) + D T V (p(•) q(•))

Proof of the proposition.

Let p t φi (s, a) be the probability of being in state φ i (s) at time t, and executing action a, for i = 1, 2. Since both MDPs have the same reward function, we have by definition that η i = s,a t γ t p t φi (s, a)r t (s, a), so we can write :

EQUATION

But p t φ1 (s, a) = p t φ1 (s)π φ1 (a|s) and p t φ2 (s, a) = p t φ2 (s)π φ2 (a|s), Thus (Lemma 1) :

D T V (p t φ1 (•, •) p t φ2 (•, •)) ≤ D T V (p t φ1 (•) p t φ2 (•)) + max s D T V (π φ1 (•|s) π φ2 (•|s)) ≤ D T V (p t φ1 (•) p t φ2 (•)) + K π φ 1 -φ 2 ∞ (4)

We still have to bound D T V (p t φ1 (•) p t φ2 (•)). For s ∈ S we have that :

|p t φ1 (s) -p t φ2 (s)| ≤ s |p φ1 (s t = s|s )p t-1 φ1 (s ) -p φ2 (s t = s|s )p t-1 φ2 (s )| = s |p φ1 (s t = s|s )p t-1 φ1 (s ) -p φ2 (s t = s|s )p t-1 φ1 (s ) + p φ2 (s t = s|s )p t-1 φ1 (s ) -p φ2 (s t = s|s )p t-1 φ2 (s )| ≤ s p t-1 φ1 (s )|p φ1 (s|s ) -p φ2 (s|s )| + p φ2 (s|s )|p t-1 φ1 (s ) -p t-1 φ2 (s )|

Summing over s we have that

D T V (p t φ1 (•)||p t φ2 (•)) ≤ 1 2 s E s ∼p t-1 φ 1 [|p φ1 (s|s ) -p φ2 (s|s )|] + D T V (p t-1 φ1 (•)||p t-1 φ2 (•))

But by marginalizing over actions : p φ1 (s|s ) = a π φ1 (a|s )p φ1 (s|a, s ), and using the fact that p φ1 (s|a, s ) = T φ1 (s|a, s ) = T φ2 (s|a, s ) = p φ2 (s|a, s ) := p(s|a, s ), we have that -π φ2 (a|s )| And using s p(s|a, s ) = 1 we have that :

1 2 s E s ∼p t-1 φ 1 [|p φ1 (s|s ) -p φ2 (s|s )|] ≤ 1 2 E s ∼p t-1 φ 1 a [ s p(s|a, s )]|π φ1 (a|s ) -π φ2 (a|s )| ≤ max s D T V (π φ1 (•|s) π φ2 (•|s)) ≤ K π φ 1 -φ 2 ∞

Thus, by induction, and assuming

D T V (p 0 φ1 (•)||p 0 φ2 (•)) = 0 : D T V (p t φ1 (•)||p t φ2 (•)) ≤ tK π φ 1 -φ 2 ∞ Plugging this into inequality 4, we get D T V (p t φ1 (•, •) p t φ2 (•, •)) ≤ (t + 1)K π φ 1 -φ 2 ∞

We also note that the total variation distance takes values between 0 and 1, so we have

D T V (p t φ1 (•, •) p t φ2 (•, •)) ≤ min(1, (t + 1)K π φ 1 -φ 2 ∞

) Plugging this into inequality 3 leads to our first bound,

|η 1 -η 2 | ≤ 2r max t γ t min(1, (t + 1)K π φ 1 -φ 2 ∞ )

Our second, looser bound can now be achieved as follows, 2016) with randomized textures. As opposed to the regularization scheme proposed in our work in which we regularize the final hidden layer of the network, they propose regularizing the output of the policy network. Regularizing the last hidden layer as in our scheme more clearly separates representation learning and policy learning, since the final layer of the network is only affected by the RL loss.

|η 1 -η 2 | ≤ 2r max t γ t (t + 1)K π φ 1 -φ 2 ∞ |η 1 -η 2 | ≤ 2r max (1 -γ) 2 K π φ 1 -φ 2 ∞ B COMPARISON

We hypothesized that regularizing the output of the network directly could lead to the regularization loss and the RL loss competing against each other, such that a tradeoff between policy performance and generalization would be necessary. To test this hypothesis, we performed experiments on the visual cartpole domain with output regularization with different values of regularization parameter λ. Our results are shown in figure 7 . We find that increasing the regularization strength adversely affects training. However, agents trained with higher values of λ do achieve more consistent results over the randomization space. This shows that there is indeed a tradeoff between generalization and policy performance when regularizing the network output as in Aractingi et al. (2019) . In our experiments, however, we have found that changing the value of λ only affects generalization ability and not agent performance on the reference domain.

C EXPERIMENTAL DETAILS:

All code used for our experiments will be made available online.

C.1 STATE PREPROCESSING:

For our implementation of the visual cartpole environment, each image consists of 84 × 84 pixels with RGB channels. To include momentum information in our state description, we stack k = 3 frames, so the shape of the state that is sent to the agent is 84 × 84 × 9.

We note that because of this preprocessing, agents trained until convergence achieve average returns of about 175 instead of the maximum achievable score of 200. Since the raw pixels do not contain momentum information, we stack three frames as input to the network. When the environment is reset, two random actions are thus taken before the agent is allowed to make a decision. For some initializations, this causes the agent to start in a situation it cannot recover from. Moreover, due to the low image resolution the agent may sometimes struggle to correctly identify momentum and thus may make mistakes.

In CarRacing, each state consists of 96×96 pixels with RGB channels. We introduce frame skipping as is often done for Atari games (Mnih et al. (2015) ), with a skip parameter of 5. This restricts the length of an episode to 200 action choices. We then stack 2 frames to include momentum information into the state description. The shape of the state that is sent to the agent is thus 96 × 96 × 6. We note that although this preprocessing makes training agents faster, it also causes trained agents to not attain the maximum achievable score on this environment. Given that regularized agents are stronger in interpolation over their training domain, it is natural to wonder what the performance of these agents is in extrapolation to colors not within the range of colors sampled within training. For this purpose, we consider randomized and regularized agents trained on Ξ big , and test them on the set {(x, x, x), 0 ≤ x ≤ 1}. None of these agents was ever exposed to x ≤ 0.5 during training.

Our results are shown in figure 8 . We find that although the regularized agent consistently outperforms the randomized agent in interpolation, both agents fail to extrapolate well outside the train domain. Since we only regularize with respect to the training space, there is indeed no guarantee that our regularization method can produce an agent that extrapolates well. Since the objective of domain randomization often is to achieve good transfer to an a priori unknown target domain, this result suggests that it is important that the target domain lie within the randomization space, and that the randomization space be made as large as possible during training.

C.2.2 FURTHER STUDY OF THE REPRESENTATIONS LEARNED BY DIFFERENT AGENTS:

We perform further experiments to demonstrate that the randomized agent learns different representations for different domains, whereas the regularized agent learns similar representations. We consider agents trained on Ξ split = [0, 0.2] 3 ∪[0.8, 1] 3 , the union of darker, and lighter backgrounds. We then rollout each agent on a single episode of the domain with a white background and, for each state in this episode, calculate the representations learned by the agent for other background colors. We visualize these representations using the t-SNE plot shown in figure 9 . We observe that the randomized agent clearly separates the two training domains, whereas the regularized agent learns similar representations for both domains. We are interested in how robust our agents are to unseen values ξ ∈ Ξ split . To visualize this, we rollout both agents in domains having different background colors : {(x, x, x), 0 ≤ x ≤ 1}, i.e ranging from black to white, and collect their features over an episode. We then plot the t-SNEs of these features for both agents in figure 10 , where each color corresponds to a domain. (2017) proved a generalization bound in terms of the norms of each layer, which is proportional to the Lipschitz constant of the network. Oberman & Calder (2018) studied generalization through a general empirical risk minimization procedure with Lipschitz regularization, and provides generalization bounds. Similarly, we show that the Lipschitz constant of the network with respect to the randomization parameters plays an important role in achieving zero-shot transfer to a target domain.

:



Figures and Tables:

Figure 3: Training curves over randomization spaces Ξ small (left) and Ξ big (right). Shaded areas indicate the 95% confidence interval of the mean, obtained over 10 training seeds.

Figure 4: Comparison of the average scores of different agents over different domains. The scores are calculated over a plane of the (r,g,b) cube in Ξ big , where g = 1 is fixed, averaged over 1000 steps. The training domain for both the regularized and normal agents is located at the top right.The regularized agent learns more stable policies than the randomized agent over these domains.

Figure 5: Left: Visualization of the representations learned by the agents for pink and green background colors and for the same set of states. We observe that the randomized agent learns different representations for the two domains. Right: Standard deviation of estimated value functions over randomized domains, averaged over 10 training seeds.

Figure 6: Left: frames from the reference and a randomized CarRacing environment. Right: training curves of our agents, averaged over 5 seeds. Shaded areas indicate the 95% confidence interval of the mean.

|p φ1 (s|s ) -p φ2 (s|s )| = | a p(s|a, s )(π φ1 (a|s ) -π φ2 (a|s ))| ≤ a p(s|a, s )|π φ1 (a|s )

Figure 7: Top: training curves of agents with different regularization strengths when following the scheme of Aractingi et al. (2019). Shaded errors correspond to 95% confidence intervals of the mean, calculated from 10 training seeds. Bottom: scores obtained by trained agents for different regularization strengths on a plane within the RGB cube.

Figure8: Generalization scores, with 95% confidence intervals obtained over 10 training seeds. The normal agent is trained on white (1, 1, 1), corresponding to a distance to train= 0. The rest of the domains correspond to (x, x, x), for x = 0.9, 0.8, . . . , 0.

Figure 9: t-SNE of the representations over Ξ split of the Regularized (Left) and Randomized (Right) agents. Each color corresponds to a domain. The randomized agent learns very different representations for [0, 0.2] 3 and [0.8, 1] 3 .

Figure 10: t-SNE of the features of the Regularized (Left) and Randomized (Right) agents. Each color corresponds to a domain.

also guide the domain randomization procedure by training a Decep-tionNet, that learns which randomizations are actually useful to bridge the domain gap for image classification tasks.

Table content: <table><tr><td>2.3 LEARNING DOMAIN-INVARIANT FEATURES AND DOMAIN ADAPTATION</td></tr><tr><td>Learning domain-invariant features has emerged as a promising approach for taking advantage of</td></tr><tr><td>the commonalities between domains. For instance, in the semi-supervised context, Bachman et al.</td></tr><tr><td>(2014); Sajjadi et al. (2016); Coors et al. (2018); Miyato et al. (2018); Xie et al. (2019) enforce that</td></tr><tr><td>predictions of their networks be similar for original and augmented data points, with the objective</td></tr><tr><td>of reducing the required amount of labelled data for training. Our work extends such methods to</td></tr><tr><td>reinforcement learning.</td></tr><tr><td>In the reinforcement learning context, several other papers have also explored this topic. Tzeng et al.</td></tr><tr><td>(2015) and Gupta et al. (2017) add constraints to encourage networks to learn similar embeddings</td></tr><tr><td>for samples from both a simulated and a target domain. Daftry et al. (2016) apply a similar approach</td></tr><tr><td>to transfer policies for controlling aerial vehicles to different environments. Bousmalis et al. (2017)</td></tr><tr><td>compare different domain adaptation methods in a robot grasping task, and show that they improve</td></tr><tr><td>generalization. Wulfmeier et al. (</td></tr></table>

Peer review (iclr2020):
This paper proposes a regularization scheme for training vision-based control policies that are robust to variations of the visual input, which the paper classifies as *visual randomization*. Visual randomization, as the paper notes, is one of the state-of-the-art techniques proposed from simulation to real robot transfer of vision-based policies trained on synthetic images. The regularization proposed in the paper aims to produce policies that rely on features that are invariant to the visual randomization, so that the resulting behaviour of the agent is consistent across different variations of its visual input for the same states. The paper proposes that forcing a policy trained under randomization to have a Lipschitz constant of K, over the randomization parameters, causes the optimal policy to be similar across randomization parameters, with the difference in expected returns of two randomized environments being bounded by K times the maximum difference in their randomization parameters. I recommend this paper to not be accepted until the following issues are addressed. * There are missing details from the experimental setup, which makes the results hard to interpret (see, below). * There are missing details on how vanilla domain randomization was implemented. Domain randomization aims to maximize the expected performance over the environment distribution. This can be implemented properly by computing the expected gradients with data from more than one environment. From the algorithm descriptions in the appendix, it is not clear that this is how vanilla domain randomization was implemented. * The title, introduction and conclusions do not reflect the scope of the paper. The paper only addresses situations where the same behaviour is achievable on all environments, an assumption (Mehta et al, 2019) also makes, and its proposed regularization is based on the assumption that the optimal behaviour is achievable with the same policy on all environments. But this is not true in general: for dynamics randomization, different environments may require different policies (e.g. driving a car on a road vs driving off-road). The regularization method may result in overly conservative policies i such situations. Questions about experimental details: What are the maximum returns for Cartpole when trained until convergence without randomization? (175? 200? 1000?) If the maximum returns are higher than 175, how does Figure 4 look with more data? This is crucial to understand, for example, the results in Figure 11. That figure shows the proposed regularization slightly hinders the performance for the environments near l=1, g=50 (that region is a darker shade of green on the left subfigure). How do we know if the task has been successfully solved in the green vs purple regions? In all experiments, are the training curves showing the performance of the policies over the same environments (same seeds)? If not, how are the training curves comparable? Other things to improve: The conclusions of this paper can be made stronger by adding a comparison with EpOpt-PPO (i.e. optimizing the worst case performance over a set of trajectories sampled from multiple environments)