Domain Randomization via Entropy Maximization
Authors: Gabriele Tiboni
Pascal Klink
Jan Peters
Tatiana Tommasi
Carlo D'Eramo
Georgia Chalvatzaki
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Sim-to-Real Transfer
Domain Randomization
Primary area: reinforcement learning
TL;DR: A novel approach for sim-to-real transfer with domain randomization that directly maximizes the entropy of the dynamics distribution during training, while retaining convergence and generalization capabilities.

Abstract: Varying dynamics parameters in simulation is a popular Domain Randomization (DR) approach for overcoming the reality gap in Reinforcement Learning (RL). Nevertheless, DR heavily hinges on the choice of the sampling distribution of the dynamics parameters, since high variability is crucial to regularize the agent's behavior but notoriously leads to overly conservative policies when randomizing excessively. In this paper, we propose a novel approach to address sim-to-real transfer, which automatically shapes dynamics distributions during training in simulation without requiring real-world data. We introduce DOmain RAndomization via Entropy MaximizatiON (DORAEMON), a constrained optimization problem that directly maximizes the entropy of the training distribution while retaining generalization capabilities. In achieving this, DORAEMON gradually increases the diversity of sampled dynamics parameters as long as the probability of success of the current policy is sufficiently high. We empirically validate the consistent benefits of DORAEMON in obtaining highly adaptive and generalizable policies, i.e. solving the task at hand across the widest range of dynamics parameters, as opposed to representative baselines from the DR literature. Notably, we also demonstrate the Sim2Real applicability of DORAEMON through its successful zero-shot transfer in a robotic manipulation setup under unknown real-world parameters.

Published as a conference paper at ICLR 2024
DOMAIN RANDOMIZATION VIA ENTROPY MAXIMIZATION
Gabriele T2,, Pascal Klink Jan Peters2.4.5,6 Tatiana Tommasi', Carlo D'Eramo 2,3,1 Georgia Chalvatzaki2.47
DDparrmment of Control and Computer Engineering, Politecnico di Torino, Italy
Departmennt of Computer Science, Technische Universität Darmstadt, Germany
*Center for Artificial Intelligence and Data Science University of Würzzbrgg Germany "Hessian Center for Artificial Intelligence (Hessian.Al), Darmstadt, Germany 5 *Centre for Cognitive Science, TU Darmstadt, Germany.
@Systems AI for Robot Learning, German Research Center for AI (DFKI)
7 "Center for Mind, Brain and Behavior, Uni. Marburg and JLU Giessen, Germany *Corresponding author: gabriele. tiboni @polito.
ABSTRACT
Varying dynamics parameters in simulation a popular Domain Randomization (DR) approach for overcoming the reality gap in Reinforcement Learning (RL). Nevertheless, DR heavily hinges on the choice of the sampling distribution of the dynamics parameters, since high variability is crucial to regularize the agent's s behavior but notoriously leads to overly conservative policies when randomiz- ing excessively. In this paper, we propose novel approach to address sim-to- real transfer, which automatically shapes dynamics distributions during training in simulation without requiring real-world data. We introduce DOmain RAndom- ization via Entropy MaximizatiON (DORAEMON), a constrained optimization problem that directly maximizes the entropy of the training distribution while re- taining generalization capabilities. In achieving this, DORAEMON gradually in- creases the diversity of sampled dynamics parameters as long as the probability of success of the current policy is sufficiently high. We empirically validate the consistent benefits of DORAEMON obtaining highly adaptive and generaliz- able policies, iee. solving the task at hand across the widest range of dynamics parameters, as opposed to representative baselines from the DR literature. No- tably, we also demonstrate the Sim2Real applicability of DORAEMON through its successful zero-shot transfer in a robotic manipulation setup under unknown real-world parameters.
INTRODUCTION
The low sample efficiency of Reinforcement Learning (RL) algorithms and the expensive data col- lection routine on real hardware have limited the development of fully autonomous robots for the real world. In addition, trial and error approaches such as RL require random exploration to find op- timal policies, which raises crucial safety concerns for applications in roboties (Kober al., 2013). For these reasons, training in simulation is promising alternative direction for data-driven robot learning, providing a safe way to collect data with minimal costs (Zhao et al., 2020). However, the discrepancy between the simulated environment and the real setup- tolup-commolly referred to as the re- ality gap-ultimatel) hinders policy transfer (Valassakis et al., 2020). Sim-to-real methods attempt to close the reality gap in various ways, e.g., inferring simulator parameters from real data (Zhu et al., 2018), randomizing parameters to encourage policy robustness (Peng et al., 2018; Antonova et al., 2017), combining both approaches (Tiboni et al., 2023; Chebotar al., 2019).
Domain Randomization (DR) is the sim-to-real approach that varies simulator parameters according to given distribution at training time (Muratore al., 2022b), effectively inducing an additional source of stochasticity over the environment dynamics. Intuitively, DR trades optimality for ro- bustness (Josifovski et al., 2022): excessively high randomization leads over-regularized policies

---

Published as a conference paper at ICLR 2024
that are no longer able to reach optimal performance; the other hand, unnecessarily narrow dis- tributions may result in policies that fail to generalize. Notably, to address the artificially induced non-stationarity by DR. we can cast the problem into partially-observabee one, and employ non- Markovian (e.g., bistory-dependent) policies to learn adaptive behaviors over the varied dynamics parameters (Chen al., 2022). However, DR still requires tedious manual tuning to get candidate training distributions which would generalize best to the real world (Vuong et al., 2019b).
In response to these issues, various methods in the literature propose automatize DR for achieving better zero-shot transfer, in the absence of real-world data. Yu et al. (2018) work around the partially- policies, but then rely on a test-time parame-
observable setting by training imaamics-condiioone ters search to find an optimal policy. More recent works suggest gradually guiding the training distribution over time, to maximize the average performance over a fixed reference range of dynam- ies (Mozian et al., 2020), or over the boundary of the training uniform distribution (Akkaya et al., 2019). While promising, these works require iteratively testing the policy on -ittrr-disttibutionn dynamics or biasing sampled training data towards the boundary of the current distribution. As result, considerably more environment interactions than a traditional DR pipeline are needed.
In this paper, we propose novel method to automatically guide the training dynamics distribution, without requiring worldworld data. We follow the intuition that better generalization can be achieved with increased diversity of sampled dynamics parameters. Therefore, propose our method for DOmain RAndomization via Entropy MaximizatiON (DORAEMON), which directly maximizes the entropy of the distribution training time, ie.,, gradually solving the task over more diverse environment dynamics. Crucially, we constrain the growth of entropy by the probability of success of the current policy, to ensure fair convergence along the process and avoid performance collapse from excessive randomization. DORAEMON only requires a notion of task success to be provided given the task hand, such as a task-solved return threshold or a success indicator function based on custom metric-e,, e.g., distance from a goal location. This way, the DR sampling distribution can be automatically widened to reach the maximum entropy value such that policy success still occurs with arbitrarily high probability. As a result, the complexity of the problem shifts from tun- ing a number of dynamics parameter distributions, to simply defining a binary rule that determines whether trajectories are considered success not. In particular, we found consistently more sample efficient than Mozian et al. (2020) and Akkaya et al. (2019), as no addi- tional environment interactions are needed for updating the distribution besides episodes naturally experienced at training time. Our thorough experimental evaluation in simulation demonstrates the capabilities of DORAEMON to solve common benchmark tasks in much wider range of dynamics parameters, hence achieving better generalization than DR baselines. Finally, we show the remark- able Sim2Real transferability of policies trained through DORAEMON, enabling a 7-DoF robotic arm to push a box under unknown center of mass, weight, and contact dynamics.
approach to be
2 RELATED WORK
Domain Randomization (DR) has been widely investigated in recent years as a method to over- come the reality gap (Muratore al., 2022b; Zhao al., 2020). Although initially studied in the context of randomization for visual properties of simulators (Tobin et al., 2017), our work deals with the non-stationarity of dynamics distributions--e.2 varying friction coefficients, masses. This a.auulationa.kka. Dynamics getloommmiiaiinngnraallll aims at learning policies that are robust to changes in dynamics, and demonstrated a number of success stories of zero-shot sim-to-real transfer for deep DRL (Antonova et al., 2017: Tan et al., 2018; Ding et al., 2021). Interestingly, more recent policies to allow for implicit system identification and adaptive behavior, which demonstrated superior performance (Peng et al., 201 8; Akkaya et al., 2019). This permitted by the natural implementation of DR where new dynamics parameters are sampled at the start of each training episode, effectively inducing latent MDPs (Chen et al., 2022).
works employ history-dependes
Despite better training convergence through adaptivity, training policies with DR still requires con- siderable manual tuning (Vuong et al., 2019a). For this reason, finding automatic ways to obtain suit- able training DR distributions would be a promising solution. Simulator- based inference methods allow the collection of real-world data for estimating posterior distributions over dynamics parame- ters, which are ultimately used for DR (Tiboni et al., 2023; Chebotar et al., 2019; Tsai et al., 2021; Muratore et al., 2022a). These methods are often referred to Adaptive Domain Randomization,
2

---

Published as a conference paper at ICLR 2024
and generally depart from the inference objective used and/or assumptions over the data collection routine (Tiboni et al., 2022), i.e. whether only fixed off-policy trajectories are given vs. iteratively allowing policy evaluation in the real world.
A complementary research direction investigates the sim-to-real transfer challenge in the absence of real data, as in the scope of this work. Muratore al. (2019) proposed a novel stopping criterion for training RL policies in simulation by assessing generalizability within the DR distribution with ad- ditional policy evaluations. Similarly, Mozian et al. (2020) suggest directly optimizing for average performance over a reference dynamics range but instead propose to train on a moving distribution which is guided accordingly. Notably, these methods require frequently evaluating policies through Monte-Carlo rollouts, resulting in poor data efficiency. Alternatively. Mehta et al. (2020) attempt to deviate from the usual DR formulation and design a learned sampling strategy guided by policy search itself: sampler policy is rewarded for choosing dynamics parameters where the task policy behaves noticeably more differently than normal. Note how the aforementioned approaches still crucially rely on reference DR distribution to be provided at training time, as it is necessary for assessing generalizability. In contrast, Akkaya et al. (2019) promises a fully automated algorithm for DR, by leveraging the performance of the policy at the boundaries of a uniform distribution as a proxy for assessing generalizability in each dynamics dimension separately. As a result, however, this approach is by definition confined to uniform distributions and significantly biases policy train- ing on dynamics parameters collected at the boundaries (e.g. half the training time in the original implementation). In comparison, our work relaxes both these assumptions while addressing the same general setting-no reference DR distribution must be defined. Interestingly, Akkaya et al. (2019) also demonstrate that the gradual growth of the sampling distribution yields superior overall performance w.r.t. fixed target, likely due to an effect of curriculum learning (Bengio et al., 2009).
In the context of RL. curriculum learning has shown promising results by adjusting the task difficulty over time (Baranes & Oudeyer, 2010; Fournier et al., 2018; Cho et al., 2023)-i.e. the curriculum. This line of work resembles our setting in that the training environment is non-stationary and guided to maximally benefit the agent. In particular, self-paced learning approaches recently addressed the problem of automatically generating a curriculum towards a final target task by ensuring a suf- ficiently high degree of performance along the process (Klink et al., 2020b;a; 2022; Koprulu & Topcu, 2023). Although related, our work deviates from self- self-paced curriculum learning methods by maximizing the entropy of a distribution over hidden dynamics parameters to facilitate the gener- alization of learned policies, instead of making an agent proficient on a specified distribution of a target task-See Appendix B for a thorough comparison of the two problem settings.
BACKGROUND
Consider the discrete-time dynamical system described by a Markov Decision Process (MDP) M, with state space S, action space A, initial state distribution μ.c. S + Rt, transition dynamics dis- tribution P:S x x S + IR+ and reward function R:S x A x + R. At each time t, the environment M evolves according to the current state 8 S and action at A taken by an agent, Le. the decision maker, with initial state Sp ~ ((()). As a result of the state transition, a scalar re- ward signal r. R.=R(st, de, 8141) returned. We then formulate the DR problem by modeling the simulator as a set 2/ of MDPs with tunable latent dynamics parameters { E C R Each MDP Mg U shares the same state space, action space, and reward function, but differs by its asso- ciated transition dynamics Pe Pe(st=+ az). Dynamics parameters & are generally assumed to be random variables distributed according to parametric distribution uy: - R parametrized by @ E Φ S R Such distribution defines the sampling probability of environments Me at training time. In other words, the agent is iteratively presented with a random sample Mg and can learn from experience by collecting trajectories T = {(5,, de. re, s1+1))T= E T, encoding the resulting state-action rewtard-rewadddd tuples visited. Under this formulation, DR for RL addresses the problem of find- ing an optimal stochastic policy Tī (4g (8x) such that the expected (discounted) cumulative reward is maximized, while acting over an unknown distribution of MDPs MM, induced by ~ us())
J(0,0) = Bgguo(() R Bampo(ris)
ar, EnD ("R(ss,
(I)
Pe(TTE) = u(so) 11 Pe(sc411se, at Te)lss)))))
3

---

Published as a conference paper at ICLR 2024
with discount factor [0, 1). In particular, notice how To is not conditioned the latent dynamics parameters Ç. In turn, this formulation limits the capabilities of the policy to adapt its behavior according to different environment dynamics. As in partial observable environments, we therefore adopt the standard approach of considering a history of transitions và E V, where Vi is the set of all possible histories of transitions of the form the {so: ao 81, "1,--.,,, :,- 5, E A),
where pl-) and az ~ Iaa (-))))))
4 METHOD
Our method learns a policy To proficient on a real-world MDP M by training it purely in simula- tion a set of MDPs Me E U. Given that we do not know which parameters &" correspond to the real-world MDP M we maximize the chance of good performance in M by training a policy ma that generalizes well to the maximum range of environments in U. To effectively measure the degree of generalizability across U1 we leverage a notion of success for the underlying trajectories TET, as it gives a better definition of "acceptable" behaviors across tasks. For example, defining suc- cess through task-specific knowledge is often more natural for real-world tasks in robotics, as done in (Florensa al., 2018: 2017). Thus, without loss of generality, we introduce a simple success indicator function o: T {0, }. which may be defined through, e.g., distance thresholds from target goal location, task-specific tolerance to errors, or a lower bound on the expected return. Based on the success indicator (r), the probability of success
g(O, 6) = F[(((T) = 1|0, ø]
(2)
assesses the capabilities of the policy vo to solve the task in simulation over multiple dynamics & ~ (.).). The trajectories are now effectively distributed according to the marginal distribution (F(()) Wider dynamics distributions PE(E) will generally make it harder for the policy to keep high success rate G( Ø). On the other hand, high variability of environments the real environment dy- namics. Following this intuition, we introduce Domain Randomization via Entropy Maximization (DORAEMON), constrained optimization problem formulated as
(T)
Mg sampled at training time will likely lead to better generalization
max
H(vs) s.t. G(0, 2 az
with (((lvs))-BEnuaae log (E)] being the (differential) entropy of vs, and desired in in-distribution success rate a E c
DORAEMON aims at automatically finding policy that generalizes well to the widest range of dynamics parameters &. To this end, we propose maximizing the entropy of the training distribu- tion vo(E) to gradually increase the diversity of sampled environments, towards assigning equal (uniform) probability density to each Me E U. While doing so, we ensure that the policy is still able to retain desired probability of success a, which prevents policy performance from collapsing due to excessive randomization. In general, V/8 might not converge to the maximum entropy uni- form distribution-rrferre probability of success a may no longer be maintained. It is worth noting that does not directly determine the probability of success ((0, Vimax) of the policy across Vmmax. Notably, policies with lower in-distribution success rate a may still generalize better thanks to higher-entropy training distribution (see success rate vs. entropy trade-off Sec. 5).
as Vmax assuming bounded support-but, instead, will stop when the
4.1 ALGORITHMICC IMPLEMENTATION
The objective in (3) requires the joint optimization of the training distribution us and the policy ma- In practice we consider a decoupled optimization of the policy with any RL subroutine of choice, which interleaves dynamics distribution updates. Therefore, we conveniently rewrite (3) as
max H(Vo..)) s.t. G(O,, (.+1) 2a DKL [ S 1+:1E
(4)
where ne, has been trained on dynamics parameters drawn from & ~ Vo, (-). Importantly, Te, is conditioned on fixed-length history over previous state-action pairs to account for the non- observability of environment parameters ç. In this work, we train policies with Soft Actor-Critic (SAC) (Haamoja al., 2018), and additionally condition the critic network with the true dynamics
4

---

Published as a conference paper at ICLR 2024
parameters, as in Peng al. (2018); Akkaya al. (2019), to further mitigate the non-stationarity induced by sampling dynamics from Vs- We then constrain the Kullback-Leibler (KL) divergence between subsequent dynamics distribution updates. The resulting trust region around Va. prevents abrupt changes and controls the growth of the training distribution during the optimization process. Note that the probability of success ( Φ) is equal [r] as (T) is effectively distributed as a Bernoulli random variable. Thus, we can approximate the success rate G(O,, d(41) in (4) through the importance sampling (IS) estimator
Votti (Ex) Vo. (Ex) -1 (TET:
G(0 oi, 01+1)
Σ
(5)
In turn, this allows the optimization problem in (4) to be solved using only the of data ()) naturally collected while training the policy no, for K episodes. In principle, one could collect additional Monte-Carlo evaluations of the policy to approximate G(O,, 0++1) with the most recent parameters 0,+1. However, recycling training data works sufficiently well throughout our experiments and consequently leads to a more efficient pipeline.
While convenient, approximating the probability of success G(0;, (1+1) through IS may lead to over- estimation of the real success rate under the new distribution Φi+1, resulting in a violated constraint when trying to solve (4) at the next iteration. This occurrence may be particularly troublesome for DR settings, as there are no guarantees that a policy may solve the task over the entire set of MDPs U. To account for potential constraint violations, we introduce a back-up optimization problem which attempts to find a new feasible starting solution of within the current trust region, formulated as
0e
= arg max ((),, Oi, d') S.t. DKL(VO S. 0,50
(6)
In simple words, we find sufficiently close distribution that has maximum (approximated) in- distribution success rate. We observed this addition to be crucial for recovering policy performance by backtracking the distribution entropy, which would otherwise be prevented. The overall practical implementation of DORAEMON is summarized in Algorithm 1.
In this work, we parameterize (C) as uncorrelated univariate Beta distributions, allowing for sim- pler comparison with methods that require bounded support. In addition, dynamics parameters are often inherently bounded due to physical constraints-such as positive values for masses and friction coefficients-mall
Beta distributions a reasonable choice. However, the formulation is not re- stricted to a particular family of parametric distributions or even continuous random variables (See Sec. A.4) e.g., discrete distributions over object shapes could be used.
4.2 ToY PROBLEM
We report a practical example of DORAEMON in toy problem to convey a clear intuition of our method. Let us consider a frictionless inclined plane, with inclination angle w (Fig. la). An actuated cart (i.e. the agent) is placed on the plane surface and is subject to gravity force Fg. The cart's goal to apply a counter force a, E [-amax: 4max] to fight gravity and balance itself around the center of the plane. In this setting, E == w E I- 2: 3] C IR is the underlying dynamics parameter for our problem. Therefore, the objective is to learn a dynamics-agnosti
policy Te which can balance the cart across the widest range of inclinations W. First, note how the policy benefits from keeping a history of past actions and observations for solving the task (see Sec. 3), as w can be implicitly inferred from previous ttaeetrranitiion. The agent can only successfully solve the task for instances of w where gravity has a sufficiently low impact. This represents the general problem of learning under potentially infeasible dynamics in DR settings as the distribution gets wider and wider. There- fore, we use DORAEMON to automatically find the maximum entropy distribution where the task may still be solved with probability a without requiring additional domain knowledge. Notably, the simple task at hand allows us to analytically compute the feasibility boundaries of w. namely |w| I We arcsin Smax assuming S Gmax I Fg. This makes it easy assess the capabilities of DORAEMON to solve the task over all parameters across the true, feasible range.
F.
We report the results of policies trained with our method in Fig. 1, together with illustration of the underlying task. We highlight the effect of the hyperparameter a in shaping the final distribution (colored in dark blue): policies with lower values encounter infeasible tasks more often while
5

---

Published as a conference paper at ICLR 2024
Algorithm 1: Domain Randomization via Entropy Maximization (DORAHMON)
input : Initialize dynamies distribution Φ1, policy parameters 0,, trust region size e, trajectories distribution update K, number of iterations M, success indicator function (T).
output: Generalizable policy T8xt
for i ,....,M .M do
Sample K dynamics parameters {Ex 13k] EN ~ Vo, (5)
Collect K associated trajectories [T&] k=11== Tk Po, (T(
Policy update:
Obtain 0i+1 through any RL algorithm on collected data {Tu]k=]
Dynamics distribution update:
(0,, dis Φ.) < a then
Obtain Φ by optimizing (6)
((,,, , di, ( < a then jump to next iteration with Φ1+1 + o
Satii o
10
11
else
Sstart di
13
end
Obtain Uorts by optimizing (4) with 9 x G(O,, "m .d+11) (Eq. 5)
end
We
-ik
di
+2
-0k 6cc
Angle
Angle
Angle
(a) Task illustration
(b) a = 0.5
(c) = 0.75
(d) a 0.9
Figure 1: DORAEMON's moving Beta distributions over the plane inclination angle w, for different values of in-distribution success rate a. The converged "max-entropy" distribution is such that the policy can solve the task with probability a (green for success, red for failure). The physically infeasible dynamics region is highlighted with red background.
training. whereas higher a lead to a more conservative final entropy. For each policy, we finally sample 50 dynamies parameters from the converged final distribution and display them green dots to indicate success and red for failure. Here, we define a trajectory to be successful if the agent balances the cart around the center of the plane for least 25 timesteps. Overall, regardless of the in-distribution success rate, all policies learn to solve the task across the widest range of feasible parameters [-We; We]. which is our ultimate goal.
5 EXPERIMENTS
5.1 BASELINE METHODS
No-DR. We refer to policies trained single simulator instance with fixed dynamics parameters as No-Domain Randomization (No-DR). This naive approach reflects the baseline capabilities to generalize the range of environments Fixed-DR. As our method guides the sampling DR distribution over time, we compare it with the simple popular approach of keeping fixed distribution at training time. We consider set of para- metric MDPs 24 with bounded support and train Fixed-DR policies with a uniform distribution that covers the whole range 14, í.e. the maximum entropy distribution Pmax-
when no randomization is applied.
LSDR. Mozian et al. (2020) introduce novel method to guide DR distributions. In contrast to DO- RAEMON, LSDR requires a reference DR distribution to be given. Then, a (multivariate Gaussian) training distribution V9 is found such that maximum generalization the reference distribution is achieved. our experiments, we set the entropy of the initial distribution to be the same as DO- RAEMON's s and set Umax the reference distribution.

---

Published as a conference paper at ICLR 2024
Cartpole (20)
SwingUpCartpole (2D)
Hopper (7D)
(13D)
HalfCheetah (8D)
86
03
00
10
00
5.0
Entroyy
10
30
1.0
0.0
No-DR
Fixed-DR
LSDR
AutoDR
DORAEMON (ours)
Figure 2: Sim-to-Sim results: global success rate computed the maximum-entropp uniform distri- bution (top) and entropy of the current training DR distribution (bottom). The number of randomized
parameter dimensions reported in parenthesis (see Table for details).
No-DR
Fixed-DR
LSDR
AutoDR
DORAEMON (ours)
02
5000
Frction
03
4030
06
52 112 Torso mass
13
Torso mass
Torso maas
Torso mass
Torso mass
Figure 3: Average test return across 2 dynamics parameters, in the HalfCheetah environment (10 seeds). The green outline highlights the region of cells where a success occurs (a(T) = 1).
AutoDR. Automatic Domain Randomization (AutoDR) by Akkaya et al. (2019) gradually increases the entropy of uniform DR distribution according the performance of the policy measured at the boundaries. Analogously to our work, notion of success defined through an average return threshold, which determines whether the uniform bounds may be widened. As in the original paper, the sampling distribution Vo initialized to approach zero variance and located at the center U.
5.2 SIM-TO-SIM TRANSFER
We conduct thorough experimental evaluation of DORAEMON on six benchmark tasks in simu- lation, from the OpenAl AI Gym (Brockman et al., 2016) MuJoCo environments. We define a specific set of randomized dynamics parameters E tailored each environment-e,, e.g. link masses and sizes, surface friction coefficients- success through a lower bound on trajectory return, to allow a fair comparison with AutoDR and LSDR which have not been tested before performance metrics beyond the reward function (see Appendix A for the full details). We then conveniently use (T) to measure the generalization of each method across all MDPs namelyyy G(,, Vmmx) referred
with associated success indicator function (T)). ). We define
the Global Success Rate.
Effect of entropy maximization on policy generalization. The core results of our evaluation are illustrated in Fig 2. During training. progressively report the entropy of the (moving) distri- butions and the global success rate of the policies on the maximum entropy Vmax distribution (10 seeds per method). In general, observe that policies trained with DORAEMON demonstrate consistent trend of better and/or faster convergence across all tasks. We found LSDR to steadily converge intermediate entropy values as consequence of their opt. problem formulated as an M-projection (see Appendix B for details). In contrast, AutoDR was able to regularly increase the training distribution entropy. albeit with considerably higher variance across different seeds. Yet, DORAEMON outperforms AutoDR in all environments. We suspect AutoDR shortcomings are due to inefficient use of training data: collected returns can only be used to update one dimension of the uniform distribution (at most), or even discarded if the performance threshold is not met. Con- versely, DORAEMON makes use of all sampled dynamics parameters to update Vo,, acting on all parameter dimensions in & at the same time. Interestingly, the degradation in performance over time the Walker2D and Swimmer tasks is likely due to the agent's 's exposure to harder/infeasible param- eters, which destabilize training. To mitigate this effect, we track the best-performing policy during
7

---

Published as a conference paper at ICLR 2024
2000
In-dstributton
Entropy 1
Ernopp
0.2
Median
1600
1850
1400
Apx 1800
0.0
0.5
1.0
0.0
0.5
0.0
0.5
1.0
6.0
3.5
0.0
10
15
Timesteps
(a) Success Rate (SR) analysis
(b) Performance vs Entropy analysis
Figure 4: Analysis on the impact of the hyperparameter a (a), and of the provided lower bound return threshold JLB for defining success (b) in the Hopper environment.
training in terms of global success rate. Finally, we report the performance of the pert-perfonning policies on the HalfCheetah environment in Fig. 3, by testing a discretized 2D-slice of the dy- namics space-the remaining parameters are kept fixed at the nominal values. The figure attests the ability of DORAEMON to solve the task over the widest range of dynamics parameters.
Effect of curriculum on policy training. Comparing the performances of DORAEMON with Fixed-DR policies in Fig. 2 sheds light the importance of gradually widening the training dis- tribution, rather than sampling parameters from the maximum entropy one directly. DORAEMON always outperforms Fixed-DR policies- when it eventually converges Vmmax likely due to an induced curriculum over dynamics parameters, in line with findings in Akkaya et al. (2019).
Success rate vs. Entropy trade-off. The in-distribution success rate regulates the trade-off be- tween increasing distribution entropy and exposing the agent to dynamics that may not be solved yet. Limiting the distribution entropy encompass only successful tasks may stabilize training but, on the other hand, prevent generalization to out-of-distnn off-iiitiibttion dynamics that have never been seen. We study this trade-off and report the results for the Hopper environment in Fig. 4a. We find that the value of a = 0.5 generalizes sufficiently well-likely due to training on higher entropy e values- and selected this value for all our experiments. This way, we assure that a desirably high median return is maintained. Notably, the median is not affected by the value of catastrophic returns collected on infeasible dynamics parameters, as a plain average would. This further motivates the use of the success rate as metric to guide our entropy maximization objective, rather than the average return.
Performance vs. Entropy trade-off. While a a hyperparameter of our method, the success in- dicator function a (T) should be defined through domain knowledge. The algorithm should behave agnostically w.r.t (T). hence tracking the desired success rate a equally well, regardless of the success threshold defined, Fig. 4b illustrates this behavior for the Hopper environment, by setting (T) = 1 (reT: Ju) (Tk), for three performance lower bounds fLB=(1400, 1600, 1800], with collect
fixed a-0.5. We report the median performance and study the ability of DORAEMON returns J (T) 2 JI. at least half the time along the process. We observe that the algorithm success- fully exploits the success indicator to drop below optimal performance as much as the respective thresholds allow, trading performance for robustness to wider dynamics (i.e., higher entropy).
5.3 SIM-TO-REAL TRANSFER
We finally assess the proposed method in obtaining policies that can transfer well to the real world. To this end, we design a novel dynamics-sensitive task tailored to the study of learning generalizable policies across unobservable dynamics. We introduce the PandaPush environment, 7DoF robotic manipulation task with the goal of pushing a square box with unknown center-of-mass towards a target location'. We reproduce the real setup in simulation with MuJoCo's physics engine (Todorov et al., 2012) and use DORAEMON to train single policy that can be successfully deployed over different center-of-mass configurations (see Appendix C for details).
We consider total number of 17 dynamics parameters to mitigate the reality gap, including box mass (1), surface friction coefficient (1), joint damping and friction (14), and center of mass (1) along the perpendicular axis to the pushing direction. We remark that adding the randomization of more parameters besides the center of mass is crucial to get a smooth deployment of policies in the real world. We report the results of our method compared to the baselines in Tab. 1. Notably, we observe impressive behavior both in simulation and on the real setup when deploying policies trained with DORAEMON (cf. Fig 6).
Videos available on the project wehsite https: http://abbbreeeebbbbbaeea wwt.iioooigggubbb........... hoddamaoon......o...o.o...ooo...
htttps://gabrieeeeibbooo
8

---

Published as a conference paper at ICLR 2024
No-DR
Fixed-DR
LSDR
AutoDR
DORAEMON (ours)
Sim22Sim
Sucuess Tale
15.06%
0.02%
37.77%
30.45%
66.57%
distance to target (cm)
11.66 # 6.55
21.04 = 2.70
8.30 # 1.38
8.27 = 6.39
3.17 = 3.04
Sim2Real
success rate
13.33%
0%
46.67%
26.67%
60%
distance to target (cm)
5.05 + 6.28
23.26
2.25
7.38 1 8.69
4.17 = 2.07
2.68 + 1.01
Table I: PandaPush task: success rate and final distance of box w.r.t. goal (cm) tested for the maxi-
mum entropy distribution averaged over 10000 rollouts for Sim2Sim, and 30 rollouts for Sim2Real.
The task is successfully solved if the agent pushes the box within a 3cm radius of the goal.
Figure 6: Illustration of real-world rollout when deploying policy trained with DORAEMON, four snapshots during the 6-second trajectory. The history-basee policy shows impressive behavior right from the start, as slightly touching the box reveals the shifted center of mass, hence the gripper is promptly moved accordingly (second snapshot figure). The green dot highlights the goal location.
Importantly, we find LSDR to be poorly scalable to our 17-dimensional dynamics space-the computa- tional complexity for approximating [0, (max) through Monte-Carde policy evaluations grows exponentially with higher
the number of randomized dynamics- variance across multiple seeds. Analogously, AutoDR shows limited performances, which we attribute the data-inefficient growth of the uniform distribution, i.e. slower pace than DORAEMON. In turn, this prevented AutoDR policies from encompassing enough variability the dynamic space to generalize well to the real world. Finally, all 10 policies trained on fixed wide DR dis- tribution (Fixed-DR) are unable to learn any meaningful
Figure 5: PandaPush setup: the 7DoR robot needs push box of vary- ing center-of-mase to a desired location. behavior, highlighting the problem of learning excessively diverse dynamics.
CONCLUSION AND DISCUSSION
We presented novel method for Domain Randomization via Entropy Maximization (DORAE- MON), which constitutes a significant step in addressing the sim-to-real transfer gap in Reinforce- ment Learning (RL). DORAEMON automates the selection of simulator dynamics parameters at training time by progressively widening the sampling distribution, alleviating the need for costly real-world data and manual tuning. Importantly, we constrain the entropy maximization such that sufficiently high policy performance is maintained along the process. In turn, DORAEMON profi- ciently balances the trade-off between policy convergence and generalization, i.e. a common chal- lenge in domain randomization. Our empirical results, both in Sim2Sim control tasks and a versatile Sim2Real manipulation task with a 7DoF robotic arm, showcased the superior performance DO- RAEMON w.r.t. representative baselines, underscoring its potential in narrowing the reality gap.
Limitations. We suspect DORAEMON performance might suffer from collapsing to some "easy" region of the optimization landscape when backtracking from the current distribution see Ap- pendix B for details. Adding KL constraint between the current policy and the best-perf policy found during training could perhaps prevent catastrophic forgetting in these cases. Moreover, prior knowledge of the dynamics is given, it may be beneficial to bias the growth of the distribution stay around it, despite being harder optimization problem to solve.

---

Adrien Baranes and Pierre Pierre-Yves Oudeyer. Intrinsically motivated goal exploration for active motor learning in robots: A case study. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1766-1773, 2010. doi: 20.1100//O0.20000555561 Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was funded by the German Federal Ministry of Education and Research (BMBF) (Project: 011S22078). This work was also funded by Hessian.ai through the project 'The Third Wave of Ar- tificial Intelligence - 3AI' by the Ministry for Science and Arts of the state of Hessen. The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen Na- tional High Performance Computing Center (NHR @ FAU) of the Friedrich-Alexande--nnie Universittt Eetangen-Nürberr (FAU) under the NHR project b187cb. NHR funding is provided by federal and Bavarian state authorities. NHR @FAU hardware is partially funded by the German Research Foundation (DFG) - 4407 19683. We further acknowledge the support of the European H2020 Elise project (www e i se ai et),, for the availability of HPC resources and support. This study was carried out within the FAIR Future Artificial Intelligence Research and received funding from the European Union Gext-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.3 D.D. 1555 11/10/2022, PE00000013). This manuscript reflects only the authors' views and opinions, neither the European Union nor the European Commission can be considered responsible for them.
REFERENCES
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113. 2019.
Rika Antonova, Silvia Cruciani, Christian Smith, and Danica Kragic. Reinforcement learning for pivoting task. arXiv preprint arXiv: 1703.00472 2017.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference Machine Learning. ICML '09, pp. 41 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 1553374. 1553380.
11145//155337.15553300 URL ptttps://dci..oo 11145//
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 01166066015444 5400, 2016.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratiff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8973- 8979, 2019. doi: 10. 1009//CAA.201998773377
Xiaoyu Chen, Jiachen Hu, Chi Jin, Lihong Li, and Liwei Wang. Understanding domain randomiza- tion for sim-to-real transfer. In International Conference on Learning Representations, 2022.
Daesol Cho, Seungjae Lec, and H Jin Kim. Outcome-direed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation. arXiv preprint arXiv:2301.1174) 2023.
Zihan Ding, Ya- Yen Tsai, Wang Wei Lee, and Bidan Huang. Sim-to-real transfer for robotic ma- nipulation with tactile sensory. In 2021 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.
IEEE, 2021.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference on robot learning, pp. 482-495. PMLR, 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528. PMLR, 2018.
10

---

Published as a conference paper at ICLR 2024
Pierre Fournier, Olivier Sigaud, Mohamed Chetouani, and Pierre- Yves Oudeyer. Accuracy- baased curriculum learning in deep reinforcement learning. arXiv preprint wrXiv:1806.09611 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- machine learning. pp. 1861-1870. PMLR, 2018.
Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum rein- forcement learning using optimal transport via gradual domain adaptation. Advances in Neural Information Processing Systems, 35:10656-10677 2022.
Josip Josifovski, Mohammadhossein Malmir, Noah Klarmann, Bare Luka Zagar, Nicolás Navarro- Guerrero, and Alois Knoll. Analysis of randomization effects on sim2real transfer reinforce- ment learning for robotic manipulation tasks. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 10193-10200. IEEE, 2022.
Pascal Klink, Hany Abdulsamad, Boris Belousov, and Jan Peters. Self-paced contextual reinforce- ment learning. In Conference on Robot Learning, pp. 513-529. PMLR, 2020a.
Pascal Klink, Carlo D'Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. Advances in Neural Information Processing Systems, 149216-9227,, 2020b.
Pascal Klink, Haoyi Yang. Carlo D' D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum reinforce- ment learning via constrained optimal transport. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th In- ternational Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11341-11358 PMLR, 17-23 Jul 2022.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, (3(11):228-1274 2013.
Cevahir Koprulu and Ufuk Topcu. Reward seaadd--amiin-guudeeee self-paced reinforcement learning. In Proceedings of the 2023 International Conference Autonomous Agents and Multiagent Sys- tems, pp. 2451-2453, 2023.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J. Pal, and Liam Paull. Active domain randomization. Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), ), Proceedings of the Conference Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1162-1176. PMLR, 30 Oct-O1 Nov 2020.
Aditya Modi, Nan Jiang, Satinder Singh, and Ambuj Tewari. Markov decision processes with tinuous side information. In Algorithmic Learning Theory, pp. 597-618. PMLR, 2018.
Melissa Mozian, Juan Camilo Gamboa Higuera, David Meger, and Gregory Dudek. Learning do- main randomization distributions for training robust locomotion policies. In 2020 IEEE/RSJ In- ternational Conference Intelligent Robots and Systems (IROS), pp. 6112-6117, 2020. doi: 10. 10.110999RR04455442200099999494
Fabio Muratore, Michael Gienger, and Jan Peters. Assessing transferability from simulation real- ity for reinforcement learning. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2019. doi:
Fabio Muratore, Theo Gruner, Florian Wiese, Boris Belousov, Michael Gienger, and Jan Peters. Neural posterior domain randomization. In Conference Robot Learning, pp. 1532-1542. PMLR, 2022a.
Fabio Muratore, Fabio Ramos, Greg Turk, Wenhao Yu, Michael Gienger, and Jan Peters. Robot learning from randomized simulations: A review. Frontiers in Robotics and Al, pp. 31, 2022b.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba. and Pieter Abbeel. rimato-real transfer of robotic control with dynamics randomization. In IEEE International Conference on Robotics and Automation (ICRA), pp. 3803-3810, 2018. doi: 10. 109//C//RRA... A.VIICRA.200 A,201 2001///A...1...88881.1. 8.8466588.
11

---

Published as a conference paper at ICLR 2024
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804.10333 10332, 2018.
Gabriele Tiboni, Karol Arndt, Giuseppe Averta, Ville Kyrki, and Tatiana Tommasi. Online vs. of- fline adaptive domain randomization benchmark. In International Workshop on Human- Human-Friendlly Robotics, pp. 158-173. Springer. 2022.
Gabriele Tiboni, Karol Arndt, and Ville Kyrki. Dropo: Sim-to-real transfer with offline domain randomization. Robotics and Autonomous Systems, pp. 104432, 2023, ISSN 0921-8890. doi: hutps://doi.org/10.. 101110066...boot... r020) 104432.
2023.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. IEEE, 2012.
Ya- Yen Tsai, Hui Xu, Zihan Ding, Chong Zhang, Edward Johns, and Bidan Huang. DROID: Mini- mizing the Reality Gap Using Single-Shot Human Demonstration. IEEE Robotics and Automation Letters, 6(2):3168-1 119-31655,,, 2021. ISSN 2377-3766. doi: 10.1 2.110999LA..22113663311
Eugene Valassakis, Zihan Ding, and Edward Johns. Crossing the gap: A deep dive into zero-shot sim-to-real transfer for dynamies. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.
IEEE, 2020.
Quan Vuong, Sharad Vikram, Hao Su, Sicun Gao, and Henrik I Christensen. How to pick the do- main randomization parameters for sim-to-real transfer reinforcement learning policies? arXiv preprint 1iX.:19903.1174., 2019a.
Quan Vuong, Sharad Vikram, Hao Su, Sicun Gao, and Henrik I Christensen. How pick the do- main randomization parameters for sim-to-real transfer of reinforcement learning policies? arXiv preprint 1rXiv:190..11722 2019b.
Wenhao Yu, C Karen Liu, and Greg Turk. Policy transfer with strategy optimization. arXiv preprint arXiv:1810.0577 2018.
Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep rein- forcement learning for robotics: a survey, In 2020 IEEE symposium series on computational intelligence (SSCI), pp. 737-744. IEEE, 2020.
Shaojun Zhu, Andrew Kimmel, Kostas E Bekris, and Abdeslam Boularias. Fast model identification via physics engines for data-efficient policy search. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp.
2018.
12

---

Published as a conference paper at ICLR 2024
Parameters
Boundaries
JLB threshold
Gravity (ID)
2.39, 17.21
CartPole, SwingUpCartpp (2D)
400, 4200
Pole length (ID)
00.12, 0.88
Link masses (4D)
00.35, 9.75
Hopper (7D)
Joint damping (3D)
[0.17, 2.93
1600
Surface Friction (ID)
[0.17, 2.93
Torso mass (ID)
[0.88, 6.
Thigh masses (2D)
00.88, 6.87
Leg masses (2D)
0.668, 4.75
Foot masses (2D)
0.14,
Torso size (1D)
0.10,
Walker2D (13D)
1600
Thigh sizes (1D)
00.11,
Leg sizes (1D)
0.15,, 1.05
Foot sizes (1D)
0.05, 0.35
Friction Left Foot (ID)
0.48, 3.32
Friction Right Foot (ID)
0.22, 1.58)
Torso mass (ID)
0.32, 12.4
Back thigh mass (ID)
00.08, 2.99
Back shin mass (1D)
00.08, 3.08
Back foot mass (ID)
0.05,2.08
HalfCheetah (8D)
5000
Front thigh mass (ID)
00.07, 2.78
Front shin mass (1D)
0.06,, 2.30
Front foot mass (1D)
0.04, 1.66
Surface friction (ID)
(0.02, 0.78)
Link masses (3D)
28.42, 40.70]
Swimmer (7D)
Link sizes (3D)
0.08, 0.12]
112
Viscosity coefficient (ID)
0.08, 0.12]
Table 2: Search space boundaries for all the randomized parameters of the considered simulation tasks, together with the lower bound return threshold used for defining success.
A SIM-TO-SIM EXPERIMENTS
Here, we describe the details of the experimental setting carried in simulation, on tasks from the OpenAl gym suite of environments.
A. SIMULATION ENVIRONMENTS SPECIFICATIONS
For each test environment, we select a number of dynamics parameters to formulate the Domain Randomization problem tailored to the properties of the task. The number of randomized param- eters ultimately define the optimization space for LSDR, AutoDR and DORAEMON, Moreover, define finite boundaries for each environment search space, such that well-defined maximum entropy distribution exists for benchmark purposes. This allows the comparison of our method with Fixed-DR and LSDR-which require a reference target distribution to be defined-ann and allows deal- ing with inherently physically bounded parameters (e.g. positive values for friction coefficients).
We report the list of randomized properties for each task, together with the search space boundaries considered, in Tab. 2.
In general, we designed the boundaries to be as wide as possible, while being centered in the nominal parameter values and avoiding unstable values-ee e.g. masses or link sizes that are too close to zero. We therefore optimize policies with DORAEMON using Beta distributions Be(a, b) defined such ranges, with initial shape parameters Be(a = 100, b 100). However, in principle, any distribution may used. For LSDR, we stick to the original implementation setting which uses Multivariate Gaussian distributions, and set initial diagonal covariance matrix such that the entropy is equal to the starting distribution entropy of DORAEMON. Note that LSDR required an entropy computation that follows from truncated Gaussian probability density function, we resample parameters that lie outside of the search space. For optimization stability and consistency across different parameter scales, each dimension is rescaled to the interval [0, at optimization time for all methods. All dis-
13

---

Published as a conference paper at ICLR 2024
Figure 7: Different random configurations of the Walker2D environment when sampling dynamics parameters from the maximum entropy distribution.
tributions are initialized to the center of the corresponding search space. We illustrate an example of randomized configurations for the Walker2D environment in Fig. 7. Finally, an environment-speee speciffc success notion has been defined order run AutoDR and DORAEMON, and consequently used to measure the success rate of all methods. As stated in the main manuscript, the success notion can be freely defined given domain knowledge and tolerance to errors. Throughout our sim-to-sim experiments, we define success by means of a simple lower bound on the trajectory return, which we report in Tab. 2. We design threshold values JLB corresponding to acceptably good qualitative results e.g. Cartpole being balanced, Hopper jumping forward)- and such that roughly stay around 80% of the performance policy trained without DR on the nominal parameters.
A.2 HYPERPARAMETERS CHOICE
We benchmark all methods by training on the same RL subroutine, i.e. with SAC, with identical hyperparameters. In particular, policies are given the additional information on the 5 most recent state-action pairs, whereas critic networks are conditioned with the privileged information of the true dynamics parameters, which get rescaled before being fed to the network. We then individually tune the hyperparameters of the considered baselines for a fair comparison: for each method, we perform grid-search over its hyperparameters to obtain optimal average performance across all tasks; then, we separately tune a single selected hyperparameter per method on each environment individually. In particular, we choose separately tune a for LSDR, A for AutoDR, and for DORAEMON- the notation here is referring to the respective papers notation. Note that these parameters generally regulate the pace of the growing distribution. Fig. demonstrates an illustrative example of the tuning process of E for DORAEMON. We observe that higher values allow the distribution to grow faster, but may lead excessive update steps which hinder policy training and generalization. Inter- estingly, regardless of the choice of the trust region size e, the backup optimization problem in (6) allows the entropy to be adjusted- c.g. slightly reduced to maintain the desired in-distribution success rate are close to the max-entropy uniform state, as even considerable changes to the distribution param- eters only marginally affect the entropy in this region i.e. the derivative of the entropy w.r.t. the Beta parameters Be(a, 6) approaches zero as (1, 1). With this in mind, in Fig. 9 we observe an example of noticeably different Beta distributions, despite seemingly having the same entropy according to Fig. 8. Exceptionally for LSDR, we also vary the number of Monte Carlo (MC) evalu- ations in the reference distribution, as it's crucially affected by the dimensionality of the dynamics space. We then set LSDR to collect 20 nı episode returns- where ne is the number of randomized parameters- whhchh in line with the original paper experimental setting. Importantly, notice that while higher number of evaluations generally leads to better approximation, this translates to much higher number of environment interactions w.r.t. the baselines. In the case of AutoDR, we also have the flexibility to choose "low" return threshold value t,, used to backtrack the uniform distribution when average boundary performance falls below ty-the the analogous mechanism the we induce with our backup optimization problem in (6). In contrast to implementation, however, this required an additional parameter to be selected for AutoDR: we then set t, to be half of the return threshold Jnn throughout our experiments, done in the original work (Akkaya al., 2019).
0.5. As side note, this effeect is however harder to notice for distributions that
14

---

Published as a conference paper at ICLR 2024
In-distribution SR
00000
Entropy
5
0.4
-0000
0.5
Global
0005
0.2
0.0
0.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
(a) Hopper
In-distribution SR
1.0
10
Entropy
Global
0.5
0.005
0.2
0.01
-10
0.05
0.5
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
(b) Walker2D
In-distribution SR
0.4
0,005
Global SR
Entropy
0.2
0.05
-5
Timesteps
Timesteps
Timesteps 1e7
(c) HalfCheetah
Figure 8: Sensitivity analysis of the trust region size on Hopper (top), Walker2D (middle), and HalfCheetah (bottom) environments, with a=0.5. DORAEMON able consistently maintain the in-distribution success rate even for larger trust region sizes, SR stands for success rate.
Torso
Thigh
Log
Thigh,
Timesteps-SM
Timesteps=15M
0.25
0.50
0.25
0.50
0.75
0.5
1.0
0.1
0.2
0.3
0.5
1.0
1.5
2
Torsosize
Thighsize
Legsize
Footsize
Friction Right
Riction_
Figure 9: Comparison of Beta distributions at different times during training with DORAEMON on the Walker2D environment with 0.05 (see Fig. 8b for comparison). The green Beta distribution obtained after 5M timesteps (entropy=9.93) is adjusted along the process to maintain the desired in-distribution success rate, and eventually converges to the orange distribution after 15M training timesteps 464)..
Overall, we select the best configuration found terms of global success rate. Refer to our public code implementation at https: gabrielet iboni github. 1o/ doraemon/ for the full reproducibility of our experimental evaluation.
A.3 SUCCESS RATE VS. ENTROPY TRADE-OFF
This section augments and summarizes the homonymous analysis the main experimental section (Sec. 5.2) with further experiments and considerations on the trade-off between the hyperparameter a and its effect on policy training and generalization. We illustrate the overall results of this analysis in Fig. 10. To interpret the figures effeetively, shall look the evolution of in-distribttion success rate w.r.t. the growth entropy, and ultimately how these policies generalize by means of global success rate. Intuitively, higher value of ==0.9 constrains DORAEMON be conservative with
15

---

Published as a conference paper at ICLR 2024
In-distribution SR
a5
Global SR
Entropy
0.5
55
0.2
a=09
0.0
0.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
(a) Hopper
In-distribution SR
1.0
10
0.4
Entropy
Global
0.5
0.2
-10
0.5
1.0
1.5
0.0
0.5
1.0
0.5
1.0
1.5
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
(b) Walker2D
In-distribution SR
1.0
0.3
a a05
Global SR
Entropy
n 0
0.5
0.2
a=0
-5
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Timesteps
1e7
Timesteps
1e7
Timesteps
1e7
(c) HalfCheetah
Figure 10: Analysis on the impact of the hyperparameter a on Hopper (top). Walker2D (middle)) and HalfCheetah (bottom) environments. The desired in-distribution success a= 0.75, 0.9) is tracked effectively by DORAEMON.
the increase of entropy, leading to policies that likely overfit on in-distribution dynamics and poorly generalize. On the contrary, less restrictive value a=0.5-i.e. allowing the policy encounter more failure cases on diverse dynamics parameters- proved to be more effective, a result that we attribute the higher entropy value reached at convergence. Overall, note how DORAEMON is able to consistently maintain the desired in-distribution success rate regardless of the chosen value. This capability can be mostly attributed the backup optimization problem in (6), as demonstrated by our ablation study in Fig. 13.
A.4 BEYOND BETA DISTRIBUTIONS
DORAEMON's optimization problem as defined in (4) is not limited partieular family of para- metric distributions. While bounded-support distributions such as Beta's Beta's-are are particularly well suited for the Domain Randomization setting, DORAEMON may work with any family whose en- tropy (for the objective function) and KL-divergence (for the trust region constraint) may be conve- niently computed. We therefore compare our method with an implementation that parameterizes the DR distribution as independent Multivariate Gaussian (i.e. unbounded). Although not generally necessary, we still resample dynamics parameters training time that fall outside of the search space boundaries defined in Tab. 2 in order to fairly compare the results w.r.t. the Beta family. Overall, DORAEMON simply maximizes the entropy of the Gaussian distribution, which is allowed to grow in variance indefinitely. Note how this is in contrast with the KL-divergence term of LSDR's ob- jective, which considers the M- M-projection formulation. As a result, LSDR optimization problem is analogous maximum likelihood objective, rather than maximum entropy assuming tar- get uniforms. We illustrate the comparison of DORAEMON implemented with Beta vs. Gaussian distributions in Fig. 11. The two implementations generally exhibit similar behavior, but may still
"The Truncated Gaussian implementation of the Seipy library is used for this purpose, as naive consecutive resampling may hecome increasingly more time consuming the entropy of the Gaussian increases.
16

---

Published as a conference paper at ICLR 2024
Hopper (7D)
Walker2D (13D)
HalfCheetah (8D)
Global Success Rate
0.4
63
0.3
0.1
30.0
Entropy
10.0
5.0
0.0
0.9
Timestops
Timesteos
107
Timestops
DORAEMON (Gaussian)
DORAEMON (Beta)
Figure 11: Global success rate computed on the maximum-entrop uniform distribution (top) and entropy of the current training DR distribution (bottom), for distributions parametrized inde- pendent Beta vs. Gaussian. The number of randomized dynamics parameters per environment is reported in parenthesis.
2.5
5.0
2.5
5.0 Thigh Left
Torso
Thigh
Left
Bet (7
Gaussian (7.5M)
Gaussian (15M)
Torsosize
Thighsize
Legeize
Footsize
Friction_P Right
Priction_ JLff
Figure 12: Comparison of Beta vs. Gaussian distributions different times during training with DORAEMON on the Walker2D environment (training timesteps in parenthesis). When considering fixed feasible boundaries for DR parameters, Beta distribution proves to be more suitable as it can inherently converge to uniform distribution over the defined boundaries.
be chosen according the specific task hand. In particular, depict example of distributions encountered during the training process in Fig. 12, respectively for the two parametric families. De- spite showing a similar global success rate, is likely that the Gaussian parametrization may bias the learner around the mean of the distribution, resulting in poorer generalization towards the corner of the boundaries considered.
B CONNECTIONS WITH CURRICULUM LEARNING
The idea of training reinforcement learning agents under moving MDPs isn't confined to the do- main randomization setting sim-to-real problems, and, e.g., has been concurrently explored in the field of curriculum learning (CL.. In particular for the latter setting, self-paced algorithms (Klink et al., 2020b;a) are formulated as constrained optimization problems that automatically increase the difficulty of the task-i.e. they find curriculum over the task allowwig the agent progres- sively learn complex target tasks that would otherwise be unfeasible to solve from seratch. Likewise, Huang et al. (2022) and Klink et al. (2022) recently proposed framing CL as interpolations between a source (initial) and target task distribution by formulating it as an optimal transport problem. Interestingly, the curriculum learning problem shares similarities with the DR setting, as different simulation parameters may be analogously considered different tasks mesks-se Sec. 4.1 of the recent survey by Muratore et al. (2022b). This becomes particularly clear when considering curricula over MDPs that share the same state and action space, and only differ by transition dynamics. However, the DR problem importantly departs in that (1) no target task distribution is given, and that (2) the current underlying task is unknown to the agent at test time came-real-wee weal-world dynamics parameters unknown. turn, DR asks for agents that must not forget previously learned tasks, and that exhibit generalizable behaviors among latent dynamics parameters.
17

---

Published as a conference paper at ICLR 2024
Despite the different problem formulations and assumptions, DORAEMON draws inspiration from the curriculum learning literature and follows the intuition that more complex tasks may be obtained with increased diversity of dynamics parameters at training time. As result. DORAEMON is able to cope with the lack of assumptions in the sim-to-real problem w.r.t. a standard CL. formulation, while leveraging the flexibility of the self-paced optimization problem for automatically adjusting the DR distribution.
In this section, we therefore summarize the connections between CL and DR settings, and conse- quently those between self-paced and DORAEMON algorithms. As we break the discussion down into paragraphs for the sake of clarity, we will refer to Fig. 13 to support the stated claims with empirical results.
Unknown target distribution. The CL. problem assumes that target task distribution is given, indicating the desired tasks that we are ultimately interested in solving. This fundamentally departs from the DR problem setting, where simulator parameters are randomized to seek generalization to unknown wolld task distributions. To cope with this, LSDR proposed considering target uni- form distributions that should be designed to be as wide as possible, making it hyperparameter of the algorithm. In contrast, AutoDR removed the dependence on target distributions, and simply attempts to converge to uniform distribution that approaches infinite support. Overall, the un- known target distribution therefore makes CL algorithms impractical to be applied to the DR setting directly. A common ground between the two settings may still be identified when restricting the problem to bounded domains of dynamics parameters, and assuming uninformative-is. uniform- target distributions for CL. In such case, both problems would effectively aim to gradually solve all tasks within the predefined boundaries, with no bias towards any region in particular. As this re- striction is made throughout our experimental analysis to allow for comparison with (SDR-whidd requires target distribution to be defined in the first place- we can directly compare the results of DORAEMON with, e.g., SPDL (Klink et al., 2020b). Nevertheless, non-trivial adaptations of SPDL must still be made to cope with non-observability of dynamics parameters, clarified in the following paragraph.
Contextual MDPs vs. Latent MDPs. The CL problem assumes to work with contextual MDPs (Modi et al., 2018), where an observable context variable-nann namely the underlying task parameters-augee augments the environment and determines the underlying transition dynamics. This assumption fundamentally differs from a sim-to-real problem, where dynamics parameters are un- known to the agent at inference time. In turn, solving a sim-to-real problem requires finding an opti- mal context-agnostic policy of the form 7 (a|s), in contrast the conventional context-conditioned policies learned in CL TT (aass E). Domain Randomization may indeed be conveniently formulated by introducing Latent MDPs (LMDPs), as recently proposed by the theoretical study in (Chen al., 2022). It's worth noting that context-conditiones policies may still be learned as simulators offer full information work has, e.g., attempted to work around this by inferring dynamics parameters at test time (Yu et al., 2018). However, the more popular approach is to leverage memory-basee information on the underlying transition dynamics, and therefore implicitly on the latent parameters (see Sec. 9 in Akkaya et al. (2019)). Furthermore, tailored implementation tricks to the sim-to-real problem setting such asymmetric actor-critic have been introduced to additionally ease the train- ing process under partial observability, namely by providing full information to the learned critic networks which are nevertheless not queried test time (Akkaya et al., 2019). As reported in our method Section 4, such implementation details are already included in DORAEMON, but may be easily overlooked when simply comparing our optimization problem therefore report a thorough analysis on the effect of each of these components in Fig. 13, as we show a detailed ablation of DORAEMON from the perspective of SPDL optimization problem. Our empirical evaluation demonstrates the contribution of each added component, highlighting the im- portance of accurately dealing with the different assumptions induced by the DR problem setting. In particular, we notice that history-based policies (SPDL- History) may times even hinder pol- icy training w.r.t. standard Markovian policies (SPDL), and only become particularly effective when combined with asymmetric actor-critic (SPDL- + History + Asymmetric). Finally, we report SPDL Oracle baseline which reflects the performance of dynamics-come conditioned policies (a]s, {) as in a contextual MDP setting, despite being of little help solve our problem. Interestingly enough, such
would end up being impractical for deployment in the real world. Previous
policies to gain
self-paced methods. We
18

---

Published as a conference paper at ICLR 2024
policies sometimes appear to poorly generalize to out-of distridiitiiittionn dynamics (see global success rate curves), likely due to overfitting to the dynamics observed at training time.
I-projection vs. M-projection. Non-trivial connections among DORAEMON, LSDR, Au and the self-paced methods can also be observed when solely focusing on their optimization prob- lems. As all these methods attempt to either minimize KL-divergence objective or maximize entropy (implicitly for AutoDR and explicitly for DORAEMON), we attempt to study their differ- ences and similarities from a conceptual and technical viewpoint. Interestingly, self-paced methods rely on the I-projection objective, which effectively becomes equivalent to a maximum entropy ob- jective when considering target uniform distributions. On the other hand, however, the I-projection formulation limits the choice of parametric distributions to be of bounded support that is fully in- cluded within the target. This limitation led us to design novel implementation of SPDL with Beta distributions for the analysis in Fig. 13, departing from the original paper implementation. In contrast, LSDR proposes an M-projection objective for the DR setting, likely to avoid restricting the optimization parameters to bounded distributions when considering uniform targets. By doing so, however, the optimization fundamentally switches to a maximum likelihood objective. providing an easy explanation as to why Gaussian distributions by LSDR do not converge to high entropy values in Fig. 2. Overall, DORAEMON does not rely on KL-divergence objective, hence both (1) does not need to specify target uniform and (2) can be implemented with any parametric family (see Sec. A.4 for DORAEMON with Gaussian distributions). In turn, DORAEMON may be considered a hybrid formulation that bridges self-paced methods for the DR setting in sim-to-real, conceptually achieving the best of both worlds. Finally, notice how AutoDR also drops the dependency on the target distribution, but crucially comes with number of inherent limitations: it's confined to uni- form DR distributions only, may only adjust the distribution curriculum with fixed A steps (limited flexibility w.r.t. self-paced methods), and may only update one dimension at time (inefficient use of training samples).
Effect of the backup optimization problem. Lastly, we discuss the contribution of the backup optimization problem defined in Eq. (6), the final distinctive technical novelty in DORAEMON's formulation w.r.t. self-paced methods. Despite the clean theoretical formulation, the practical imple- mentation of SPDL must consider approximations of the performance constraint, which could lead to unfeasible regions of the optimization landscape at the following iteration-namely experience an in-distribution success rate lower than a. This aspect has not been explicitly addressed in prior self-paced works, which simply keep training the current policy when this occurs. However, pect this problem to be particularly important to overcome in the DR setting, as it's harder to assume keep training. Conversely, it's likely that problematic
sus-
that policy performance would increase dynamics parameters may even catastrophically affect the learning process. To this end, we propose backtracking the entropy distribution by directly looking for maximum in-distribution success rate within the current trust region. Our empirical analysis in Fig. 13 shows impressive tracking of the a hyperparameter when the backup optimization problem is included (DORAEMON), which holds for different environments, different trust region sizes, and different a values (also refer to Fig. 10). In- deed, notice that all other ablations struggle to keep a feasible performance constraint as the entropy of the distribution grows (see in-distribution SR curves entering the red-shaded, unfeasible area). As a side note, we noticed that such addition would not necessarily translate to better global success rates. After further investigation (see Fig. 9). we conclude that, as DORAEMON gradually attempts to find easier regions of the dynamics space through backtracking, policies would occasionally suf- fer from catastrophic forgetting and struggle retain good performance over previously experienced dynamics. Future work may investigate more sophisticated ways to track the in-distribution success rate, while ensuring that critically useful skills for generalization shall not be forgotten. While out of the scope of this work, multimodal distributions may also be leveraged to isolate problematic regions in the dynamics space, yet maintain generalizability over feasible dynamics.
C PANDAPUSH TASK
The details of our experimental setting for the PandaPush environment are reported here.
We consider the 7-DoF Franka Panda robotic arm mounted on a table and tasked with pushing a perfectly squared box of size 10cm per side. We analyze the setting where the center of mass of
19

---

Published as a conference paper at ICLR 2024
13.5
4.5
0.0
Gote 38
2/
0.2
-2.
025
252
0.7
1.2
1,50
0.50
523
220
0.73
1,00
123
1,50
5.23
0.0
5-75
1ED
2.25 1S
(a) Hopper (e=0.005)
1.0
160
Grans SR
0.4
Timesleps
Timedops
107
Thmesdape
(b) Hopper (c=0.01)
12.5
0.0
8.6
5/
0.0
0.5
255
0.75
1,00
1.25
1:50
-25
0.30
5.5
350
0.75
1,03
125
1.52
5.0
535
0.50
5.75
140 1.25 15
(c) Hopper (e=0.05)
1.0
2.5
(-480 S3t
8..8
E-1-00
C3220 12R
0.4
3
0.2
1,30
230 0.75
Timeslopa
(d) Walker2D (c=0.005)
3.0
5.4
6.6
Grabal 521
0.4
0.0
152
100
0.00 5.3
250 0.73 200 125
1,32
150
Timeslpps
limeedps
(c) Walker2D (c=0.01)
0.8
0.4
E-00-0
5
502
01
Timmllps
187
Timelags
167
Timminga
SPDI
SPDL History Asymmetric
SPOL (Oracle)
SPDL
DORAEMON
Walker2D (c=0.05)
Figure 13: Ablation of DORAEMON from the perspective of self-paced methods in curriculum learning (SPDL). The results demonstrate that all components of DORAEMON are vital to maintain desired in-distribution success rate a, while achieving high generalization. The red-shaded area indicates the region where the performance constraint is violated, i.c. in-distribution success rate lower than a. SR stands for success rate.
20

---

Published as a conference paper at ICLR 2024
the box is unknown; hence, pushing the box will result in unpredictable behavior. Notably, policies are conditioned on a history of state-action pairs, which may be used as information to compensate aim to learn versatile policies that can solve the task
for the unknown parameter. Therefore, for a varying center of mass through robust (to the reality gap) and adaptive (to dynamics changes) behavior. For the sake of clarity, we break down the task implementation into the following points.
State space and initialization. The underlying MDP considers a 20-dim state space, which in- cludes joint positions (7D), joint velocities (7D). box position on the table (2D), box rotation angle 2 about the z-axis (2D encoded as sin( czs( and goal position (2D). We initialize the robotic arm as shown in Fig. 15, such that it is already sufficiently close to the box. The initial location of the box is randomized with a uniform noise of +1cm, and we further slightly randomize the box height by 10cm = 0.005em to regularize the learned behavior against specification gaming" gaming²-iee. avoid cases where the agent would undesirably rely on the box corners edges to maximize the return. Finally, we apply a mild Gaussian noise with o = 0.0011 to the joint velocity observations-an informed guess based on the noise estimated on the real system.
Action space and controller. The agent interacts with the environment by sending joint acceler- ation commands, in normalized space [-1, 1] for each joint. We then denormalize the commands by multiplying for the predefined joint acceleration namely, [6, 3, 4, 5, 6, 8, and inte- grate the commanded acceleration forward in time to get a desired trajectory to follow, given the current joint position and velocity. We then track such trajectory through a PD controller and a feed- forward term- form-wichh is trivially computed by multiplying the inertia matrix by the commanded acceleration. Overall, we query the policy at frequency of 50Hz, z, and follow the resulting low- level toaeetrajy at 1000
Reward function and success notion. We reward the agent with the current (negative) squared distance of the box from the goal location, to drive a pushing motion. We further engineer the reward function to encourage adjustments in close proximity of the goal where the squared distance alone would be considerably flat. Therefore, we finally design a reward function of the form - dd² log(d' +0.01) with d being the distance from the target. The obtained reward signal is then rescaled such that it starts at a value of zero, given the initial distance to the target. Finally, we additionally penalize the magnitude of the accelerations commanded by the agent to encourage smooth and safe execution of the policies. For running DORAEMON and AutoDR, we then define a notion of success based on the box distance directly. Indeed, note that a desired lower bound on the return may be cumbersome to define with shaped reward formulations. We therefore denote success for trajectories where the final location of the box falls within 3em distance to the goal.
Domain Randomization. We randomize total of 17 dynamics parameters to cross the reality gap in the PandaPush environment. As for the sim2sim experiments, we define bounded search space over the parameters to keep notion of maximum entropy distribution, used for assessing generalizability across the baselines. The boundaries of each randomized parameter are reported in Tab. 3. In particular, we designed them with minimal prior knowledge, and no automatic system identification. Rather, our goal is to train on maximally wide distributions, such that generalization can be achieved even without careful estimation of the real parameters. Notably, we set the starting distribution of LSDR, AutoDR, and DORAEMON to be centered around the values of 0.1 for all joint damping and friction parameters, which is considerable lower than the center of the search space. We observe that higher values of these parameters yield to a higher task difficulty, especially the early stages of training-i.e. higher friction values make harder for the agent start col- lecting rewards from pushing the box. Interestingly, Fixed-DR pohicies-which are trained over a uniform distribution that spans the whole search space were unable to learn despite including both low and high friction values.
Evaluation setting. We found the PandaPush environment to be more challenging to learn from pure RL perspective, leading to more unstable training curves w.r.t. the MuJoCo testbed envi- ronments. However, good performing policies were still retained by iteratively evaluating the per- formance at training time over the maximum entropy distribution, and keeping track of the best-
RRadd more on specification gaming at Blog post https: /wwww/wwww.co/ deepmi nd. com / og/ speci fat on- -n-ganing- toagtthh-flii wtw-f-ll--ssdd----aaaa-a wfaidd----a--igeen--e
21

---

Published as a conference paper at ICLR 2024
Parameters
Boundaries
Box mass (ID)
0.2, 0.6]
Surface Friction (ID)
[0.03, 0.5]
PandaPush (17D)
Joint damping (7D)
[0.025, 2.5]
Joint friction (7D)
0.00, 3|
Center of mass (1D)
-0.05, 0.05
Table 3: Search space boundaries for the 17 dynamics parameters randomized in the PandaPush environment. The center of mass boundaries refer to the shift in centimeters from the geometrical center, along the perpendieular axis of the pushing direction.
eis
Easy CoM, low friction
(b) Hard CoM, low friction
(c) Hard CoM, high friction
Figure 14:
Illustration of the three configuration settings used for evaluating PandaPush policies in the real world. We adjust the center of mass (CoM) by replacing the steel bolts inside the box, and we increase the sliding friction by changing the table top (c).
performing policies by means of highest global success rate. Finally, we compare each baseline by evaluating the average return and success rate of the best-performing policy across 10000 episodes in simulation (1000 episodes for 10 seeds), and 30 rollouts in the real world (3 rollouts for 10 seeds). In particular, we carry out the tests on the real setup by varying the surface friction and center of mass of the box, resulting in 3 test configuration for which we collect one rollout each (see Fig. 14).
Supplementary results and discussion. We complete the experimental evaluation the Panda- Push task by reporting the training curves for all benchmark methods in Fig. 17, averaged over 10 seeds. The training curves reflect the final results in Tab. of the main manuscript, as we observe much higher performance of DORAEMON's policies. We attribute this to the capability of our optimization problem to efficiently guide the distribution towards high-entropy regions, while main- taining a steady increase of the global success rate. Interestingly, the performances seem to degrade significantly when the maximum-entropy distribution approached (see DORAEMON curves in Fig. 17 timesteps 3.5M). In this scenario. DORAEMON finds it hard quickly recover from such occurence, opening avenue for future work directions to mitigate such behavior. We then in- vestigate the moving Beta distribution of DORAEMON during training. and illustrate the history of encountered distributions in Fig. 16 for subset of 5 parameters. We find that, while our method converges to a near maximun-entropy uniform (yellow curve), best generalization is achieved for an intermediate entropy distribution (green curve). This is analogous to the case of the Walker2D and Swimmer tasks in Sec. 5.2. We remind that, regardless the drop in performance due to increased diversity of the dynamics parameters, for all baselines we keep track of the best- pessopeeforning policy while training by means of global success rate, and use this one for real- world evaluation.
22

---

Published as a conference paper at ICLR 2024
Figure 15: Three random configurations of the PandaPush setup simulation: center of mass (CoM) of the box is randomized along the axis perpendicular to the pushing direction of the box. This makes it a particularly interesting setup for studying the adaptivity of the learned history-based policies. The green dot highlights the goal location.
Max-Entropy
Converged
Best
Box mass
Friction
CoM
J. Damping [0]
Friction [0]
Figure 16: DORAEMON's Beta moving distribution across five representative dynamics param- eters of the PandaPush task (blue curves more opaque for more recent iterations). The final converged distribution (yellow) and the best distribution by means global success rate (green) are also reported. "CoM", "J. Damping [0]", and "J. Friction [0]" abbreviate Center of Mass, first-joint damping, and first-joint friction respectively.
0.5
No-DR
Fixed-DR
Global Success Rate
LSDR
-50
AutoDR
Entropy
0.3
DORAEMON
0.2
Fixed-DR
-150
LSDR
0.1
AutoDR
DORAEMON
-20
1
3
1
2
Timesteps
1e5
Timesteps
1e6
Figure 17: PandaPush sim-to-sim results: global success rate computed on the maximum-entropy uniform distribution (right) and entropy of the current training DR distribution (left).
23


Reviews:
[{"review_id": "9uLcR6DHqM", "reviewer": "Reviewer_BXPC", "summary": "This paper introduces DORAEMON, a novel domain randomization technique in reinforcement learning, designed to enhance policy generalization across varied environment dynamics. DORAEMON strategically increases the entropy of training distributions, conditioned on achieving a probability of success threshold and by ensuring updates to the entropy are constrained. The aim is to balance both the entropy of the dynamics parameters distribution and task proficiency. Empirical tests across OpenAI Gym benchmarks and on a real-world robotic task highlight DORAEMON's superior adaptability to diverse dynamic settings compared to conventional domain randomization approaches and also demonstrate how the success rate threshold and success definition (i.e., lower bound return threshold) impact performance. This work convincingly demonstrates the potential benefit of applying DORAEMON to systems where sim-to-real policy transfer is important.", "questions": "How sensitive is DORAEMON to its hyperparameters (e.g., trust region size, trajectories per distribution update, definition of success, etc.), and what process was followed to select them? Did you perform a sensitivity analysis?\n\nAre there potential negative impacts of DORAEMON that should be discussed, especially regarding its application to real-world systems? While it is beyond the scope of this paper, have you considered the safety of the sim-to-real policy in the real-world robotics task? How does it compare to previous domain randomization methods?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "Originality\nThe paper introduces a novel and innovative approach to domain randomization. This method is an advancement in the field of RL, not only in the scope of domain randomization but potentially in areas of research outside of domain randomization (e.g., meta-RL). The method's innovation stems from its entropy maximization technique that enables a policy to generalize across a broader range of dynamics, while ensuring that the entropy of the dynamics parameter distribution grows in a manner that does not compromise the policy's probability of success. \n\nQuality\nThe proposed algorithm, DORAEMON, has been tested in OpenAI Gym benchmarks and in a sim-to-real robotics task, generally demonstrating its superior generalization in comparison to existing domain randomization techniques. \n\nClarity\nThe paper has a clear definition of success and the presentation of the results are well-structured. The mathematics foundations of the paper are clear and sound. The figures and empirical results support the authors claims of the superiority of their method in comparison to traditional domain randomization methods.\n\nSignificance\nThis paper is significant to the development of autonomous systems and has the potential for real-world application in industry.\n\nSummary\nDORAEMON stands out as an original, high-quality research work with significant implications for both theory and application in reinforcement learning and robotics.", "weaknesses": "The paper presents empirical tests across OpenAI Gym benchmarks and a real-world task. However, there may be a need for more diverse environmental tests to fully understand the limits of DORAEMON's generalization ability. For instance, testing in environments with higher-dimensional state spaces or more complex dynamics could provide a more comprehensive picture of the algorithm's robustness.\n\nThe research presented is certainly complex and holds significant value to the field. However, some sections of the text may benefit from further clarification to enhance the paper's accessibility to a broader audience. In particular, the density of technical terms and concepts could be balanced with more detailed explanations or simplified language. This could potentially include adding definitions, or providing more background information for non-expert readers. Such revisions would likely make the paper's contributions even more impactful and ensure that a wider range of readers can fully grasp the innovative work you have presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t5uoWHEyX5", "reviewer": "Reviewer_eC78", "summary": "Domain Randomization (DR) is a common technique used to reduce the gap between simulations and reality in Reinforcement Learning (RL), which involves changing dynamic parameters in simulations. The effectiveness of DR, however, largely depends on the chosen sampling distribution for these parameters. Too much variation can regularize an agent's actions but may also result in overly conservative strategies if the parameters are randomized too much. This paper introduces a new method for enhancing sim-to-real transfer, dubbed DOmain RAndomization via Entropy MaximizatiON (DORAEMON). DORAEMON is a constrained optimization framework that aims to maximize the entropy of the training distribution while also maintaining the agent's ability to generalize. It accomplishes this by incrementally expanding the range of dynamic parameters used for training, provided that the current policy maintains a high likelihood of success. Experiments show that DORAEMON outperforms several DR benchmarks in terms of generalization and showcase application in a robotics manipulation task with previously unseen real-world dynamics.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The paper is well-written and easy to follow. The authors also conducted real-world robotics experiments.", "weaknesses": "* Limited technical novelty. The formulation (Eq. 4) is very similar to that of SPRL[1], SPDL[2], CURROT[3], and GRADIENT[4]. Setting the target distribution to be uninformative -- uniform distribution could transform their objectives into something very similar to Eq.4, and they do not necessarily converge to the final target distribution.\n* Beta distribution is often not a reasonable choice. It cannot handle multi-modal distribution, while many existing works can handle arbitrary empirical distributions [1,2,3,4].\n* Given the similarity to the existing work as discussed in the first point, the authors should compare DORAEMON to them.\n* Missing related work:\n    * Klink, Pascal, et al. \"Curriculum reinforcement learning via constrained optimal transport.\" International Conference on Machine Learning. PMLR, 2022.\n    * Huang, Peide, et al. \"Curriculum reinforcement learning using optimal transport via gradual domain adaptation.\" Advances in Neural Information Processing Systems 35 (2022): 10656-10670.\n    * Cho, Daesol, Seungjae Lee, and H. Jin Kim. \"Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation.\" arXiv preprint arXiv:2301.11741 (2023).\n\nRef:\n[1] Klink, Pascal, et al. \"Self-paced contextual reinforcement learning.\" Conference on Robot Learning. PMLR, 2020.\n\n[2] Klink, Pascal, et al. \"Self-paced deep reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 9216-9227.\n\n[3] Klink, Pascal, et al. \"Curriculum reinforcement learning via constrained optimal transport.\" International Conference on Machine Learning. PMLR, 2022.\n\n[4] Huang, Peide, et al. \"Curriculum reinforcement learning using optimal transport via gradual domain adaptation.\" Advances in Neural Information Processing Systems 35 (2022): 10656-10670.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0orWguPFJM", "reviewer": "Reviewer_QW63", "summary": "The paper introduces DORAEMON, which revisits domain randomization from the perspective of entropy maximization. Specifically, instead of maximizing the expected total return over the distribution of the dynamics parameters, this paper proposes a constrained optimization problem that directly maximizes the entropy of the dynamics distribution subject to a constraint on the success probability.  Based on this formulation, this paper proceeds to offer an algorithmic implementation that decouples this maximization into two subroutines: (1) Update the policy by any off-the-shelf RL algorithm under the current dynamics parameter; (2) Under the current policy, update the dynamics parameter to improve the entropy with the help of a KL-based trust region. Accordingly, a toy experiment is provided to demonstrate the dynamics distribution that DORAEMON converges to. The proposed algorithm is then evaluated on both sim-to-sim (MuJoCo) and sim-to-real tasks (PandaPush) against multiple baseline DR methods.", "questions": "Detailed comments/questions\n\n- In practice, is it computationally easy to optimize (4)? The constrained problem does not seem to be convex (even under Beta distribution)?\n\n- Could the authors specify the alpha values for Figure 2 and Figure 11? When the entropy matches the max entropy, the global success rate aligns with the local success rate. If the alpha is set to 0.5, why does the global success rate drop below 0.5 when entropy is at its maximum?\n\n- How to design the entropy jointly for multiple dynamics parameters? (For example, simply taking the product of multiple univariate distributions like AutoDR?)\n\n- In Section 3: The notation of reward function shall be consistent (mathcal or not)\n\n------------------ Post-rebuttal ------------------\n\nI would like to thank the authors for the detailed response. Most of my questions have been addressed, especially the ablation study. That being said, regarding the comparison of DORAEMON and AutoDR, while I can understand authors' response on their differences, these differences still appear more like subtle implementation choices, and this keeps me from giving a higher rating.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method introduced in this paper is quite intuitive and reasonable in concept and avoids some inherent issues of DR. Specifically, as the standard DR requires a pre-configured fixed prior distribution over the support of the environment parameter (which would require some prior domain knowledge), the proposed DORAEMON framework learns to maximize the entropy of dynamics distribution and hence naturally obviates this issue. (That said, in the meantime, the threshold needed for defining a successful trajectory also requires some domain knowledge, but probably a bit less)\n- The proposed algorithm is evaluated in a variety of domains (including both sim-to-sim and sim-to-real scenarios), and the empirical results demonstrate quite promising performance of the DORAEMON framework (in terms of success rate).  \n- The paper is well-written and very easy to follow, with justification and explanation whenever needed in most places.", "weaknesses": "Overall I could appreciate the proposed reformulation of DR, but there are some concerns regarding the algorithm:\n\n- DORAEMON appears to be conceptually very similar to the AutoDR, or ADR in the original paper (Akkaya et al., 2019). They both define some custom indicators of success and iteratively increase the entropy of the dynamics distribution. With that said, DORAEMON appears to be yet a somewhat different implementation of the idea highlighted by ADR.\n\n- Based on the above, while the two approaches arise from similar ideas, DORAEMON appears to have a better success rate across almost all environments. It is not immediately clear whether the performance improvement comes from which specific part of the design or it is just a matter of different choices of hyperparameters. While there is a one-sentence discussion on the authors’ conjecture in Section 5.2 (about the potential data inefficiency), it is expected to have a deeper dive into the root cause of this performance difference.\n\n- The successful rate for certain tasks, e.g., Walker in Figure 2 and PandaPush in Figure 11, decline after reaching maximum entropy. However, the algorithm does not dynamically reduce the entropy in response to a decrease in the success rate, which might be necessary for maintaining performance consistency. This appears not consistent with the objective in (4). As the discussion in Section 5.2 does not fully address this phenomenon, more explanation would be needed.\n\n- Another concern lies in the constraint based on the success rate. Specifically, the use of success rate largely ignores the effect of the poor trajectories, which could be arbitrarily poor and degrade the robustness of the learned policy. By contrast, in the standard DR, the objective is to consider the expected total return over all the possible trajectories. As the experimental results reported in the paper all focus on the “success rate”, the robustness issue is thereby largely neglected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MzYTNAFS1m", "reviewer": "Reviewer_W7po", "summary": "The paper introduces domain randomization via entropy maximization, a constrained optimization framework that directly maximizes the entropy of the training distribution while retaining generalization capabilities. The authors empirically evaluate their method in several simulated control environments. Additionally, they successfully showcase zero-shot transfer in a robotic manipulation task under unknown real-world parameters, emphasizing its practical applicability.", "questions": "Can this domain randomization method potentially be applied to object randomness? For example, in ManiSkill environments, some tasks include variations in objects. Can we use maximum entropy to gradually learn from these different objects?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework utilizes entropy maximization to gradually enlarge the randomization range, which is sound.\n2. Experiments compared to control results demonstrate the effectiveness of the proposed methods.\n3. The robot manipulation experiments indicate that the proposed method has the potential for use in real-world tasks.", "weaknesses": "1. Only Beta distributions are considered in the experiments. It is encouraged to add more distribution types to the experiments.\n2. Additional visualizations need to be included to illustrate the trade-off between performance and entropy. For example, in Figure 2, the performance of Walker2D and Swimmer decreases when the entropy increases. It is important to explore the relationship between the randomized variable and performance, and to determine the range within which performance decreases.\n3. Adding more real-world experiments is encouraged. PushCube is a relatively easy task in robot manipulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OaVIYVqrjy", "reviewer": "Reviewer_TxSB", "summary": "The paper proposes a novel approach to addressing the \"sim-to-real transfer\" challenge in reinforcement learning. The presented approach, DORAEMON, aims to maximize the diversity of dynamics parameters during training by incrementally increasing randomness while ensuring a sufficiently high success probability of the current policy. This results in highly adaptable and generalizable policies that perform well across a wide range of dynamics parameters.", "questions": "1.\tThe proposed method calls the RL algorithm to update the policy for every dynamics parameters sampling, which might lead to an inefficient algorithm. Maybe the embedded RL algorithm only need to return a relatively approximate solution? Does this works for DORAEMON. This is expected to be made clear. \n\n2.\tThe authors adopt univariate beta distributions for \\nu_{\\phi}, which might simplify problem (6). But for more general distributions, solving (6) might be challenging. Maybe this should be further investigated. Or is it the case that some commonly used distributions, e.g. Gaussian, are effective and meanwhile make (6) tractable. Or some variational methods can be adopted.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A straightforward method to explore the environment dynamics parameters is proposed for RL algorithms to enhance their generalization. \n\nThe method is simple, and its effectiveness is demonstrated in the toy example and the experiments.", "weaknesses": "The method is heuristic and lacks theoretical analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h2e5RMRKo5", "reviewer": "Reviewer_7kiW", "summary": "This paper presents a new domain randomization method that tries to overcome the performance and generalization gap by maximizing the entropy of the distribution of dynamic parameters while retaining certain success probability during training. The authors conduct simulation experiments as well as sim-to-real experiments.", "questions": "- In Table 1, the accuracy of the Fixed-DR approach is notably low, prompting the need for the authors to provide further explanations, if not overlooked, in order to clarify any potential factors contributing to this outcome.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The article presents its content in a clear and concise manner.\n- The method exhibits novelty and has been well formalized.", "weaknesses": "- The baselines utilized in this study appear to be somewhat outdated, which raises the question of whether more recent advancements in domain randomization have been considered. It is highly recommended that the authors explicitly address this concern by providing a specific clarification on the existence of any updated domain randomization approaches, and it would be beneficial for the authors to incorporate additional baseline experiments that encompass these newer methodologies.\n- While the method proposed in this study demonstrates a certain level of innovation, it does not appear to be exceptionally groundbreaking. Therefore, it is imperative to present more compelling experimental results. Conducting additional simulation experiments, as well as sim-to-real experiments, would be highly recommended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)