Variance Reduced Domain Randomization for Policy Gradient
Authors: Yuankun Jiang
Chenglin Li
Wenrui Dai
Junni Zou
Hongkai Xiong
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement learning
generalization
variance reduction

Abstract: By introducing randomness on environment parameters that fundamentally affect the dynamics, domain randomization (DR) imposes diversity to the policy trained by deep reinforcement learning, and thus improves its capability of generalization. The randomization of environments, however, introduces another source of variability for the estimate of policy gradients, in addition to the already high variance due to trajectory sampling. Therefore, with standard state-dependent baselines, the policy gradient methods may still suffer high variance, causing low sample efficiency during the training of DR. In this paper, we theoretically derive a bias-free and state/environment-dependent optimal baseline for DR, and analytically show its ability to achieve further variance reduction over the standard constant and state-dependent baselines for DR. We further propose a variance reduced domain randomization (VRDR) approach for policy gradient methods, to strike a tradeoff between the variance reduction and computational complexity in practice. By dividing the entire space of environments into some subspaces and estimating the state/subspace-dependent baseline, VRDR enjoys a theoretical guarantee of faster convergence than the state-dependent baseline. We conduct empirical evaluations on six robot control tasks with randomized dynamics. The results demonstrate that VRDR can consistently accelerate the convergence of policy training in all tasks, and achieve even higher rewards in some specific tasks.   

Under review as a conference paper at ICLR 2022
VARIANCE REDUCED DOMAIN RANDOMIZATION FOR POLICY GRADIENT
Anonymous authors Paper under double-blind review
ABSTRACT
By introducing randomness on environment parameters that fundamentally affect the dynamics, domain randomization (DR) imposes diversity to the policy trained by deep reinforcement learning, and thus improves its capability of generalization. The randomization of environments, however, introduces another source of vari- ability for the estimate of policy gradients, in addition to the already high variance due to trajectory sampling. Therefore, with standard state-dependent baselines, the policy gradient methods may still suffer high variance, causing low sample ef- ficiency during the training of DR. In this paper, theoretically derive bias-frce fres-free and state/environmenn-eo -deenndddddeedeent optimal baseline for DR, and analytically show its ability to achieve further variance reduction over the standard constant and state-dependent baselines for DR. We further propose a variance reduced domain randomization (VRDR) approach for policy gradient methods, strike a tradeoff between the variance reduction and computational complexity in practice. By di- viding the entire space of environments into some subspaces and estimating the htte//uubsace-depeede
baseline, VRDR enjoys a theoretical guarantee of faster convergence than the state-dependent baseline. We conduct empirical evaluations on six robot control tasks with randomized dynamics. The results demonstrate that VRDR can consistently accelerate the convergence of policy training in all tasks, and achieve even higher rewards in some specific tasks.
INTRODUCTION
Deep reinforcement learning (DRL) has achieved an impressive success on complex sequential decision-making tasks, such as playing video games top human-level (Ye al., 2020). contin- uous robot manipulation (Agarwal et al. 2021) and traffic signal control (Egeaa et al.) 2021). For an extensive exploration, DRL requires massive amount of samples train the policy, which usually done in simulator constructed for the task. Due to the lack of diversity, however, policy trained by DRL tends to overfit to a specific training environment (Cobbe al.] 2019), causing possibly severe performance degradation (i.c., reality gap) when the policy learned in simulator is transferred to the real deployment (Peng et al., 2018 Kang et al.) 2019). To close this gap, domain randomization (DR) has been proposed to randomize the environment parameters that may vary the dynamics and observation in simulator, imposing diversity on the trained policy (Tobin et al.) [017] Jiang et al. 2021). Recent success in robot control has shown that DR enables the trained policy to generalize in real deployment, even with presence of fundamental modeling errors in the simulator Muuratore al. 202 Xie et al. 2020p. DR has also demonstrated its capability of robustness to contend with extremely rare environments (Muratore et al. 2021).
Direct applications of DR on policy gradient methods can be thought as policy optimization on multiple randomly generated environments, which incurs additional variability of the observed data. Therefore, a critical challenge is the low sample efficiency due to the extremely high variance of the gradient estimator, where the variability not only stems from the gradient approximation of expected return from a batch of sampled trajectories (Greensmith et al. 2004), but is also imposed by the randomization of environment parameters. In standard DRL, commonly used method to reduce variance is to construct a bias-free and action- indiinnnndeeedeentt baseline that is subtracted from the expected return (Sutton & Barto 2018). For example, typical choice of bascline is the state value function of a policy. When directly applied to DR, as proposed by (Andrychowicz et al. (2020), it is equivalent to learning state value function that predicts the expected return over all possible

---

Under review as a conference paper at ICLR 2022
environments. Though remaining unbiased, such a state-dependent baseline may be poor choice in DR, since the additional variability of randomized environments is not taken into account.
In this paper, we aim to address the high variance issue of policy gradient methods for domain ran- domization, with a particular focus on reducing the additional variance imposed by randomization of environments. Our key insight is that the additional information on varying environments can be incorporated into the baseline to further reduce this variance. Theoretically, we derive the optimal suteeeeniioommee-ddeeeneedd
baseline, and demonstrate that it improves consistently the variance reduction over baselines that is constant or uses state information only. In order for the practical im- plementation strike the tradeoff between the variance reduction performance and computational baselines, we propose a variance re-
complexity for maintaining the state/cnvironmen--eepee duced domain randomization (VRDR) approach for policy gradient methods, which can improve the sample efficiency while maintaining reasonable number of baselines associated with set of specifically designed environment subspaces. Our main contributions can be summarized as follows.
baseline for DR. For the policy gradient in variety of environments that differ in dynamics, we derive theoretically an optimal baseline that depends on both the states and environments. We further quantify the variance reduction baseline over two common
Theoretically optimal state/anvironmen---eeenee
improvement achieved by the proposed state/enviromment-depeeee choices of baselines for DR, i.e., the constant and state-dependent baselines.
Criterion for constructing practical statessubspace-ddeeeneed baseline. Since the accurate baseline for each possible environment is infeasible during practical implementation of RL, we propose alternatively divide the entire space of environment parameters into a limited number of subspaces, and estimate instead the optimal baseline for every pair of state and environment subspace. We further show that the clustering of environments into subspaces should follow the policy's 's expected returns on these environments, which can guarantee an improvement of variance reduction over the state-dependent baseline.
estimation of htate/environmeen--deeeennnddd
Variance reduced domain randomization (VRDR) with empirical evaluation. To strike the tradeoff between the variance reduction performance and computational complexity for maintain- ing optimal baselines, we develop variance reduced domain randomization (VRDR) approach for policy gradient methods. Specifically, VRDR learns an acceptable number of baselines for each pair of state and environment subspace, where the environment subspaces are determined based on the above criterion. We then conduct experiments on six robot control tasks with their fundamental physical parameters randomized, demonstrating that VRDR can accelerate the con- vergence of policy training in DR settings, compared to the standard state state-dependent baselines. In some specific tasks, VRDR can even achieve a higher reward.
BACKGROUND
Notation. Under the standard reinforcement learning (RL) setting, the environment is modeled as a Markov decision process (MDP) defined by a tuple < S,A, Tp. &b >, where S and A de- note the state and action spaces, respectively. For the convenience of derivation, we assume that they are finite. Tp : S x A x S + 0, 1] is the environment transition model that is essen- tially determined by environment parameter p E P, with P denoting the space of environment parameters. In robot control, for example, environment parameter p can be a vector contain- ing the rolling friction of each joint and the mass of torso and arms. Throughout the rest of this paper, by environment we mean that the environment has dynamics determined by param- eter p. : S x A - E is the reward function. At each time step t, the agent observes its state St S and takes an action oz E A under the guidance of policy To(a, (sx) parameter- ized by 0. will then receive a reward r, +b(s,, at); while the environment shifts to the next state $t41 with probability (S5++1/se. az, p). The goal of standard RL is to search for policy that maximizes the expected discounted return (TT,P) = E, [[T]] over all possible trajectories
(St, ae Ft, St+1 ) of states and actions, where R(T) Σe 7'r, and Y E [0, 1] is the dis-
count factor. We can then define the state value function V= p) E [Σ 7 * = s). the
action value function as Qx (s, a, p) = E EE 7 rs = s, a a) and the advantage function as A, (s, a. p. (s, a. - (s
Policy gradient methods for DR. In DR, the environment parameter p is a random variable that fol- lows the probability distribution P Pverr? For the convenience of derivation, we assume finite en-
2

---

Under review as a conference paper at ICLR 2022
vironment parameter space P with cardinality [P]. tion to maximize the expected return over all possible environment parameters: Egp (7( p)]- The policy gradient with action incionndndeeeeennt baseline (Sutton & Barto 2018) can be formulated as:
By introducing DR, the goal of policy optimiza-
V PP [n( E
Va log (a(as) (x, -
(1)
(s; P with (st = s(p) denoting the probability of shifting from the initial state 8o to state ter steps under policy π in environment p. For convenience, we further denote g(A, s, a. p) & Vo log no (a a. b), which is the gradient estimator for the state-action pair under vironment p. As long as the baseline is action-independent, we have Ey [o o log (a]{)) = [b] = 0 and thus Ep,.g.= [g] = Ep,nE,a [V log To (als) p)] - Eig]. where the sub- script dropped for convenience. Therefore, by subtracting this action irrion-independent baseline the variance of gradient estimator V Var(g) (g) E[g" T gl [[g] can be reduced without introducing any bias (Greensmith ef 2004).
where define μr o
slp) as the discounted state visitation frequeney,
OPTIMAL BASELINES FOR DOMAIN RANDOMIZATION
To derive the optimal baselines for DR, we formulate the following optimization problem that aims to minimize the variance of gradient estimate w.r.t. the baseline b (Greensmith et al. 2004):
min G(a, 8) () a, (s,
b)²
Elg [gg] <> min Ep a (Qxx)))) - u]²] (2)
6"9
where we denote G(a, s) V log (a(())) TVp log (a(as)) Due to its independence b, the second term [[g]TE] does not affect the minimizer, and is thus omitted RHS of Eq. (2). In the following, we first derive for DR two common choices baselines that are constant or depend on the state only. We then propose the optimal ability to further reduce the variance incurred by the randominzation of environment parameters.
wwwtriiroonmeeddeeeeeeeeee baseline, and show its
3.1 TWo COMMON CHOICES OF ACTION-INDEPENDEN BASELINES
Optimal constant baseline. We first consider constant baseline be that depends neither on the ac- tion nor on the state. The optimization problem in Eq. (2) then becomes minimizing the expectation of a quadratic function, which can be proved to be convex. Referring to the detailed derivation in Appendix A.I the optimal constant baseline bg for DR-based policy gradient methods is:
b% Pp... GG(, 8) (= r)
Ep [V'
(3)
Ep G(a, +)]
This optimal baseline b% can be understood as the expected state value function V'(s,p) over all states and possible environments, where the state value function V'(s,)) is computed as the weighted average of the action value function: ('(s.p) Ex EEppo (G(a,)) (s,
G(o,s)
Optimal state-dependent baseline. As in the standard RL, the utilization of state- nate-dependent base- line the DR setting can considered as an expected value function b(s) = Ep [x(QQ= a, p)]] for each state over all possible environments, which predicts the expected return over the distribution of dynamics caused by the variation environment parameters. Still referring to Appendix A. the optimal state-dependent baseline for DR is given follow:
Ep(p a) Ex [G(a, s) (s, a)p)
(4)
Ex [x(G(a, s)]
3.2 OPTIMAL STATE/ENVIRONMEER WENEIIRROMMETT--EEEENNN
DEPENDENT BASELINE
In DR, the randomization environment parameters imposes additional variability for the estimate of gradients, which aggravates the instability of RL training process. In the following subsections,
3

---

Under review as a conference paper at ICLR 2022
show that this variance can be effectively reduced by further incorporating into the baseline the information about varying environments. To this end, we propose a state/envirommen--eeeaani baseline b(s, P) (b)s for DR, and rewrite the variance minimization in Eq. ( as:
min Pu Ex h(aP)
{
(5)
In Appendix A.1
following the similar deduction as in (Greensmith et al. 2004), the optimal state/environment baseline can be derived as:
5* P) = {b" (s, (s.p:)) where (s,,)))) = Ex [((a, 8) 2) Ex [G(a,
(6)
which indicates requirements of maintaining specific baseline for each environment to incorporate environment-specffic information. Note that environment parameter pt is time-varying along episodes and considered as stochastic input process that partially affects the dynamics, this optimal baseline b* (s, P) is equivalent to the optimal input-driven baseline as derived by Mao et al. (2018)
3.3 VARIANCE REDUCTION IMPROVEMENT OF b" (s,P) P)
("
Theorem 1. Let V (g) and Var (g) denote the variance of gradient estimator by incor- porating an arbitrary state-dependent baseline b(s) and the optimal saute//miionmennddeeenndeee baseline bz (s, P), respectively. Compared to the state-dependent baseline b(s), the variance can be further reduced by b' (s, P), with the following improvement:
ES G( s (r.
Var
M(0) (g) Var (4.P) = EP Ex ()
(7)
B G(a,
Proof. See Appendix A.2
Guided by Theorem we can then quantify the variance improvement achieved by the optimal satte/environment-deppene
baseline (s. P) over the optimal state-dependent baseline b" (s) by Corollary 1. Let Var (a)>
letting b(s) = 6" (s),, as shown in the following corollary.
(g) denote the variance of gradient estimator with the optimal state- dependent baseline b" (s). The variance reduction improvement bf (s, P)) over b" (s) is:
E, s (), 27
(EP(ot)) s
Var (()) b( = EEp
E, G(a,8)
(8)
Since the optimal constant baseline 6" can be thought as letting b(s) = bb for each state S, show in Appendix A.3 that further variance reduction is achieved by 6x P) over by Therefore, we analytically justify that by incorporating the additional information about environments, the optimal wttp///vvroommm--ddeppeeeee
baseline 6* (s, P) can obtain the minimum variance for DR within the baselines that consider both state and environment parameters.
VARIANCE REDUCED DOMAIN RANDOMIZATION
For the single environment case in standard RL, the theoretically optimal state-dependent baseline is rarely used in practice due to the computational concern (Wu et al.) 2018). Rather, for both implementational and conceptual benefit, a common alternative choice of b(s) is the state value function (s). Similarly in DR, even though the optimal state/environmen---deeneeed known and given in Eq. (6). the computation of [P] baselines for all the possible environments requires very high computational and sampling complexity, which is therefore infeasible during the practical training process. Fortunately, similar system dynamics often emerge on environments having the similar parameters, on which the same policy will also obtain the similar expected returns.
baseline is

---

Under review as a conference paper at ICLR 2022
In the following, aim strike tradeoff between the variance reduction performance and the computational complexity required for maintaining multiple stat/cawirromenn-ddeeennd baselines.
Specifically, we propose to maintain an acceptable number M of fattc/environmeet-depprree lines in practice, where M < (P). I. The entire environment parameter space P is then divided as M subspaces {P,) each of which has size |Pyl- For each subspace Pj, can compute the We then baseline as bM (s, P) = (b*(s, Pÿ)] Assuming uniform sampling on P. and referring to Appendix [..4] the total variance of the gradient estimator by incor- porating the state/aubspace-deeeedde baseline bdd (s, P) can be written as:
base-
optimal state-only baseline as (s Pÿ) P (ri [G( p)l/E,,(Ga,,
denote the state/subspace-deepeddee.
Pul varight oir = Vologmolal) P
Var" (g) (g)= = Σ
b's .r. E P,, (9)
where Fς is the i-th environment parameter subspace j with sampling distribution |p]] and Var Var(g's) is computed over pt E P,, following s ~ px((1pp)) and a ~ π. Theorem 2. The variance difference of gradient estimate between the optimal state depende base- deppsded baseline is:
line and the
Var h2 (s) (g) Var (g)
(10)
[ (Pr6l))())(0)) - yg (s)) (X)))
27
M ΣΣ Σ P. (s'p')
Eg s)]
where Yp Yp(s) A E, [G(a, S)(x a,p)), For policy be updated at a certain step, and assuming the uniform distribution for sampling environment parameters, we have b (+) (+))g) 2 Var (g) if for each subspace Pj:
Var (Yz(s)) Var (Y,(s))
(11)
where the variance Var (Y(()) and Var (Y,())) is computed over P and Pj. respectively.
Proof. See Appendix A.5
Theorem 2 provides a practical guideline for constructing subspaces to reduce the variance of gra- dient estimate, by holding the variance of (s(s) in each subspace not exceeding that in the entire environment space. However, it still introduces extra computation to estimating G(a, s) in (s). When the inner product of log policy G(s, a)) is loosely correlated to the Q-value function"] we have E,, (G 8) 8)Q (8, a, p)] = E., G((a, 8) B)]EEn[(... (s, a, p)). and Eq. 11 is thus equivalent to:
Var P (s,)) Z Var (V,,))))),
(12)
It indicates that by clustering the entire environment space into some subspaces and holding the variance of expected return in each subspace not greater than that in the entire environment space, the cttc:/uuupace-depedeee baseline can further reduce the variance of gradient estimate com- pared to the optimal state-only bascline. Note that by increasing the number of subspaces, the paate/subspace-eepedeee
baseline bM (s,P) will approach the optimal sate/enviromment-deeeddeee
baseline b* (s, PP). However, we argue that the more baselines we employ, the more samples are required for estimating these baselines. As will be shortly shown by the experiment, increasing M may not monotonically improve the policy's performance during the training process.
Based on our theory, we propose practical variance reduced domain randomization approach (VRDR) for policy gradient and summarize it in Algorithm [ where the statc/subspace-deeeedde baseline b(s, P,) Ep V (s.p)) V(s, P.) is employed for each state and subspace pair. At each
Take the Gaussian policy that commonly used in policy gradient methods for example. The policy takes as input and outputs for action selections a Gaussian distribution with the expectation ma and variance o² The action me has the minimum gradient of zero, i.e. G(ma, 0. while this state and action pair (m. 8) is not correlated with the Q value before the trained policy obtains high Q value.
5

---

Under review as a conference paper at ICLR 2022
Algorithm Variance Reduced Domain Randomization (VRDR) for Policy Gradient
Initialize policy mo. number of environment parameters sampled per iteration H, number of
subspaces M, clustering interval Ne, maximal number of iterations NN, subspace prototypes
u =(u, {u,};-m] 1;-1+ baseline (s))P for each subspace Pi, and cluster labels = {4, MM as shuffled
with (1)
2: for & = 0 N - do
3: if 1%,N, == 0 then
4:
Sample H environments {peJe=o H-1 with L trajectories for each environment Pc.
5: Compute Ω (V=(p=))== where Vx(p.) Σ5=o R(Tea)/
/ L is the average return of
environment Pe:
6:
Apply clustering method on S to get the cluster set [c.),= with each C, labeled by i. Determine corresponding prototypes u' by computing un = jEC, P5/|Ci]-
7:
8:
Determine min Σ, 14/, uyl where (4:), update prototypes by u, u/y and relabel
cluster with 4, else
9:
10:
Sample set of environment parameters } H-1 uniformly and determine their cluster label by min; Ipe uil to obtain the cluster set M.
11:
Sample I. trajectories (Tr..) L-1 for each environment Pe using policy mk- end if'
12:
Use PPO for policy optimization get the updated policy ux+1 on trajectories, with the baseline ((s)P;) of the cluster that has been grouped to.
Update (ss(P;) on the trajectories in cluster C,.
15: end for
iteration k, we sample M environments and group them into M clusters by finding the nearest sub- space prototype. Then, we train the policy on the trajectories sampled on each environment with the corresponding baseline b(s, P3) and update the baseline b(s, P,) on the corresponding trajectories. During the clustering phase, as guided by Eq. (12). we perform the clustering method on { 1111 w.r.t. the average return Vn (pe) to get the M clusters with corresponding subspace prototypes v/. The clustering methods are usually applied group fixed dataset, with the labeling done once and for all. In VRDR, however, we perform clustering multiple times during training. hence the proto- types between two clustering phases would be similar but have different labels. To avoid the case where the new prototype un close to a certain current prototype is assigned with different label, we relabel these new prototypes u', and update the current ones u according to w' as described Line 8 of Algorithm I between new prototype u' and current prototype Then, after the clustering phase, the subspace P, can be reconstructed according to the sampled environments in cluster Cz. We utilize loss function Ly = E (st,, P.) Er= 7 'ra"" 2 to update the baseline V(s, Pi) for each cluster C,
where we aim to find a set of labels 1 {4} that minimizes the distance |u/ uil
It has been proved by Ghadimi & Lan (2013) that the optimization of gradient variance can accel- erate the convergence of gradient descent methods. In Appendix A.7]] we show that our proposed VRDR method gains a x (K >> 1) acceleration on the convergence of policy gradient methods, as compared to the direct application of optimal state-dependent baseline for DR.
EXPERIMENTS
In this section, we evaluate our proposed VRDR method on six simulated robot control tasks, where the fundamental environment parameters that affect dynamics are randomized for the generaliza- tion of trained policy. The experimental environments are implemented based on the robot control simulator as described (Brockman al., (016). We compare VRDR with two baseline algo- rithms: the uniform domain randomization (DR) (Mehta et al. 2020) and the multiple value net- work (MVN) (Mao et al. | 2018). We utilize the proximal policy optimization (PPO) (Schulman et al.) 2017) to train generalized policy for all the comparison algorithms. In DR, we uniformly sample the enviromment parameter p and collect trajectories by using the current policy on the sampled envi- ronments for policy update, where value function that is learned from trajectories from all sampled environments is employed as the state-dependent baseline. In MVN, multiple value functions are learned for the pre-sampled environments at the beginning of training, while each value function V,

---

Under review as a conference paper at ICLR 2022
Table I: Range of parameters for each environment.
Environment
Environment Parameters
Training Range
Material Density
750, 1250
WalkezZD/Hoopcr/affccccchhh
Sliding Friction
0.5, 1.1]
Pendulum
Pole length
0.5,, 1.5
Pole mass
0.5, 1.5
Pendulum2D
Pole length
0.5, 1.5
Polcl length
0.50, 1.00]
InvertedDoubbeePnnduuum
Pole2 length
[0.50, 1.00]
is utilized baseline for the gradient estimator of trajectories collected environment parame- ters that are closest to the corresponding pre-sampled p;- For VRDR, we apply hierarchical cluster method (Jain 2010 to divide the entire environment space during training, and group the trajecto- ries collected at each interaction phase as described in Algorithm [ See Appendix [A. for detail of hierarchical clustering, and Appendix AA. for comparison of clustering methods. We keep all the hyper-parameterss about policy optimization for PPO the same as the implementation in [Broc-] man et al./ (016), and purely discuss the gain brought by variance reduction for gradient estimate in policy gradient methods. Our experiments are designed to answer the following questions:
Can VRDR effectively improve the convergence rate of the policy training. Will the final reward achieved by the converged policy trained by VRDR outperform the other two baseline algorithms?
How would the policy performance of all the three algorithms degrade on the out-of-distribution unseen environmentsvyynamies?
How would the specific pppeepprraetter in VRDR affect the performance of training process.
5.1 TRAINING CURVES WITH RANDOMIZED DYNAMICS
The six robot control tasks are as follows. 1) Walker2D: control a planar biped robot to run as fast as possible; Hopper: control planar monopod robot to hop as fast as possible; 3) Halfcheetah: control a planar cheetah robot to run fast; 4 & 5) Pendulum & Pendulum2D: apply the force to swing a pendulum and keep upright: 6) LnverteiDoublePenduluu control cart (attached to a two-link pendulum) to balance the whole system and keep the pendulum upright. We modify the environment parameters in the configure file to generate six environments with the same control task and different dynamics. The possible sampling range of the environment parameters is shown in Table [ Specifically, we randomize the material density that determines the mass and inertia, and the sliding friction acting along the tangent plane in Walker2D, Hopper, and Halfcheetah. We randomize the pole length, respectively, in Pendulum, Pendulum2D and InvertedDoublleendulu.. and the pole mass additionally in Pendulum2D.
We run the training process on environments sampled on the preset parameter range for the same number of iterations N until all the algorithms are converged. At each iteration kt generate H = 100 environments by uniformly sampling the parameter range in all algorithms. We run at least 15 seeds for each algorithm
all the environments to obtain the training results. The policy and value networks in each algorithm are trained for the same number of epochs and sampled trajectories each iteration. We show the training curves of the six tasks, respectively, in Figs. [[a) The solid curve is the average returns on all the seeds at a certain iteration. while the shaded area denotes the standard deviations. It can be seen that by applying DR on PPO, we can improve the average return on the whole training range, while VRDR can accelerate the training process on the six tasks. In some specific tasks, such as Walker2D, Hopper and Halfcheetah, VRDR even achieves higher average return than DR and MVN. In balancing tasks like Pendulum and Pendulum2D, VRDR has shown a significant accelcration on the convergence of policy training.
5.2 GENERALIZATION TO UNSEEN ENVIRONMENTS
In this subsection, we evaluate the generalization capability of trained policies on the unseen range during training. We learn 15 policies for DR, MVN and VRDR, respectively, on training environ- ments with different random seeds and apply the trained policies on corresponding testing unseen environments. The generalization performance for each algorithm is measured by the average score over 15 policies. The generalization evaluation of DR, MVN and VRDR are shown in Figs. 2(a) ((b] VRDR shows the best generalization performance on Walker2D. DR has a broader range of generalization than MVN, although MVN achieves higher score on the training range than DR.
7

---

Under review as a conference paper at ICLR 2022
DR Wi- VIDR
DR MIN VRDR
DR
NVN VADA
30:00
3000
Ferations
100 190
keradons
Iteration
(a) Walker2D
(b) Hopper
(c) Halfcheetah
NVN VRDR
-400
-200
3ED Ferations
Rerations
iterations
(d) Pendulum
(e) Pendulum2D
(f) InvertedDoubleeenduuuun
Figure 1: Training curves of average return.
di
0000 length
JA0
(a) Walker2D
Pendulum2D
Figure 2: Heatmap of return in unseen environments Walker2D and Pendulum2D with policies trained by DR, MVN and VRDR in the training environments.
For Pendulum2D, however, all the algorithms present close performance. This result indicates that the generalization performance is consistent with the average return achieved after convergence in training, and would be affected by the convergence rate.
5.3 ABLATION STUDY OF VRDR
Number of baselines M: In Theorem
show that by applying (s, P), maintain state/enviromment baseline (s, p.) for each environment and can obtain the minimum variance of gradient estimate. Theoretically, increasing M [P] will achieve a better variance reduction improvement. Figs. 3(a) 3(c)
show the training curves of VRDR on Walker2D, Pendulum, and Pendulum2D with different numbers of baselines. Note that VRDR is reduced DR when setting M = 1. We find that the increase of M will decrease the number of training samples for approx- imation of each baseline if we sample the same number environment hence retards the convergence rate of VRDR. This suggests that we can set M to a small number in practice to strike the tradcoff between the variance reduction and computational complexity.
each iteration, which
Clustering interval N.: Each new clustering result may change the group that each environment belong to, hence altering the value function and affecting the policy training in the next clustering interval. The underlying dynamic mechanisms vary in different environments, and thus the shifting of value function may affect the training with varying degrees. We show the training curves of VRDR with different clustering interval N, Walker2D, Hopper, and Halfcheetah in Figs. [{a)) 4(c) and the clustering results in Appendix A.9 The results suggest the need of a specific tuning of N. for each task.
RELATED WORK
State-of-the-ar methods that have endeavored tackle the generalization and robustness of DRL, including DR, meta RL and robust RL, all implicitly explicitly impose diversity the policy training. DR and meta RL both tend to learn policy that can generalize to a variety of environ- ments (Jiang al. 2021| Lin et al. 2020), which may differ in dynamics and reward function during training. By introducing an adversary that acts to disturb the dynamics or the observation of the envi- ronment during training (Zhang et al. 2020), the learned policy can gain robustness on deployment.
8

---

Under review as a conference paper at ICLR 2022
39crogg roburn
180
355 200
nerotions
100
500 14x
30 Denations
Nerottons
(a) Walker2D
(b) Pendulum
(c) Pendulum2D
Figure 3: VRDR training curves of average return usine different M.
580
M,
Nu- XL0
No 1.30
Naw
140
No
N,-33
170
LS0
N. 50
201
N.
N2
Iterotions
Iterations
Iterations
(a) Walker2D
(b) Hopper
(c) Halfcheetah
Figure 4: VRDR training curves of average return using different No.
The introduced diversity for training environments enables the generalization of DRL policy, while the consequent high variance aggravates the sample efficiency of DRL training.
For policy optimization under DR, Mehta et al. (020)) declare that the uniform sampling of DR may lead to suboptimal and high-variance policy, and propose active DR to encourage the sampling distribution select informative environments. Several works have been proposed to optimize the sampling distribution for DR (Andrychowicz et al., [020) Paul et al. 2019). while the baseline opti- mization is seldom considered in the policy search using policy gradient and the direct application of state-dependent baseline is commonly used as formulated in Eq.
Policy gradient methods usually suffer high variance due to the estimation of gradient from sampled trajectories. By learning a Q-function (critic) instead of using its sampling estimation, actor-criti methods reduce the variance with the expense of introducing bias (Sutton & Barto, (2018). Another line of research follows the idea of control variate (Cheng et al. 2020), by subtracting an commonly used state-dependent baseline from the gradient estimate to reduce the variance without introducing bias. By assuming the independence of cach dimension in the action vector, Wu et al. (018)) pro- posc a eetioo/state-deeemdeett baseline, which reduces more variance than the state-only baseline. These methods are designed for standard RL and require further discussion under the context of DR. Variance reduced gradient algorithms like stochastic variance- reduced gradient (SVRG) (Johnson & Zhang 2013) and momentum- based variance reduction (Cutkosky & Orabona 2019) methods have been studied in the policy gradient for RL (Huang et al. 2020; Xu et al., (020)). These methods, however, do not consider the variance reduction for the DR setting. |Mo al. (2018) propose an ieput-dependent baseline to reduce the variance of stochastic input process, which is similar to our suute//niionmmee--dppeeddee
/arto/2018]
baseline, Then, the setiii-alle-netwwon (MVN) approach and meta- learning approach are proposed to learn the baseline. Meanwhile, Liu et al. (2019) propose to learn the per-task control variate and the meta control variate for each meta-RL. task, where the per-task variate is constructed by the eetoon-dependent baseline. In essence, the MVN and per-task control variates approaches are the same, since they both aim to learn a baseline for each input or task. And both of these meta-learning approaches are built upon model-agnostic meta learning (MAML) (Finn et al. 2017) to scale when the number of input instantiations or tasks are large.
7 CONCLUSION
In this paper, we have focused on reducing the high variance of gradient estimator in policy gra- dient methods for DR with additional randomness on environment parameters. Specifically, we derived the optimal haate//mniioomeen-deeeedddd environment information further variance reduction could be achieved over the optimal constant or state-only baselines. We then proposed variance reduced domain randomization (VRDR) approach for policy gradient methods, to strike a tradeoff between the variance reduction and computational complexity in practical implementation. We have validated VRDR several robot control tasks, demonstrating an overall faster convergence speed of policy training, and even better policy perfor- mance in some specific tasks.
baseline, and verified that by incorporating the

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265, 2021.
OpenAI: Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 2016.
Ching-An Cheng, Xinyan Yan, and Byron Boots. Trajectory-wise control variates for variance reduction in policy gradient methods. In Conference on Robot Learning, pp. 1379-1394. PMLR, 2020.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. In International Conference on Machine Learning, pp. 1282-1289. PMLR, 2019.
Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd.
Advances in neural information processing systems, 32, 2019.
Alvaro Cabrejas Egeaa, Raymond Zhanga, and Neil Waltona. Reinforcement learning for traffic signal control: Comparison with commercial systems. CoRR, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1135. PMLR, 2017.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth methods for nonconvex stochas- tic programming. SIAM Journal on Optimization, 23(4):3341-2368, 2013.
Evan Greensmith, Peter L. Bartlett, and Jonathan Baxter. Variance reduction reduction techniques for gradient estimates in reinforcement learning. J. Mach. Learn. Res., 5:1471-1530, 2004.
Feihu Huang, Shangqian Gao, Jian Pei, and Heng Huang. Momentum-based policy gradient meth-
ods. In International Conference on Machine Learning, pp. 4422-4433. PMLR, 2020.
Anil K Jain. Data clustering: 50 years beyond k-means. Pattern recognition letters, 31(8):651-666, 2010.
Yuankun Jiang, Chenglin Li, Wenrui Dai, Junni Zou, and Hongkai Xiong. Monotonic robust policy optimization with model discrepancy. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4951-4960. PMLR, 18-24 Jul 2021.
Rie Johnson and Tong Zhang. Accelerating stochastie gradient descent using predictive variance reduction. Advances in neural information processing systems, 26:315-323, 2013.
Katie Kang, Suneel Belkhale, Gregory Kahn, Pieter Abbeel, and Sergey Levine. Generalization through simulation: Integrating simulated and real data into deep reinforcement learning for vision-based autonomous flight. In 2019 international conference on robotics and automation (ICRA), pp. 0008-6014 IEEF, 2019.
Zichuan Lin, Garrett Thomas, Guangwen Yang, and Tengyu Ma. Model-based adversarial meta- reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta- reinforcement learning. In International Conference on Machine Learning, pp. 4061-4071. PMLR, 2019.
10

---

Under review as a conference paper at ICLR 2022
Hongzi Mao, Shaileshh Bojja Venkatakrishhan, Malte Schwarzkopf, and Mohammad Alizadeh. Variance reduction for reinforcement learning in input-driven environments. In International Conference on Learning Representations, 2018.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning, pp. 1162-1176. PMLR, 2020.
Fabio Muratore, Christian Eilers, Michael Gienger, and Jan Peters. Data-efficient domain random- ization with bayesian optimization. IEEE Robotics and Automation Letters, 6(2):911-918, 2021.
Supratik Paul, Michael Osborne, and Shimon Whiteson. Fingerprint policy optimisation for
robust reinforcement learning. In International Conference Machine Learning, pp. 5082- 5091. PMLR, 2019.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3803-3810 10. IEEE, 2018.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv: 1610.0018,, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. arXiv preprint arXiv:/803.0724 2018.
Zhaoming Xie, Xingye Da, Michiel van de Panne, Buck Babich, and Animesh Garg. Dynamics randomization revisited: A case study for quadrupedal locomotion. CoRR, abs/201 1.02444, 2020.
Pan Xu, Felicia Gao, and Quanquan Gu. Sample efficient policy gradient methods with recursive variance reduction. In International Conference on Learning Representations, 2020.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6672-6679, 2020.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations, 2020.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 DERIVATION OF OPTIMAL BASELINES
Following the derivation in (rreenmmih al., 2004), give the derivation of three kinds of optimal baselines by solving the optimization problem as proposed in Eq. (2).
Derivation of optimal constant baseline.
can be verified that the optimization problem in Eq. and denote:
is convex w.r.t. the constant baseline be.
min F(be) E (() tcl2
(13)
By letting the first-order derivative equal to zero:
db((b))
22P ((a,s) bel] 0,
(14)
db
we have the optimal constant baseline for DR as:
E E Pr 8) (Q a)p
(15)
Ep G((a,)) (G(a,
Derivation of optimal state-dependent baseline.
The variance minimization problem w.r.t. b(s) is denoted as:
min F(b(s)) & Eppp.. (Qa,,)) b(*)
b(s)P²
(16)
Following the generating process of joint distribution of p, 8 and a, we have P(p, 8, a) = P(p) P(slp) pa (s|p)r((|s)) (a]s), where the last equality is because that policy a is constructed without knowing p. The RHS of Eq. (16) can be expanded as:
Bn G(a, s) (s, b(s, [)]]]
E E
P(p))
Mr (3, (p: G(a, (,), - (1).)))1
1 IP|
((p:)ua G(a ())0) b(s),
(17)
By letting the first-order partial derivative equal to zero:
aF(b(s))
[P]
-2
j=1 (())) [[aa
(C. (,la,,,)1Q0ss))aaap) - b(,,)] = =0,
(18)
Ab(s)
where b(s) {b(s;)}, have:
P(p.)u. G(a, ((a,,,)1Q0(s, b(s,)] 0, for each b(s,). (19)
can be further written as
P(p, EE (l.,s))1((()(() =
P(ps, 83 Er (,),,)))(({))]]
(20)
and
P(s) P(ss))
P(s,,))) ((a (21)
12

---

Under review as a conference paper at ICLR 2022
where P(s,) can be further canceled both sides. We then get the optimal state- deteedeepnddnnt baseline for 8, as
Pl Σ ==
P(pils,)BB G(a, (,)(Qn (1)),)) 85)
E P(rls)) Ba G(a, 85)(Q
(*()
Σ Pl
P(pils)
G(a, ((pils))EB E
B G(a, 85))
(22)
For a continuous state space, we have
I P()aa)) B
G(a, (s,
6" (s)
(23)
1 G(a, 8)
Derivation of optimal statecavviroumeut--deeeeedd baseline.
The variance minimization problem w.r.t. b(s, P) in Eq. 5 denoted as:
min F(b(s, P)) = Ep- E G(a,, ((Q,,,)) b(s r) b(s,P)
(24)
By letting the first-order partial dirivative equal to zero:
aF(b(s, -2E, [Gaa, s) - ab(s.p:) p:)
b(s, b(s.p))
= 0,
(25)
we obtain the optimal sune//ewinommeen-ddeeenddn baseline (x) = (s.(): where
(' p.) [G(a s)Q(s, s))(s....))))/..((((..)
A.2 PROOF OF THEOREM
The variance of policy gradient using the optimal stuc//eniioommenn-deepeeeenn baseline b" (s P) is:
(s, E,
2
(..P)) = E G(a, 8)
Blg]" [B[]]
Ex (G(,,, s)]
(x(s)
(s))
=E
Giaa 8)
(2, - 22 a,p)
Z
(s)
Epput [Xp(s)
"Eig
[igT)[[g)
(26)
Z
a. p)), Yp(s) E, (G(a, (), app and Z A E, [G(a,))) s)]. For arbitrary state-dependent baseline b(s), the variance difference between b(s)
where denote Xp(s) A E, GG(a, 8) (s,
and b² (8, , P) is:
Var Varb
(Qr(s, -b(c)))))ss)" (Q)
Yg(s)
=Ep Te Ex
s)
Z
Yp(s)
(²)
P,, Eg
s)
(2(s) 2Q, + 2Q, (s. p)
Z
Z
E G(a, s){(s,
=Ep
E. B=(a(, ((sss +
E, G(a, s)]
Ex[G(a, Q
=E -Ep
G(a s)}b(s)
(27)
/ F ((a,)) s)]
13

---

Under review as a conference paper at ICLR 2022
A.3 VARIANCE REDUCTION IMPROVEMENT OF (s P) OVER b%
The application of optimal constant baseline Dy can be considered as letting b(s) b%2 for each state s E S. Hence, by direct applying Theorem we can obtain the variance reduction improvement of 6" (s, P) over bg as:
Var" (g) Var (s.P)(g) (g)
Ex G[aa, s)Qx
B [G(a,
BB (((a
Ep Qx,
En G[a S) app]
E P
Ex [r(G((,ss) s]]
(28)
Ep s(
Em G(a, s]]
A.4 VARIANCE OF POLICY GRADIENT ESTIMATOR BY APPLYING (s,
Under uniform domain randomization, demonstrate that the variance of policy gradient in the whole environment space can be formulated as weighted sum of the variance subspaces:
V far
(g)
(s(pk)
Σ T(als)o((,,s.a.dl s. - ((() s,
P
M
IPJ
P
Mn (s/p))
Σ
x ((0,))) Eg - s, Eg(0,s,a,1d)
Pj P
Var(g").
(29)
where  is the gradient estimator of environment in subspace and pt is the i-th environment in subspace j.
A.5 PROOF OF THROREM 2
In this subsection, we study how to divide subspace reduce variance of gradient estimator. First, we derive the variance difference in Eq. (10). For certain state-dependent baseline b(s) and an arbitrary sampling distribution P of environment parameters, the variance of gradient estimate is:
M
P
b()
b()) (g)
P(p:)
Σ
Mn (s(p))
Σ ((als)G(als)) [o((s,a.))) 20, + (+)
M
I2
(2
(3
P(ps) Σ Mr (slpt)
X
26(s)) (8) (b))
pi
Z
Z
Y2
(Y
b(s)VZ) )
P(ps)
Σ Mr (slpt)
X
(30)
Z
v7
The variance of gradient estimate by incorporating the state/subsspace-deeendde baseline is
M
IP5l
(2)
E (p)())))
(
Y 2
Var (g)
P(p.)
Σ ur (slp/)
X
xx
Vz
1=1
(31)
14

---

Under review as a conference paper at ICLR 2022
and variance of gradient estimate by incorporating the optimal state- dete-deepnddett baseline is
M P3 P(p))
Y2 (s)
Ep(pls) (x())
Varb
Σ
(s(p)))
X
P.
Z
VZ
v2
(32)
The difference between them is
Var (x) VarnM (g)
[EPC))))(()]] - (x) (Ep 2 -X)
Pg1 P(p.) Σ Mn (slp))
Ex [G(a,
(33)
Thus, Eq.
10 can be obtained by letting P(p:) - Then, discuss the dividing process of (g) Var (g) hold. Under the
subspaces make the variance reduction improvement V (*))")))))))))))))))))))))( uniform sampling distribution P(p) [P] we have
M
P3l Var P (Yg(s)) = Var ? (Vz(()) (V(() - (x)))
P
M IP
Yr² (s))²
M IPs
1 P(slp!)
(Ep(p a) [x((s]] yyp (s))². (34)
Pl
P(s)
The first equality holds since
Pl P
1. And
==
Pg PP]
P3
V
(Yy(s))
Σ
(E (s))
M
P3
P(s
(Bry (o)(
PP
Payl
P(s)
M
P
P(slp!)
(Ergcels) (x2
(35)
PP|
P(x)
Note that P(s|p/) = (slp/)))
Thus, as long as Var (Yy(s)) 2 Var (Yp(s), we have
Var (s)(s)) ΣS P. arr (Y,(s)) En p V. (Yp(s)) Then, combing Eq. 10
M pP
1p
we have V Var (6) (g) > Var
A.6 DOMAIN RANDOMIZATION WITH AN ARBITRARY SAMPLING DISTRIBUTION
In this subsection, we study how to divide subspace to reduce variance of gradient estimator in domain randomization with arbitrary sampling distribution. Specifically, we prove that for an arbitrary sampling distribution of environment parameters, as long as Eq. (II) holds, we still have Var (5) (g) 2 (()) Under arbitrary sampling distribution P. the variance difference of gradi- ent estimate between the optimal state-dependent baseline and the state/subspace--dependen baseline
15

---

Under review as a conference paper at ICLR 2022
18:
Var (g) -0r0 (g)
(BP())))(((S)) - (s)) - -(PPGON))((S)) - -Yp, (s)) (s))
LE J-PPEP
P(p)) (s(ps)
E, s]]
(36)
where P(pi) is the sampling probability of environment from the entire environment space P. P(p.) Hence, Emer, P(p.) can be used to denote the sampling probability of environment p; within the
subspace Pj. For Var VarP:(Y(())) we have
P(P) P(s|p:)
Var (Ys())) Σ
(R (V,))) (s))
(37)
EpEp, P(p.) P(s)
P.EP,
and thus
M
M
p(spp.)
Σ P(pi)
Var (X,(s)
= Σ
Σ
P(p.)
( ((r()()))s) (s))² (38)
P(s)
PiEP;
For Var (Yp (s)), we have
P(pi)
Var Var?(V)())) =V -V,r?(Y()))
PiEPs
Σ (irl)s)(Err)))) - (s))²
PLEP
M
(s,, (s))
1=1 P.PP3
M
P(s|p))
Σ P(p.)
(X - Y,, (s)) (39)
P(s)
#=1 PPES
where the first equality follows that E/= [E P(p:) = 1 and the second quality is from the definition of Var ar (Y)(
Note that P()lr() = Mx (slve)) Thus, as long VarP(K 2 VarP:(y we have ン Ep.Ep, P(p.) Var ((Y)s)) 2 E/ Eper, P(p)) Varp: Then, combing Eq. 360, have Var (0) (0) 2 Var (g)-
A.7 CONVERGENCE ANALYSIS OF VRDR
Let VJ(0) = Vo (T( p)) be the policy gradient. In this section, we present the theorem proposed by Ghadimi & Lan (2013) under the DR setting. It shows that under Assumption that (@) is Lipschitz continuous w.r.t policy parameter 6, the variance of gradient estimator dominates the convergence rate.
Assumption 1. [VV](0,) VJ(02)) < LI 02l1. 10,, 02 R..
Applying arbitrary gradient estimator, the convergence of policy gradient is upper bounded as follows.
Theorem 3. (Ghadimi & Lan
2013 Under Assumption 1 and assuming that the step size ar
minf) 1. /Var(au)K J. we have
I(J(0)) - J(O))
L L Var
En E
(40)
K/2
K
16

---

Under review as a conference paper at ICLR 2022
where ((0)) is the maximum expected cumulative discounted reward that can be achieved by policy Tg., K is the maximum number of iterations for policy gradient method, 9k = g(0,, 8, a) is the
gradient estimator at the k-th iteration for (s.0), a), and
E -= (ap -/22a]]B/
E E
(41)
Σ (ckk L/20
L/2a2 the expectation w.r.t distribution PR((O)) E (ap--1/22 and all the possible randomness.
By holding Eq. (12). we have Var (0) and gains a =
Vert"in
(N >> 1) acceleration Var
Var
on the convergence of policy gradient methods.
A.8 ANALYSIS OF ADDITIONAL COMPUTATIONAL COST OF VRDR
In DR, the computational cost contains mainly the feed-forward computation and back-propagation computation of the policy network and the value network. For iterations, H trajectories with length of Li, size of state space (S] and constant computational cost of feed-forward and back- propagation computation d1, the computational cost of DR is O
L,, |S| di).
The additional computational cost introduced by VRDR is mainly from the clustering phase. Con- sidering the hierachical clustering method in Algorithm - The maximum number of outer loop is H M H At each outer loop, the computational cost of distance matrix is m² H
2 be d2, the computational cost is Then, let the cost of distance computation in Line 6 of Algorithm O(N- H d2).
Note that compared with the simple computation of distance between two scalars uy and uj. the computational cost of forwaforward and back proppproppgation computation are implemented for the en- tire policy and value networks, both of which contain multiple layers and neurons. Therefore, in practice, d1 is usually multiple order-of-magnitude larger than d2- Thus, under the condition that LA and [S] are comparable with H, term O(N H Ln [S] di) will become the dominate term the overall computational complexity of VRDR, while the additional computational complexity introduced by the clustering phase of VRDR can be neglected.
A.9 CLUSTERING RESULTS DURING TRAINING
In this subsection, we show the clustering results of three successive clustering updates (iteration k n.er (n + N) (n + 2.) Walker2d, Hopper, and Halfcheetah in Figs. 5(a) 5(c)
A. ABLATION STUDY OF VRDR ON CLUSTERING METHOSS
In this subsection, we study the influence of different clustering methods on VRDR. Considering that we require to specify the number of clusters in VRDR and the clusters should be generated with similar state values to satisfy Eq. [22. we additionally chose k-means for the comparison to the originally adopted hierachical clustering. The comparison results of different clustering methods shown in Fig. [6(a)aan and Fig. [6(b] It can be seen that applying k-means and hierachical clustering methods present similar average scores on Hopper and Halfcheetah.
A.11 COMPARISON TO COMBINED STATE- DEPEN-TE BASELINE
The optimal state/environmen baseline can be considered as combined state-dependent baseline, where combined state is composed of original state and environment parameter p, if we treat p as part of state of the MDP. In this subsection, we conduct experiment to evaluate DR with combined state-dependent baseline (DRCS) and the training curves are shown in Fig. 7(a) and Fig. 7(b) The curves of DRCS show that applying regular deateddepnndent baseline on this combined state degrades the average score during training as compared to DR and VRDR.
17

---

Under review as a conference paper at ICLR 2022
k = n=
k = (n + 1)Nc
k = (n + 2)Nc
776
776
803
density
96
10
1013
1197
friction
aa) Walker2D
k= nNc
k 1)
k= (n 2)N
density
10.99
1066
000000
friction
(b) Hopper
k == ==nNc
k = (n + 1)Nc
k = (n + 2)Nc
750
750
density
005
1092
1082
S229 000060
000
060590000000 200.0000000000.
650 0000000
friction
(c) Halfcheetah
Figure 5: Clustering results in three successive clustering update.
A.12 COMPARISON TO OTHER BASELINE ALGORITHMS
In this subsection, compare VRDR with other baseline algorithms including DR with meta base-
line (DRMB) (Mo et al. 2018 Liu et al. 2019) and Epopt (Rajeswaran et al. 2016). We use PPO with the setting Section
DRMB, a meta baseline is applied to learn the simpler
18

---

Under review as a conference paper at ICLR 2022
VRDRkmeans
VRDR VRDRKmeans
average return
2500
average return
2000
1000
500
iterations
iterations
(a) Hopper
(b) Halfcheetah
Figure 6: Training curves of average return achieved by using k-means and hierachical clustering in
VRDR.
DR
DR
5020
3020
VRDR
VRDR DRCS
average return
4000
DRCS
average re tutur
2500
2020
3000
3300
2000
200
400
600
800
1000
1286
1400
230
400
600
1000
1206
iterations
iterations
(a) Walker2D
(b) Hopper
Figure 7: Training curves of average return achieved by DR, VRDR and DRCS.
VRDR VRMB
VRDR
3000
VRMB
average return
average return
2500
2020
1900
1000
500
200
400
600
800
1030
1203
3430
230
403
600
a00
1000
1206
iterations
iterations
(a) Walker2D
(b) Hopper
Figure 8: Training curves of average return achieved by VRDR and VRMB.
alternative Eg [Qx (s, a, p)] instead of Satte/amiiommeen-ddeeeeeen baseline b" (s, p). The training curves are shown in Fig. 8(a) and Fig. 8(b) VRDR shows better average scores than DRMB Walker2d and a similar average score on Hopper.
In Epopt, where we set a = 5. The training curves are shown in Fig. 9(a) and Fig (b)) And can be seen that VRDR outperforms Epopt both environments.
purely trajectories from the a-percentile worst-case environments for training,
19

---

Under review as a conference paper at ICLR 2022
VRDR
3000
VRDR
5000
Epopt
Epopt
2500
average return
average return
4000
1000
15000
2000
10000
1000
400
600
1000
1200
200
800
1200
iterations
iterations
(a) Walker2D
(h) Hopper
Figure 9: Training curves of average return achieved by VRDR and Epopt.
DR
MVN VRDR
mean variance
0.16
0.14
0.10
0.00
100
200
300
400
iterations
(a) Pendulum2D
Figure 10: Training curves of mean variance of gradient estimates over random seeds achieved by DR, MVN and VRDR.
A.13 VARIANCE OF GRADIENT ESTIMATES DURING TRAINING
In this subsection, we empirically evaluate the variance of the gradient estimates that are practically achieved by DR, MVN and VRDR on Pendulum2D. It can be seen that at the beginning of training, VRDR can achieve significantly lower variance of gradient estimates than the other two baselines, which coincides with the faster convergence behavior of VRDR shown in Fig. ((e)
A.14 HIERACHICAL CLUSTERING FOR VRDR
In this subsection, we detail the hierarchical clustering method used in the implementation of VRDR, and summarize it Algorithm
A.15 FROM EQ. 12 TO EQ.
In this subsection, we prove that Var (V>(s,)) condition of Var (x)) 2 Var (Y, (Yy(s)) in Eq. 11 When the above condition holds, have
Var VV in Eq. 122 is a sufficient
Var (V- (s,p)) Var (V+(s,))),
(42)
and by multiplying both sides with non-negative term E, [( s)}, we have
Ea s) (V+(s.)) E, [G(a (V+(s,))
(43)
20

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Hierachical Clustering
Input Ω (V= (po))
2: Initialize number of clusters M, current number of clusters m H, prototypes {u
(. distance matrix Ma
3: while m 2 .M do
4: for = 0 - -1 do
5:
j= m 1
6:
Ma(i,j) [1: Uj M)( = Ma(1. end for end for
7:
8:
9:
merge the two clusters ui- and uy- having the minimum distance: u,- u. uy
11: m = m -
end while
13:
Output prototypes [
M
By expanding the LHS with the definition of variance, we have
[.( a (V>(s,))
(44)
IP]
=E,[G((a s)]
P(pils)
Ep [s Va (s,
IP]
P(p)(s) (alG(a,s) E. (s, E.[G(a, (0 as,
Eg (G(a, }]]
PP]
P(pi(s)
Ep E. [G(a, [(a.,,)) E, (G(a, s)
E, (G(a,, s)]
i=1
Var (Yp(s))
Ea[G((a,s)
where the third equality can be obtained by using the condition Ea(G(a, 8) s)Q= [))(a, a.p) p)]. Similarly, we can obtain aar (V, p)) (Y,
[(a,ss)
Hence, from the inequality in Eq. (43) we have Var Var?((y (Yp(#)) 2 Var (Y,(s)) and thus Var""())(g) 2 VarM(g) according to Eq. (10). Namely, the improvement of variance reduc- tion by applying subspaces generated by following Eq. (12) can also be quantified by Eq. (10).
21


Reviews:
[{"review_id": "tlNfoFH0mfg", "reviewer": "Reviewer_ePpB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths\n1. this work has a good structure, is well-written, and I feel pleased to read this paper\n2. this work is well-motivated and I believe the topic is highly-significant and related to the venue\n3. this work has a good balance of theoretical contribution and empirical evaluation. it is technically novel and the proposed algorithm works well in practice compared to baselines\n\nweaknesses/questions\n1. this work targets the variance of the gradient estimates. so why not directly measure the variance of the gradient estimates? e.g. empirically? I think this is the most straightforward way.\n2. I agree lower variance gradient estimates could converge fast, but in general, it's also easier to converge to local optimal. what are your opinions on this?\n4. for sec.5.2, could you elaborate more on why VRDR generalizes better to unseen environments? I don't find it intuitive to understand why lower-variance gradient estimates find a solution that generalizes better (as opposed to sgd in supervised learning)? what's the connection or insight?\n5. what's MRPO at the end of sec.5.1? probably a typo?\n6. could you elaborate on the added computational cost compared to the baselines?", "summary_of_the_paper": "this work studies policy gradient methods for domain randomization (DR). in particular, it investigates baselines for policy gradient under the dr settings such that the gradient estimate can have a lower variance to ensure better policy updates and learning. this paper derives optimal state/environment-dependent baseline theoretically, gives general recipes for building the baseline, and proposes an algorithm called variance reduced domain randomization (VRDR). vrdr is evaluated on several continuous control tasks and it performs better compared with two baselines.", "summary_of_the_review": "I enjoy reading the paper and I find it theoretically novel and sound. However, I do not find the experiments very straightforward, I believe it's necessary to compare the empirical variance of the gradient estimates with the baselines. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mJtls49Futj", "reviewer": "Reviewer_uerm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths ###\n- It’s great to see that the proposed algorithm demonstrates good generalizability compared to other methods in Figure 2. I have a question about a plot though. In Figure 2.(a), the most left subplot, the DR’s score is no higher than 1400. However, in Figure 1 (a), the mean of the DR curve is around 3000. Did I interpret the plots correctly?\n- Domain randomization is commonly used in practice to train good policies using simulators. The problem that the submission tries to address is relevant to the community. \n- I appreciate the efforts of deriving and analyzing the practical algorithms based on theoretical development, although it would be better to analyze the variance reduction of the practical baseline proposed. \n\n### Concerns ###\n- The novelty of the submission is limited. As authors noted, the baseline in Section 3.2 is a special case of the input-driven baseline derived by Mao et al. (2018). And it's not discussed how proposed method is different from Liu et al.’ s per-task control variate. \n   - If I understand it correctly, the environment parameters can be treated as part of the state of the MDC. The combined state (original state + environment parameter)-dependent baselines can be learned just as regular state-dependent baselines based on function approximation without additional techniques, such as the subspace clustering. \n  - In Algorithm 1, clusters are formed based on Q values, but the nearest neighbors are found based on environment parameter. This inconsistency is a bit odd to me. \n\n- There may be a bug in the theoretical analysis. I believe the expectation of environment parameters need to be conditioned on s, just as the action distribution is conditioned on s (policy \\pi). This includes E_P in Eq. (4), corollary 1, and Theorem 2. \n- The clarity of the paper can be improved. \n  - The math notations are sometimes confusing and math arguments are not always precise. Examples:\n      - On the fifth line after Eq. (1), the meaning of equation E_{P, \\mu, \\pi}[g] = .. := E[g], and the meaning of g in Eq. (9) are not very clear. \n      - The symbol “|” in probability represents conditioning. However, it seems to represent parameterized functions, for example,  \\eta(\\pi |p), b(s|p_i), b^*(s|P_j)\n     - In Theorem 2, \"i.f.f.\", the condition (Eq. (11)) seems to be sufficient but not necessary.\n     - “can obtain the minimum variance for DR” right above Section 4. I think Corollary 1 shows that state/environment-dependent baseline reduces more variance than state-dependent baseline, but I don’t think this implies state/environment-dependent baseline can obtain the minimum variance. \n  - The algorithm requires more explanation, such as, how the baselines are updated in Line 14, how the hierarchical cluster method works. And more description about relabel may help. \n\n### Other comments ###\n- Multiple symbols are randomly written in Italic or non-italic font, p in the second before the last line on Page 2 and a in the fifth before the last line in Section 2.\n-The meaning of N_C and N in Line 3, Algorithm 1. \n- What does $\\kappa = (\\kappa > 1)$ represent, right above Section 5?\n-  {p_c}_{c=1}^M  on the fourth line after Algorithm 1 box.  Should it be {p_c}_{c=0}^{H-1} instead?\n- hve -> have in the middle of page 9.\n\n### After rebuttal ###\nI really appreciate that the authors incorporated the comments in such a short amount of time. Although I think the clustering idea and experiments are interesting, due to the amount of changes made, I am slightly leaning toward suggesting the authors take some more time improving the manuscript and submit it to future conferences. For example, I think the experiment section can be improved with the new experiments (and discussions on the results) and more analysis on the proposed algorithm (e.g. how b* in the second paragraph on Page 5 is related to the algorithm proposed (we can not sample from (P(p|s)) and quantify the correlation between G(a, s) and Q(s, a, p)). \n", "summary_of_the_paper": "This paper tackles the variance of policy gradient due to the domain randomization used in RL in simulations. The authors prove that the policy gradient variance can be further reduced by learning a state-dependent baseline for each environment parameter compared to state-dependent baselines. The authors then develop a practical algorithm based on the analysis and analyze the properties of the algorithm. The algorithm is implemented and tested on six robot control tasks. It consistently accelerates policy training. ", "summary_of_the_review": "Because my concerns outweigh the strengths now, I am leaning towards rejecting the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aCFETA6HTk", "reviewer": "Reviewer_kFGn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This manuscript is a re-derivation of the control variate given that the randomness in the environment is partially artificial (domain randomization). The control variate induces a baseline that depends not only on the state but also on the environment. This term unsurprisingly has some traceability issue in practice and then the authors provide a divide-and-conquer idea to partially address it. Experiments show that this method is marginally better than vanilla domain randomization.\n\nPros:\n1. The paper tackles an interesting problem: how to reduce the variance and improve sample efficiency during the training of DR by developing a better baseline.\n2. The paper is clearly written in general.\n\nDiscussion:\n1. Theorem 1 is not useful. The term can be both positive and negative. I don't think control variates in RL has guarantees in variance reduction so I would suggest removing the theorem.\n2. The proposed method is based on the premise that the environment parameter is known and used to calculate the clustering prototypes during the training, which may not be valid for the real setting where the agent only can observe the environment.\n3. The convergence improvement in Figure 1 is not significant for some tasks like Pendulum and Pendulum2D yet with the additional cost of the clustering process.\n4. The paper uses the hierarchical cluster method to partition the environment space. Does the cluster method influence the experimental results? Why are other cluster methods like k-means not considered? And choosing a proper clustering interval seems nontrivial with grid search since the values in Figure 4 fall into different ranges for different tasks.\n5. The parameter Nc is used without definition in Alg 1.\n6. In section 5.2, the authors learn 15 policies for each algorithm on testing environments to test the generalization to the unseen environment. I guess that the authors wanted to express that they use the trained policies on testing environments directly.\n7. The paper uses the uniform domain randomization as a baseline, which is not the proposed method in [Mehta et al., 2020] (cited by the paper). Why does the paper not compare to the active domain randomization algorithm developed in [Mehta et al., 2020]?\n", "summary_of_the_paper": "This paper tackles the high variance problem caused by the randomization of environments for estimating the policy gradients. The idea is to derive a bias-free and state/environment-dependent optimal baseline for domain randomization. The authors further develop a VRDR method by dividing the entire environment space into subspaces and estimating the state/subspace-dependent baseline.", "summary_of_the_review": "This is a marginal improvement of domain randomization with marginal improvement in experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TownUggomtG", "reviewer": "Reviewer_LpEU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the mathematical derivation is sound. The idea of extending the state values as baselines to the additional parameterization of the environment variations is natural. Because the variance reduction technique in standard single-environment training is well-studied, the extension to the additional parameterization is relatively straightforward (the technical details are well-presented). \n\nMy main concerns are about the experimental results: I think the analysis is too weak and not enough baselines are compared with. By looking at Figure 1, it is quite hard to conclude that the proposed methods really help (whereas the standard variance reduction does make a crucial difference in training). I hope the authors delve more into the results and in particular elaborate on possible reasons when the benefits are unclear, rather than just a simple paragraph on a few environments where the method is marginally better (and I would guess there can be some examples where it is worse?). I do not believe these are all the baselines that should be considered, for instance there's no comparison with meta learning or robust RL methods. ", "summary_of_the_paper": "The paper derived an optimal state/environment-dependent baseline and a variance reduced domain randomization approach for policy gradient methods. ", "summary_of_the_review": "The paper is well written and the mathematical derivation is sound. My main concerns are about the experimental results: I think the analysis is too weak and not enough baselines are compared with. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers appreciated the clarity of the work, there is a concern about the meaning of the proposed result and method. It is known that adding knowledge about an additional variable, in this case the environment, leads to a lower variance estimate. What is not known is the practical impact of using this new baseline or perhaps some other intuition stemming from that use of the baseline (for instance the origin of the variance). However, the results shown are not that compelling, a point which was raised by the reviewers, making the work below the bar for publication.
Decision: Reject