Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
Authors: Jinxin Liu
Hao Shen
Donglin Wang
Yachen Kang
Qiangxing Tian
Venue: NeurIPS 2021 Poster (2021)
Keywords: unsupervised reinforcement learning
dynamics adaptation
sim2real

Abstract: Unsupervised reinforcement learning aims to acquire skills without prior goal representations, where an agent automatically explores an open-ended environment to represent goals and learn the goal-conditioned policy. However, this procedure is often time-consuming, limiting the rollout in some potentially expensive target environments. The intuitive approach of training in another interaction-rich environment disrupts the reproducibility of trained skills in the target environment due to the dynamics shifts and thus inhibits direct transferring. Assuming free access to a source environment, we propose an unsupervised domain adaptation method to identify and acquire skills across dynamics. Particularly, we introduce a KL regularized objective to encourage emergence of skills, rewarding the agent for both discovering skills and aligning its behaviors respecting dynamics shifts. This suggests that both dynamics (source and target) shape the reward to facilitate the learning of adaptive skills. We also conduct empirical experiments to demonstrate that our method can effectively learn skills that can be smoothly deployed in target. 

Unsupervised Domain Adaptation with Dynamics- Aware Rewards in Reinforcement Learning
Jinxin Liu 124
Hao Shen
Donglin Wang Yachen Kang 124 Qiangxing Tian
Zhejiang University. 2 Westlake University. 3 UC Berkeley.
Institute of Advanced Technology, Westlake Institute for Advanced Study.
liujinxin @ westlake.edu.com haoshen @ berkeley.edu, wangdonglin, kangyachen, tianqiangxing ) @westlake.edu.cn
Abstract
Unsupervised reinforcement learning aims to acquire skills without prior goal representations, where an agent automatically explores an open-ended environment to represent goals and learn the goal-conditioned policy. However, this procedure is often time-consuning. limiting the rollout some potentially expensive target environments. The intuitive approach of training in another interaction-rich envi- ronment disrupts the reproducibility of trained skills in the target environment due to the dynamics shifts and thus inhibits direct transferring. Assuming free access to a source environment, we propose an unsupervised domain adaptation method to identify and acquire skills across dynamics. Particularly, we introduce KL regularized objective to encourage emergence of skills, rewarding the agent for both discovering skills and aligning its behaviors respecting dynamics shifts. This suggests that both dynamics (source and target) shape the reward to facilitate the learning of adaptive skills. We also conduct empirical experiments demonstrate that our method can effectively learn skills that can be smoothly deployed in target.
Introduction
Recently, the machine learning community has devoted attention to unsupervised reinforcement learn- ing (RL) acquire useful skills, ie, the problem of automatic discovery of goal-conditioned policy and its corresponding goal space [8] As shown in Figure | (left), the standard training procedure of
learning skills in unsupervised way follows: (1) representing goals, consisting of automatically generating the goal distribution p(g) and the corresponding goal-achievement reward function ry: (2)
learning the goal-conditioned policy my with the acquired p(g) and rg. Leveraging fully autonomous interaction with the environment, the agent sets up goals, builds the goal-achievement reward function, and extrapolates the naal-conditioned policy in parallel by adopting off-the-shelf RL methods (40 19].
While we can obtain skills without any prior goal representations (p(g) and ryy) in an unsupervised way, major drawback of this approach is that requires large amount of rollout steps represent goals and learn the policy itself, together. This procedure is often impractical in some target environments (eg, the robot in real world), where online interactions are time-consuming and potentially expensive.
That said, there often exist environments that resemble in structure (dynamics) yet provide more accessible rollouts (eg, unlimited in simulators). For problems with such source environments available, training the policy in a source environment significantly reduces the cost associated with interaction in the target environment. Critically, we train a policy in environment and deploy another by utilizing their structural similarity and the excess of interaction. Considering the navigation in room,
learn arbitrary skills through the active exploration in source simulated
Work was done at Westlake University. Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The standard unsupervised RL: learning skills for the targetenv.
Target
Unsupervised domain adaptation RL
Source
Target
learning skills for the eavgetevv.
1. Representing goals:
1. Representing goals:
p(g)
a) Learning p in target env.
a) Leamingg p(g) in source and target.
p(g) Ta
T
b) Learning f in targetens env.
b) Learning ly source and target.
Tg
2. Leaming Ty in targetenv.
2. Leaming Ty in source
Figure 1: The training procedures of (left) the standard unsupervised RI. in a single target environment, and (right) the unsupervised domain adaptation in RI. with pair of source and target environments. p(g): the goal distribution; rg: the goal-achievement reward function; лe: the goal-conditioned policy.
room (with different layout or friction) before the deployment in the target room. However, it is reasonable to suspect that the learned skills overfit the training environment, the dynamics of which, dictating the goal distribution and reward function, implicitly shape goal representation and guide policy acquisition. Such deployment would then make learned skills struggle to adapt to new, unseen environments and produce large drop in performance in target due to the dynamics shifts, as shown in Figure 2 (top). In this paper, we overcome the limitations (of limited rollout in target and dynamics shifts) associated with the (source, larget) environments pair through unsupervised domain adaptation.
In practice, while performing full unsupervised RL method target that represents goals and captures all of them for learning the entire goal-conditioned policy (Figure Maauueeeeett left) can be extremely challenging with the limited rollout steps, learning a model for only (partially) representing goals is much easier. This gives rise to learning the policy in source and taking the limited rollouts in target into account only for identifying the goal representations, which further shape the policy. As shown in Figure [ (right), we represent goals in both environments while optimizing the policy only in the source environment, alleviating the excessive need for rollout steps in the target environment.
Furthermore, we introduce KI. regularization address the challenge of dynamics shifts. This objective allows us to incorporate reward mod- ification into the goal-achievement reward fune- tion in the standard unsupervised RL, aligning the trajectory induced the target environment against that induced in the source by the same policy. Importantly, it enables useful inductive biases towards the target dynamics: it allows the agent to specifically pursue skills that are competent in the target dynamics, and penalizes the agent for exploration in the source where the dynamics significantly differ. As shown in Figure 2 (bottom), the difference in dynamics wall in the target while wall in the source) will pose a penalty when the agent attempts to go through an area in the source wherein the get stands a wall. Thus, skills learned in source with such modification are adaptive to the target.
Source
Target
Source
Target
DARS Lhospeen commevviad RT.
Inmppryya Compervised.o RL.
Source
Targer
Targer
DARS
Figure 2: Skills learned in the source environment, each represented by distinct color, are deployed in the source and target respectively, Top plots de- pict states visited by the standard unsupervised RL method, where skills fail to run in the target envi- ronment. Bottom plots depict trajectories induced by policy Ty trained with our DARS, resulting in successful deployment in the target environment.
We name our method unsupervised domain adaptation with dynamics-awar rewards (DARS), sug- gesting that source and target dynamics both shape ry: (1) employ latent-conditioned probing policy in the source to represent goals [1,, making the goal-achievement reward source- orrented and (2) we adopt two classifiers [I]] to provide reward modification derived from the KL regularization. This means that the repertoires of skills are well shaped by the dynamics of both the source and target. Formally, further analyze the conditions under which our DARS produces near-optimal goal-conditioned policy for the target environment. Empirically, we demonstrate that our objective can obtain dynamics-awe aware rewards, enabling the condiionnt policy learned in source to perform well in the target environment in various settings (stable and unstable settings, and sim2real).
2 Preliminaries
Multi-goal Reinforcement Learning: We formalize the multi-goal reinforcement learning (RI.) as goal-conditioned Markov Decision Process (MDP) defined by the tuple Mg [S,A, P,R, 7, Po).

---

where S denotes the state space and A denotes the action space. P S x A S + Rzo is the transition probability density. Rs - To (G,,))())); where G' denotes the space of goals, T denotes the corresponding goal-achievement reward function rgg : G x S x A x S + IR, and p(g) denotes the given goal distribution. y is the discount factor and Pu is the initial state distribution. Given g ~ p(g), the didisouuteed return R(g, T) of a goal-oriented trajectory T = (8o, Gu. $1,...,. .8)) is DT- o "rrrrg(,,, at, 8t+1). Building on the universal value function approximators (UVFA, Schaul et al. (8). the standard multi-goal -ooal RI. seeks learn a unique goal-conditioned policy my x S G IR. [R(g,T) })}, where A denotes the parameter of the policy.
to maximize the objective
Unsupervised Reinforcement Learning: In unsupervised RL, the agent is set in open-endes environment without any pre-defined goals or related reward functions. The agent aims to acquire a repertoire of skills. Following Colas et al. [8) we define skills as the association of goals and the goal- conditioned policy to reach them. The unsupervised skill acquisition problem can now be modeled by goal-free MDP M = {S, P, Y, Pu } that only characterizes the agent, its environment and their possible interactions. As shown in (leff), the agent needs to autonomously interact with the environment and (1) learn goal representations (eg, discovering the goal distribution p(g) and learning the corresponding reward rg), and (2) learn the goal-conditioned policy To as in multi-goal RL.
Here we define universal (information theoretic) objective for learning the goal-conditioned policy Ty in unsupervised RL, maximizing the mutual information sp (9; 7) between the goal g and the trajectory t induced by policy To running in the environment M (with P and po).
max Ip to. (g: (g: = H(()) H(g'r) = HH()) + Ep po. [log((lr))
(1)
For representing goals, the specific manifold of the goal space could be set of latent variables (eg. one-hot indicators) or perceptually-spen specifie goals (eg. the joint torques of ant). In the absence of any prior knowledge about p(g), the maximum of H(g) will be achieved by fixing the distribution p(y) to be uniform over all g E G. The second term Pa.ppo.ss.ptl .pg)) (log p(g)T]] in Equation I is analogous to the objective in the standard multi-goal RL, where the return R(g. T) can be seen as the embodiment of log p(g[T) rr. The objective specifically for learning r9 in p(g[T) is normally optimized by lens of the generative loss (33 or the contrastive loss [42]. With the learned goal distribution p(g) and reward Tg it is straightforward to learn the conlicond policy To using standard RL algorithms 40 19]. In general, optimizations iteratively alternate for representing goals (including both goal-distribution p(g) and reward function rgg) and learning the goal-conditioned policy ma, as shown in Figure 1 (left).
3 Unsupervised Domain Adaptation with Dynamics-Awar Rewards
3.1 Problem Formulation
Our work addresses domain adaptation in unsupervised RL, raising expectations that an agent trained without prior goal representations (p(g) and rg)) in one environment can perform purposeful tasks in another. Following Wulfmeier et al. (54), we also focus on the domain adaptation of the dynamies, as opposed to states. In this work, we consider two environments characterized by MDPs Ms (the source environment) and My (the target environment), the dynamics of which are Ps and PT respectively. Both MDPs share the same state and action spaces S, A, discount factor and initial state distribution po. while differing in the transition distributions Ps, PT. Since the agent does not directly receive Rg from either environment, we adopt the information theoretic Ip Po. (g:T) to acquire skills, equivalently learning a goal-conditioned policy To that achieves distinguishable trajectory by maximizing this objective. For brevity, now omit the Pu term discussed in Section 2
In our setup, agents can freely interact with the source Ms. However, has limited access to rollouts in the target MT with which are insufficient to train policy. To ensure that all potential trajectories in the target My can be attempted in the source environment, we make the following assumption:
Assumption 1. There is no transition that is possible in the target environment My but impossible in the source environment Ms: PT((5+)) Stt, 04) > 0 - PPs(St41 18t, 0z) > 0.
3.2 Domain Adaptation in Unsupervised RL
We aim to acquire skills trained in the source environment Ms, which can be deployed in the target environment Mr. To facilitate the unsupervised learning of skills for the target environment MT
3

---

Ro
KL
KL
KL
Pr
Mg.
Ro,
Relabel
Relaba
(a)
(b)
(c1)
(c2)
Figure 3: Graphical models of (a) the standard unsupervised RI., and DARS with goals (b) directly in-
putted, (cl) relabeled with latent variable and (c2) relabeled with state induced by probing policy.
(with transition dynamics Py), we maximize the mutual information between the goal g and the trajectory т induced by the coal-conditione policy To over dynamics Py, as shown in Figure 3 (a):
I (g
However, since interaction with the target environment My is restricted, acquiring the goal- conditioned policy To by optimizing the mutual information above is intractable. We instead maximize the mutual information in the source environment Ipa.no (g: modified by a KL di- vergence of trajectories induced by the goal-conditioned policy me both environments (Figure 3b):
Ip (g: - 3DKL
(3)
where 9 >> 0 is the regularization coefficient, Prs (g. 7) and Pe (g, denote the joint distribu- tions of the goal g and the trajectory r induced by policy To in source Ms and target My respectively.
Intuitively, maximizing the mutual information term rewards distinguishable pairs of trajectories and goals, while minimizing the KL divergence term penalizes producing a trajectory that cannot be followed in the target environment. In other words, the KL term aligns the probability distributions of the nutuallinformaan Py. This indicates that the dynamics of both environments (Ps and PP) shape the goal- coal-conditioned policy mo (even though trained in the souree Ps), allowing my to adapt to the shifts in dynamics.
maximizing trajectories under the two environment dynamics Ps and
Building on the KI. regularized objective in Equation 3 we introduce how to effectively represent goals: generating the goal distribution and acquiring the (partial) reward function. Here we assume the difference between environments in their dynamics negligibly affects the goal distribution Therefore, we follow GPIM 33 and train a latent-conditioned probing policy TT. The probing policy тu explores the source environment and represents goals for the source to train the goal-conditioned policy with. Specifically, the probing policy म. is conditioned on a latent variable w ~ p(w and aims to generate diverse trajectories that are further relabeled goals for the conl-conditioned ma. Such goals can take the form of the latent variable w itself (Figure or the final state of a trajectory (Figure c2). We jointly optimize the previous objective in Equation with the mutual information between w and the trajectory 7 induced by T in source, and arrive at the following overall objective:
max [1,,0) Ipa. (w: + Ips. (g:T) DKL.. (p (g, (g,,))lPp (g,T))
(4)
where the context between p(g) and p(w) are specified by the graphic model in Figure (cl or c2). Note that this objective (Equation 4) explicitly decouples the goal representing (with T.) and the
policy learning (wrt me)), providing a foundation for the theoretical guarantee in Section 3.4
3.3 Optimization with Dynamics-Awar Rewards
Similar to Goyal et al. [6]. we take advantage of the data processing inequality (DPI 3 which
implies Ips.to (g: T) > Ir (w: r) from the graphical models in Figure (cl, c2). Consequently, maximizing Ipc. T): can be achieved by maximizing the information of w encoded progressively
ng. We therefore obtain the lower bound of Equation 4
I(, 0) 2 Ips.,, (w; + Ips (w; - 3KL (Prg (g. (0
(5)
2See Appendix D for extension when Ms and MT have different goal distributions. Following DIAYN 10 and DADS 43). set p(w) as fixed prior.

---

For the first term Ipe. (w; 7) and the second term Ip (wi we derive the state-conditioned Markovian rewards following Jabri al. [24]
1 T-1
[p:((w:)) 2 (H (wbeet)) = (w) + (ppr((ise))) ((w)(1))))1))
(6)
T H (w)) (p.eee) [log qo (a (w|s8+1))
where pr.5 (w, 8t+11 p( pr (8 (w), and Pp:= (8t+1 (w) refers to the state distribution (at time step + 1) induced by policy conditioned on w under the environment dynamics P; the lower bound in E derives from training a discriminator network qs due to the non-negativity of KL divergence, p. (#++1) [DxL (p(a (w/())))]((((1 (c+)))))) 2 O. Intuitively, the new bound rewards the discriminator 9á for summarizing agent's behavior with w as well as encouraging a variety of states.
With the bound above, we construct the lower bound of the mutual information terms in Equation 5 taking the same discriminator qa:
FI A Ips. (w; + Ip (w; 2 27 (w) + Ppa [log go (1+1) + log go(w (8+1)),
(8)
where Pjoint denotes the joint distribution of w, states 5r41 and 8-4-1- The states 8 and 8:411 induced by the probing policy म conditioned on the latent variable and the policy To conditioned on the relabeled goals respectively, both in the source environment (Figure c2).
Now, we are ready to characterize the KL term in Equation Note that only the transition probabilities terms (Ps and PT) differ since agent follows the same policy we in the environments. This con- veniently leads to the expansion of the KI. divergence term as a sum of differences in log likelihoods of the transition dynamics: expansion Pe. (g. Po I [P(s St, a (o St. g)l, where P (Ps, pr]. gives rise the following simplification the KL term in Equation
BDKL (Pos. (g,
mo (0. ()) (.
Epp. [3Ar(s ae st+1)l.
where the reward modification Ar(st, at st+1) log Ps ( az) - log Py (8e+ 1 az).
Combining the lower bound of the mutual informa- tion terms (Equation () and the KL divergence term pursuing the aligned trajectories in two environments (Equation 9). we optimize T (14, 8) by maximizing the following lower bound:
Same Limited Rollout
Rollout
Source Erw. n
Target Erv.
T
2H (w) + Eppiu [log (w(1))++1) + log go (w((s))1)
(4)). 9
Eps. [3Ar(8e, at, $e+1)]
(10)
BAr
A ssocited reward for me Tg log 9 - BAr
Overall, shown in Figure 4 DARS rewards the goal- conditioned policy To with the dynamics-awarr re- wards (associating log 96 with (Ar), where (1) log 96 shaped by the source dynamics Ps. and (2) 3Ar is derived from the difference of the two dynamics (Ps and P.)) This indicates that the learned goal- conditioned policy No is shaped by both source and target environments, holding the promise of acquiring adaptive skills for the target environment by training mostly in the source environment.
Figure 4: Framework of DARS: the latent- conditioned probing policy тμ provides p(g) and qs for learning coal-conditioned my. as- sociated with the reward modification BAr.
3.4 Optimality Analysis
Here discuss the condition under which our method produces near-optimal skills for the target environment. We first mildly require that the most suitable policy for the target environment My does not produce drastically different trajectories in the source environment Ms:
Assumption 2. Let x arg max, Ip i (g: T) be the policy that maximizes the (non-kl- noon-klreeularizedd objective the target environment (Equation 2). Then the joint distributions of the goal and its trajectories differ in both environments by no more than a small number e/ 0:
DxL. (Prgir) (g. 11p (g,7)) -
(II)
5

---

Algorithm DARS
1: Input: source and larget MDPs M,s and My;
11: Relabel goals: # According Figure (cl. c2)
ratio R of experience from source vs. target.
9 + (ws f).
2: Output: goal-reaching policy Tg.
12: Collect source data:
3: Initialize parameters μ4, 6, & and th.
Bs + Bs U (IL
4: Initialize buffers Bs. Bs and By.
13:
if iter mod R = 0 then
5: for iter = o.... MAX ITER do
14:
Collect target data:
6: Sample latent variable: w ~ p(w).
B7 <- BT ROLLOUT(To.
7: Collect probing data in source:
15:
end if
B5 + Bs U (T
16:
Update classifiers qu for computing Ar:
Update discriminator qu: & + Update( (o, Bs)
Update(1), (1) Bs, Br). (Equations) 12
9: Set reward function for the probing policy Tu:
17:
reward function
no:
rg f- log (w([((++)]] - BAr(s,, Ot, 8t+1).
f = log Ia (w]))+1).
18:
Train policy Te: A t SAC(O, f Bs, rg).
10: Train probing policy Tui μ t- SAC(μ,, Bs. f).
19:
end for
Given desired joint distribution p" [g, r) (inferred from a potential goal representation), our problem
can be reformulated as finding a closest match [29 (8 Consequently, we quantify the optimality of
a policy by measuring DKL (pp.x (g. r) I) (g. the discrepancy between its joint distribution and the desired one. With potential goal representation, we prove that its joint distributions with the trajectories induced by our policy and the optimal one satisfy the following theoretical guarantee.
Theorem 1. Let DARS be the optimal policy that maximizes the KL regularized objective in the source environment (Equation]) 3) let π be the policy that maximizes the (oon-regularize) objective in the target environment (Equation P. let p'+, (g,T) be the desired joint distribution of trajectory and goal in the target (with the potential goal representations), and assume that " satisfies Assumption 2 Then the following holds:
DEL (Per (g. T) 11p (9.T)) & DKL (p (g.T)) +
+2/
2c
Lmaa:
3
where Linaz refers to the worst case absolute difference between log likelihoods of the desired joint distribution and that induced by policy.
Please see Appendix C for more details and the proof of the theorem. Note that Theorem Trequires a potential goal representation, which be precisely provided by the probing policy nh in Equation 4
3.5 Implementation
As shown in Algorithm I we alternately train the probing policy тy and the goal-conditioned policy Ty by optimizing the objective in Equation with respect to μı, d, and Ar. In the first phase, we update Te with reward i = log (.41). This is compatible with most RI. methods and we refer to SAC here. We additionally optimize discriminator 4s with SGD to maximizing E,, &. [40 at the same time. Similarly, Ma is updated with r. log (w(w))(+1) BAr by SAC in the second phase, where no also collects (limited) data in the target environment to approximate Ar by training two classifiers qy (wrt state-action
(w(++++1)
g and state-action-state g in according Bayes' rule:
ffa (source)ss,, ae, 8c41]] + Ex, [log " (target|st, az, 8141)], max Esg |log 43
(12)
max Esg [log t (source) 8t, a.)] + Exs, [log o8 (target(s;,
(13)
Then, we have Ar(st, ae, 8141) = log 4 (source)ss, 2 (targtl
q3 (source)er.0e)) log
4 (urgei)es,as))
3.6 Connections to Prior Work
Unsupervised RL:
Two representative unsupervised RL approaches acquire (diverse) skills by maximizing empowerment [10 43 or minimizing surprise (.. Liu et al. [31 also employs a latent-
conditioned policy to explore the environment and relabels goals along with the corresponding reward,

---

Map-a
Map-b
Map-o
Map-d
Ami
Half chodtsh
Humned
Resson XTT
agent
aggend
agont
agent
callcheest
Rooon Iucranil
Figure 5: We evaluate our method in 10 (source, larget) transition tasks, where the shifts in dynamies are either external (the map pairs and the attacked series) or internal (the broken series) to the robot.
which can be considered as a special case of DARS with identical source and target environments. However, none of these methods can produce skills tailored to new environments with dynamics shifts.
Off-Dynamics RL: Eysenbach et al. [TI] proposes domain adaptation with rewards from classifiers (DARC), adopting the control inference framework [29] to maximize - -DKL (Pr (T) (T)), but this objective cannot be directly applied to the unsupervised setting. While we adopt the same classifier to provide the reward modification, one major distinction of our work is that we do not require a given goal distribution p(g) or a prior reward function ry. Moreover, assuming an extrinsic goal-reaching reward in the source environment (ie, the potential P (T)). our pro-
posed DARS can be simplified decoupled objective: maximizing -KI (P Te (T) Ps (T))
DKL (7) (T)). Particularly, DARC can be considered as a special case of our decou- pled objective with the restriction - a prior goal specified by its corresponding reward and / = In Appendix E, we show that the stronger pressure (3 >> 1) for the KL term to align the trajectories puts extra reward signals for the policy my to be Ar oriented while still being sufficient to acquire skills.
Related Work
The proposed DARS has interesting connections with unsupervised learning
and transfer most ap- proaches this field consider learning features 18 41 of high-dimensional (eg, image-based)
10
learning [55] in model-free RL. Adopting the self-supervised objective 126 39
states in the enviromment, then (1) adopt the non parametric measurement function acquire re- wards [23 33 |3
[6]] 16 [3] 22 over the learned features. the perception level (20), while focus the action level 20 wrt the transition dynamics of the environment, and we consider both cases (learning the goal-achievement reward function and enabling policy transfer between different environments).
44 32) or (2) enable policy transfer [23
These approaches can be seen
procedure
Previous works on the action level (20) have either (1) focused on learning manamics-orienter rewards in the unsupervised RL setting [21 51
31 or (2) considered the transition-oriented modification the supervised RL setting (given prior tasks described with reward functions or expert trajectories) [ [54 25 14
49
9 [22 30). Thus, the desirability of our approach is that the acquired reward function uncovers both the source dynamics (qa) and the dynamics difference (3Ar) across source and target environment. Complementary to work, several other works also encourage the emergence of a state-covering goal distribution [37 6 27] or enable transfer by introducing the
regularization over policies [45 15 46
36
48 instead of the adaptation over different dynamics.
5 Experiments
In this section, we aim to experimentally answer the following questions: (1) Can our method DARS learn diverse skills, in the source environment, that can be exccuted in the target environment and keep the same embodiment in the two environments? Specifically, can proposed associated dynamics-awar rewards (log 4s BAr) reveal the perceptible dynamics of the two environments? (2) Does DARS lead to better transferring in the presence of dynamics mismatch, compared to other related approaches, in both stable and unstable environments? (3) Can DARS contribute to acquiring behavioral skills under the sim2real cireumstances. where the interaction in the real world is limited?
We adopt tuples (source, target) to denote the source and target environment pairs, with details of the corresponding MDPs in Appendix F.2. Illustrations of the environments are shown in Figure 5 For all tuples, we set / = 10 and the ratio of experience from the source environment vs. the target environment R = 10 (Line 13 in Algorithm See Appendix F.3 for the other hyperparameters.
7

---

Map. We consider the maze environments: Map-a, Map-b, Map-c and Map-d, where the wall can block the agent (a point), which can move around to explore the maze environment. For the domain adaptation tasks, we consider the following five (source, larget) pairs: (Map-a, Map-b), (Map-a, Map-c), (Map-a, Map-d), (Map-b, Map-c) and (Map-b, Map-d).
Mujoco. We use two simulated robots from OpenAl Gym [5]: half cheetah (HC) and ant. We define two new environments by crippling one of the joints of each robot (B-HC and B-ant) as described in where B- is short for broken. The (source, target) pairs include: (HC, B-HCC) and (ant, B-ant).
Humanoid. In this environment, (source) simulated humanoid (H) agent must avoid falling in the face of the gravity disturbances. Two target environments each contain a humanoid attacked by blocks from fixed direction (A-H) and a humaniod with part of broken joints (B-H).
Quadruped robot. We also consider the sim2real setting for transferring the simulated quadruped robot to a real quadruped robot. For more evident comparison, we break the left hind leg of the real- world robot (see Appendix F.2). We adopt (sin-robot, real-robot) to denote this sim2real transition.
5.1
Emergent Behaviors with DARS
Map-a 3
Map-b
Map-h
Map-c
balf chorch
Ant broken aut (w-Y velecity yuw andie)
biill
Aillo skill
601-2
(a) (Map-a, Map-b)
(b) (Map-b, Map-c)
(c) (HC, B-HC)
(d) (ant, B-ant)
Figure 6: Visualization of skills. (a, b): colored trajectories in map pairs depict the skills, learned with DARS, deployed in source (left) and target (right). (c, d): colored bars and dots depict the velocity of each skill wrt different environments of mujoco and models. The variation (blue) across velocities for HC and ant corroborates the diversity of skills. DARS demonstrates its better adaptability by perform- ing similar skills on broken agents (green) to the original ones (blue) while DIAYN (orange) fails.
Visualization of the learned skills. We first apply DARS to the map pairs and the mujoco pairs, where we learn the goal-conditioned policy To in the source environments with our dynamics-aware rewards (log 9e 3Ar). Here, we relabel the latent random variable w the goal g for the goal- conditioned policy πेद g & Relabel(T w, F)
(Figure 3 cl). The learned skills are shown in Figures| 2 6 and Appendix F. We can see that the skills learned by our method keep the same embodiment when they are deployed in the source and target environments. we directly apply the skills learned in the source environment (without BAr), the dynamics mismatch is likely to disrupt the skills (see Figure D top, and the deployment of DIAYN in half cheetah and ant pairs in Figure 6).
Map-a
Map-b
Trajectorics
Apooctrre revards
Time (single rolloa))
Time voliono)
(a) Heatmaps of log 9o-
(b) Three trajectorics and and the associated rewards.
Figure 7: (a): The value of log 96 in Map-a for (Map-a, Map-b) and log qa in Map-b for (Map-b,
Map-c). (b): Three trajectories in Map-h for the (Map-b, Map-c) task, and the recorded rewards.
Visualizing the dynamics-aware rewards. To gain more intuition that the proposed dynamics- aware rewards capture the perceptible dynamics of both the source and target environments and enable an adaptive policy for the target, we visualize the learned probing reward log is and the reward modification AAr throughout the training for (Map-a, Map-c) and (Map-b, Map-c) pairs in Figure 7
The probing policy learns qa by summarizing the behaviors with the latent random variable w in source environments. Setting Map-a as the source (Figure 7 (a) left), we can see that log do resembles the usual I. 1.2-norm-based punishment. Further, in the pair (Map-b, Map-c), we can find that the learned log q6 is well shaped by the dynamics of the source environment Map-b (Figure 7 (a) right):
8

---

even if the agent simply moves in the direction of reward increase, it almost always sidesteps the wall and avoids the entrapment in a local optimal solution produced by the usual L2-norm based reward.
To see how the modification Bär guides the policy, we track three trajectories (with the same goal) and the associated rewards (log qs (3Ar) in the (Map-b, Map-c) task, as shown in Figure (b). We see that Traj.2 2 receives an incremental log de along the whole trajectory while a severe punishment from Ar around step 6. This indicates that Traj. 2 is inapplicable to the target dynamics (Map-c), even if it is feasible in the source (Map-b). With this modification, we indeed obtain the adaptive skills (eg. 3) by training in the source. This answers our first question, where both dynamics (source and target) explicitly shape the associated rewards, guiding the skills to be domain adaptive.
5.2
Comparison with Baselines
Map-a to Map-b
0.0
Map-b Mapsc
H H A-H
Humanoid pairs
Negaiiee torancerogoooo
A Mipiy
hoight
Lower vanaance more stabl: skll
-0.1
-0.2
2.08
-0.
B-H
-0.4
-0.3
Av had
0.01
0.4
0.6
100k
2008
0.00
Steps orgge Map.b
Steps oargel Map
100k
200k
500k
DARS
GPS in tagect
GIMM in (IC)
Skill mouce
Steps burget cov. (A B-H)
GIM oeget X10
GIMM cOurE (TND
Skill argott (SMIRL)
SMIRL Faadtusing (A
DARS (A-8I)
DARC
GPM limenenng
Skil arget (DARS)
SMIRL
Flnduuiig (B H)
DARS H)
Figure 8: Comparison (training process) with alternative methods for learning skills for target environments. We plot each random seed as a transparent line; each solid line corresponds to the average across four random seeds; the dashed lines denote the performance of trained policies.
Figure (left): The visualization of skills for hu- manoid (avoid falling) and the comparisons with SMiRL Finetuning, where the stable skills for hu- manoid keep the average height around 1. (right): Training process. The decrease in the variance of the height implies the emergence of stable skill.
Behaviors in stable environments. For the second question, we apply our method to state-reaching tasks: 9 A Relabel( (T w. f) = šy (Figure c2). We adopt the negative L2 norm (between the goal and the final state in each episode) as the distance metric. We compare our method (DARS) against six alternative goal-reaching strategie
Fl (1) additionally updating me with data By collected in the based reward (DARC L2); training skills with GPIM in the source and target respectively (3) GPIM in source and (4) GPIM in target); (5) updating GPIM in the target 10 times more (GPIM in target XI0; R = 10 and see more interpretation in IT); (6) finetuning GPIM in source in the target (GPIM Finetuming in target)
target (DARS Reuse); (2) employing DARC with a negative
We report the results in Figure 8 GPIM in source performs much worse than DARS due to the dynamics shifts as we show in Section 5.T With the same amount of rollout steps in the target, DARS achieves better performance than GPIM in larget X10 and GPIM Finetuning in larget, and approximates GPIM in target within 1M steps in effectiveness, suggesting that the modification 3år provides sufficient information regarding the target dynamics. Further, reusing the buffer By (DARS Resue) does not significantly improve the performance. Despite not requiring a prior reward function, unsupervised DARS reaches comparable performance (supervised) DARC L2 in (Map-a, Map-b) pair. The more exploratory task (Map-b, Map-c) further reinforces the advantage of our wyyaamics-awar rewards, where the probing policy T. boosts the representational potential of qs-
Behaviors in unstable environments. Further, when we set p(w) as the Dirac distribution, p(w) = 6(w), the discriminator da will degrade to density estimator: qo (8(+))1). which keeps the same form as in SMiRL.
Assuming the environment will pose unexpected events to the agent, SMiRI seeks out stable and repeatable situations that counteract the environment's 's prevailing sources of entropy.
With such properties, we evaluate DARS in unstable environment pairs, where the source and the target are both unstable and exhibit dynamics mismatch. Figure (left) charts the emergence of a
*we do not compare with other unsupervised RL methods (eg. Warde-Farke et al. (33) because they generally study the rewards wrt the high-dimensional states. DARS does not focus on high- sigeedimensional states. Domain randomization [35 50) and system (dynamics) identification [12] [ are also not compared because they requires the access of the physical parameters of source environment, while we do not assume this access.

---

Moving forward (Full real))
Failure x
Keeping belance. (Full-in-res
Failure x
Moving forward. (Finetuming)
Failure
Keeping helance (Fineuming)
Failure
Moving forward (DARS)
Success
Keeping bxlancee (DARS)
Success
Deploying the learned skills into the real quadruped robot, where all models are trained with limited interaction (three hours for moving forward and one hour for keeping balance) in real.
Figure 10:
stable skill with DARS, while SMiRL suffers from the failure of domain adaptation for both (H. A-H) and (H, B-HI). Figure 9 (right) shows the comparisons with SMiRL Finetuning, denoting training in the source and then finetuning in the target with SMiRL. With the same amount of rollout steps, can find that DARS can learn a more stable skill for the target than SMiRL Finetuning, revealing the competence of our regularization term for learning adaptive skills even in the unstable environments.
5.3 Sim2real Transfer on Quadruped Robot
We now deploy our DARS on pair (sim-robot, real-robot) to learn diverse skills (moving forward and moving backward) and balance-keeping skill in stable and unstable setting re- spectively. We compare DARS with baselines: (1) train- ing directly in the real world (Full-in-raa),, (2) finetuning the model, pre- trained in simulator, in real (Finetuning). As shown in Figure TO after three hours (or one hour) of real-world interaction, our DARS demonstrates the emer- gence of moving skills (or the balance- balance-keeping skill), while baselines are unable to do so. As shown in Table Fine- tuming takes significantly more time (four hours vs. one hour) to discover valance-keeping skill in the unstable setting, and the other three comparisons are unable to acquire valid skills given six hours of interaction in the real world. Supplementary material contains videos from this sim2real deployment.
Table 1: Time (hours) spent for valid skill emergence real-world interac- tion (covering the manual reset time).
forward & keeping backward
balance
Full-in-real Finetuning
6 h
6 h 4
> 6 h
DARS
3
1
6 Conclusion
In this paper, we propose DARS to acquire adaptive skills for a target environment by training mostly a source environment especially the presence of dynamics shifts. Specifically, we employ latent-conditioned policy rollouting in the source environment to represent goals (including goal-distribution and acheeechieeeeen reward function) and introduce KI. regularization to further identify consistent behaviors for the coal-conditioned policy in both source and target environments. We show that DARS obtains a near-optimal policy for target, as long as mild assumption is met. We also conduct extensive experiments to show the effectiveness of our approach: (1) DARS can acquire dynamics-awarr rewards, which further enables adaptive skills for the target environment, (2) the rollout steps in the target environment can be significantly reduced while adaptive skills are preserved.
Acknowledgments and Disclosure of Funding
The authors would like to thank Hongyin Zhang for help with running experiments on the quadruped robot. This work is supported by NSFC General Program (62176215).
References
[1] Adam Allevato, Flaine Schaertl Short, Mitch Pryor, and Andrea Thomaz. Tunenet: One-shot residual tuning for system identification and sim-to-real robot task transfer. In Leslic Pack
10

---

Kaelbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 November 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 445 455. PMIR, 2019. URI
http://proceedings..rss .ml.iiinnsmmr.....mmm.mm.. http://g......ppeeee//o////attaa
[2] Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. arXiv preprint arXivv/906.000 2019.
[3] Normand J. Beaudry and Renato Renner. An intuitive proof of the data processing inequality, 2012.
[4] Glen Berseth, Daniel Geng, Coline Devin, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Smirl: Surprise minimizing RL in dynamic environments. CoRR, abs/1912.05510,, 2019. URL http://arxiv.. org. abs 1912.05510
[5] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. 2016.
[6] Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i-Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills, 119: 1317-1327, 2020. URL http::/pproceedings mlr press/vii91 ppeps///111/ccmmos222e
campos20a. html
[7] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan D. Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019, pages 8973 8979. IEEE, 2019. doi: 10.1 109//A./ 2019.8 8019.8793789. URL pottps:://oo. org/ 10. 1109/ ICRA 2019. 87937789
[8] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Y Vver Oudeyer. Intrinsically motivated goal-conditioned reinforcement learning: short survey. CoRR, abs/2012.09830, 2020. URL
https://arxiv wwg.
09830
[9] Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems, 33, 2020.
[10] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06077 2018.
[11] Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Ruslan Salakhutdinov, and Sergey Levine. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arXiv preprint (xXix22006.19116 2020.
[12] Alon Farchy, Samuel Barrett, Patrick MacAlpine, and Peter Stone. Humanoid robots learning to walk faster: From the real world to simulation and back. In Proceedings of the 2013 international conference Autonomous agents and multi-agent systems, pages 39-46, 2013.
[13] Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojeiech M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized rl, 2019.
[14] Tanmay Gangwani and Peng. State-only imitation with transition dynamics mismatch. ArXiv, 18872 2020.
[15] Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning, 2018.
[16] Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew Botvinick, Hugo Larochelle, Yoshua Bengio, and Sergey Levine. Infobot: Transfer and exploration via the information bottleneck, 2019.

---

[17] Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng, Sergey Levine, and Yoshua Bengio. Reinforcement learning with competitive ensembles of information- primitives. In International Conference on Learning Representations, 2020. URL https hetp://wwweeetoommmm/dxxgxxgogo
[18]
Zhaohan Daniel Guo. Bernardo Ávila Pires, Bilal Piot, non-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent-predictive representations for multitask reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3875-3886. PMLR, 2020. URL http::/pproceeddngg/ mlr. paress//119/guo2gg..o hhmm
[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint 181. 2018.
[20] Danijar Hafner, Pedro A. Ortega, Jimmy Ba, Thomas Parr, Karl Friston, and Nicolas Heess. Action and perception as divergence minimization. CoRR, hos/2009.0179) 2020. URL petps::/arxivy: htt///aivv.oggaaaaaa.ee abt.//b.//0000001111
[21] Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. arXiv preprini arXiv: 10Xiv:1907.08225. 2019.
[22] Leonard Hasenclever, Fabio Pardo, Raia Hadsell, Nicolas Heess, and Josh Merel. Comic: Complementary task learning & mimicry for reusable skills. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4105-4115. PMLR, 2020.
URL httpp://rroceedings http://wp.eeee//////////aaaaaaaa 1tte1//eeeeeeeeeeeee001 html
[23] Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volumee 70, pages 1480-1490 JMLR. org, 2017.
[24] Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Un- supervised curricula for visual meta-reiiforremment learning. In Advances in Neural Information Processing Systems, pages 10519-10531, 2019.
[25] Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and S. Ermon. Domain adaptive imitation learning. In ICML, 2020.
[26] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
[27] G. Kovač, A. Laversanne- Finot, and Pierre Yves Oudeyer. Grimgep: Learning progress for robust goal sampling in visual deep reinforcement learning. arXiv: Learning, 2020.
[28] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching, 2020.
[29] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018 http:p/t///axiii http://www..rr/a////...... httg////s/10550000
[30] Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment- based imitation learning. arXiv preprint 1911 2019.
[31] Jinxin Liu, Donglin Wang, Qiangxing Tian, and Zhengyu Chen. Learn goal-conditioned policy with intrinsic motivation for deep reinforcement learning, 2021. UtR.lattpps//oopeeevvee
eet//oorum2dd-Mocyyoororr
[32] Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong, Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervised robotic learning. In Conference on Robot Learning, pages 530-539. PMLR. 2020.
12

---

[33] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Vi- sual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pages 9191-9200, 2018.
[34] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:180..0m 1807.037447 2018.
[35] Xue Bin Peng. Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamies randomization. 2018 IEEE International Conference on Robolics and Automation (ICRA), May 2018. doi: 10. 10.11009/era.21188884400 288668000..8.8 URL
http://dx.doi. dow. ht.. 11..:..1110IRRAAA 2018. 8460528
[36] Janith C. Petangoda, Sergio Pascual-Diaa Vincent Adam, Peter Vrancx, and Jordi Grau-Moya. Disentangled skill embeddings for reinforcement learning. CoRR, abs/1906.0922) 2019, URL
bstp:://arxiv.orgg www.g/abb/19955 www.1999222000
[37] Vitchyr H. Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning, 2020.
[38] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxi- mators. In International conference on machine learning, pages 1312-1320, 2015.
[39] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: unified embedding for face recognition and clustering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 815-823, 2015.
[40] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[41] Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictiv representations. 2021.
[42] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1134-1141. IEEE, 2018.
[43] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. 2020. URL HJgLZR4KvH
[44] Avi Singh, Larry Yang, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforce- ment learning without reward engineering. In Antomio Bicchi, Hadas Kress-Gazit, and Seth Hutchinson, editors, Robotics: Science and Systems XV, University of Freiburg, Freiburg im Breisgau, Germany, June 22-26, 2019, 2019. doi: 10.15607/RSS..019.XVV00 https: /ttps:///do.... /ttp::///ooo../////// 10. 15607 /RSS 2019 .03
URL
[45] Daniel Strouse, Max Kleiman- Weiner. Josh Tenenbaum, Matthew Botvinick, and David J. Schwab. Learning to share and hide intentions using information reg- URL https:://proeeddii html
ularization, 2018.
[46] Yee Whye Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforce- ment learning, 2017. URL hattps://proceeding http://poooeeddnngg.eeiiieeppppp httppppss..ppappp/////////a///
Balc60a00100aeescc11881115555 htt.://ccc..ee11eaaaaaa--tttttta Abstract.html
[47] Qiangxing Tian, Guanchu Wang, Jinxin Liu, Donglin Wang, and Yachen Kang. Independent skill transfer for deep reinforcement learning. In IJCAI, pages 2901-2907, 2020.
[48] Qiangxing Tian, Jínxin Liu, Guanchu Wang, and Donglin Wang. Unsupervised discovery of transitional skills for deep reinforcement learning. In 202/ International Joint Conference on Neural Networks (IJCN), pages 1-8. IEEE, 2021.
13

---

[49] Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Model-based visual planning with self supf-vupe functional distances. CoRR, abs/2012. 153733, 2020.
a0tg/2bss/01...... 2012. 15373
[50] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017.
[51] Srinivas Venkattaramanujam, Eric Crawford, Thang Doan, and Doina Precup. Self- supervised learning of distance functions for goal-conditioned reinforcement learning. CoRR, abs/1907.02998. 2019. URL htttp://arxxv.org//bbb//// 1907 02998
[52] Luca Viano, Y. Huang, Kamalaruban, and V. Cevher. Robust inverse reinforcement learning under transition dynamics mismatch. ArXiv, abs/2007.01174, 2020.
[53] David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards, 2018.
[54] Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. In Conference Robot Learning, pages 281-290. PMLR, 2017.
[55] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE Symposium Series on Computational Intelligence, SSCI 2020, Canberra, Australia, December 1-4, 2020, pages 737-744. IEEE, 2020. doi: 10. 100/
URL. https: / htt////i.ooo/// org/ 10. /
2020. 9308468
14


Reviews:
[{"review_id": "_ct4yXOJMN", "reviewer": "Reviewer_UYK4", "summary": "This paper presents an approach for unsupervised learning of a goal-conditioned policy that is robust to changes in dynamics and can be transferred to a target domain that is different from the source domain in terms of the underlying dynamics. There are several components. First, a latent-conditioned probing policy is used to generate goals (either in the source or target domain). This probing policy is trained to generate diverse goals by maximising the information gain between the goals and the resulting trajectories generated by the policy, i.e. we want diverse goals that are predictable given the resulting trajectories. This “predictability” is approximated using a discriminator that scores the likelihood of the goal latent given the resulting trajectory; this is also trained alongside the goal-generating policy based on rollouts in the source domain. \n\nNext, the probing policy is used to generate goals in the source and target domains (these are either the latents themselves or the final states of trajectories from the probing policy), and the goal-conditioned policy (which is conditioned on these goals) is executed in both these domains to generate rollouts. A key assumption here is that it is costly to execute the policy in the target domain (e.g. real world), so the approach tries to minimise this by training the probing policy, discriminator and to a major extent, the policy, primarily on the source domain. The policy is again trained to maximise the log-likelihood of the discriminator, thereby leading to predictable trajectories in the source domain. To ensure that the policy does not overfit to the source domain, and consequently fail on the target domain, an additional KL regularisation term is added to the policy training objective. This regularisation penalises trajectories that provide inconsistent results between the source and target domains, as measured by the KL divergence of the resulting transition distributions when executing the policy for the same goal in both these domains. This effectively ensures that the policy learns trajectories that are executable consistently in both domains, thereby ensuring success even in the presence of differences in dynamics between the two domains. Note that two key assumptions that are necessary for this to work (and the theoretical results) is that there is no transition in the target domain that is not possible in the source domain, and that the goal distribution in the source is also valid in the target (a relaxation of the latter is provided in the supplementary material). This regularisation term is approximated by a pair of classifiers that are trained on data from both domains; these are also trained alongside the policy. The final algorithm alternates training the probing policy & discriminator that parameterise the goal distribution and rewards against training the goal-conditioned policy and the classifiers for approximating the dynamics difference KL.\n\nThe approach is tested on several simulated tasks of varying complexity. First, the approach is tested on a 2D point mass with pairs of source and target tasks where the arena has walls of different configurations. Next, the approach is evaluated on continuous control tasks from the OpenAI Gym with the ant, half-cheetah and humanoid embodiments. The source tasks are the standard gym tasks and the target tasks have certain joints not working or additional perturbations in the form of obstacles. Lastly, the approach is also tested in a sim2real setup with a quadruped; the source task is in simulation while the target task is on the real robot. The resulting policy and goal distribution show good performance on all these tasks; the skills and goal distribution learned by the method is able to successfully transfer across changes in wall configuration, joint issues, stability problems and also to real world robots. Several visualisations of different parts of the algorithm (e.g. discriminator, skills) etc. are provided along with comparisons to baselines, namely variants of DARC, SMiRL, DIAYN and GPIM. The supplement also provides additional analysis and proofs of the theoretical guarantees of the near-optimality of the proposed method under the pertinent assumptions.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an approach to unsupervised RL that trains a goal-conditioned policy which is capable of adapting to changes in dynamics between a source and target domain. A key advantage of the method is that it is able to train in an unsupervised fashion in target domains where data collection can be hard or expensive. This is done by training the goal distribution, goal-based reward function and goal-conditioned policy primarily on the source domain where data collection is inexpensive, while regularising for differences in dynamics between the two domains. This regularisation encourages the policy to find skills that lead to consistent results in the source and target domains while utilising an order of magnitude lower amount of data from the target domain. This is particularly promising for real world robotics, especially in the context of sim2real methods, where simulated data is easy and cheap to generate but has differences in dynamics to the real world where data is also expensive to generate. \n\nThe approach is tested on several simulated and real world source and transfer tasks and shows promising results. There are several assumptions that are key for the method to work. Key amongst those is the assumption that there is no transition that is possible in the target domain but impossible in the source domain; interestingly, one of the point-mass experiments does violate this assumption and while the approach does work on this transfer setting it effectively over-constrains the search space leading to a potentially sub-optimal solution. It would be interesting to see if this assumption can be relaxed. Another important assumption is that the goal distribution is consistent under dynamics changes between the source and target domains; an approach to relax this assumption is derived in the supplementary material wherein the goal distribution training can be regularised to respect the source/target dynamics changes and is validated with a set of experiments (it would be useful in this context to see (Map-a, Map-e) discriminator visualisations in Figure 6). \n\nOverall, I am quite encouraged by the results presented in the paper, which are strong and show that the method is able to do well on a variety of control tasks both in simulation and in the real world. I am particularly excited about the applications to sim2real settings where the data limitations can make learning quite hard. It would be interesting to see if further reductions in the data is possible, and it would be great if there was an ablation study with respect to the number of real world data points used (i.e. fraction of target domain data) on some of these tasks. To add further clarity to the paper it would be useful if the approach for training the classifiers to estimate the dynamics regularisation is also detailed either in the main text or supplementary material. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have clearly explained the limitations of the approach, explaining in detail the assumptions that are needed to get the approach working (and for the theoretical results), the resulting restrictions on the method and potential relaxations for some of the assumptions with experimental results for these relaxations. Brief discussions of the broader impact is provided in the supplement, though some more details could be provided with regards to the social impact (especially since the approach is applicable for sim2real).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "SkFAA7gGT-B", "reviewer": "Reviewer_3aSe", "summary": "The paper handles the problem of domain adaptation in RL, specifically the setting where rewards and prior goal representations are not available. They do so by combining a skill learning method (in the source environment) with a transfer learning method that aims to match the joint $(goal, \\tau)$ distribution arcoss environments. By doing so they show that skills can both be learned in the source domain, and they transfer more easily to the target domain.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors want to use domain adaptation to learn a policy that will perform well in a target environment, assuming access to a source environment where it is easier to get many samples. In addition, the authors want a method that will do so without access to rewards in either the source or target environment.\n\nThey propose a method called DARS, standing for dynamics-aware rewards. It is based off 2 primary components. First, there is a skill learning policy. For this policy, they use a common trick in the literature: an exploratory policy $\\pi$ is conditioned on some latent skill vector $\\omega$. As trajectories are generated, a learned $q_\\phi(\\omega|s)$ acts as a discriminator that tries to identify $\\omega$ from the episode. Intuitively, if the mutual information between $\\omega$ and episode $\\tau$ is maximized, then different $\\omega$ should lead to distinguishable trajectories in the environment. (The specific citations for this are DIAYN and GPIM but something analogous to this has shown up in other recent papers as well.)\n\nThey then combine this with a dynamics based reward. The aim is to match distributions $(goal, \\tau)$ across the source environment and target environment, given the same policy $\\pi$. Through some derivation, you can show that when taking the KL divergence $p(g,\\tau)$, all terms between the two cancel out except for $p_{source}(s_{t+1}|s_t,a_t)$ and $p_{target}(s_{t+1}|s_t,a_t)$. The ratio of these probabilities can then further be approximated by learning 2 classifiers $p(target|s_t,a_t,s_{t+1})$ and $p(target|s_t,a_t)$. This closely follows the Domain Adaptation with Rewards from Classifiers (DARC) paper.\n\nCombined, they demonstrate that skills learned in the source domain transfer better to the target domain.\n\nI found the presentation in this paper a bit dense, although that is to be expected since they are aiming to explain 2 papers worth of ideas in 1. At a surface level, I personally found it confusing that $\\Delta r$ was defined opposite to how it was defined in the DARC paper (i.e as (source - target)), and I initially thought this was an error until I noticed that it was subtracted instead of added to the reward. I think negating it and treating $\\Delta r$ as an additive reward bonus is slightly more intuitive.\n\nThis is a fairly minor issue, my more detailed reservation with the paper is that it doesn't feel like a very interesting research contribution. To me, it feels like it is primarily about taking DARC then replacing the source rewards with a skill learning method. The 2 working together doesn't seem novel to me because they are somewhat orthogonal methods (one is skill learning in 1 env and the other is transfer learning between 2 envs). Both are based on defining rewards for an RL method to optimize. A priori I would assume the two would work together as long as you balanced their reward objectives appropriately, and this is basically what the paper shows (adjusting the $\\beta$ weight is what affects whether the method works or not). Moreover it doesn't seem like their comparisons are very strong - they are essentially comparing against a skill learning / intrinstic motivation (SMiRL, GPIM), or they are comparing against a transfer learning method (DARC with a proxy reward), and again it seems like the method is supposed to work if tuned properly. And it does, by a margin that appears small.\n\nTo me, if it a priori is supposed to work, and the improvement is fairly small, then I'm not sure the paper needs to exist. This is especially true because the target problem they are solving feels narrow, as it is the intersection of 2 other problems. They aim to solve \"transfer learning, between 2 environments, where we have ready access to 1 and not the other, but we also don't have rewards in the source env we have ready access to\", and this doesn't feel like a common scenario of interest, which weakens significance.\n\nWithout an especially strong empirical result, to be strong enough for acceptance the paper needs some argument that the combination of the two is better than the sum of its parts. It attempts to argue this by saying that the dynamics from the target domain are affecting the skill learning process (in the Map env visualizations). I think this is basically just a consequence of \"weighting the objectives correctly\" - in the limit, at $\\beta = 0$ you recover DIAYN and at $\\beta = \\infty$ you recover DARC, so at some $\\beta$ in between you expect to see episodes that 1) are closer to the target episodes but 2) still have diversity within that restricted part of state space, and that's effectively what we see here.\n\nEdit: I have read the other reviews and rebuttals. Please see the comment later in the thread for more detailed replies. In short,\n\n* I agree that $\\beta$ does not inherently limit diversity of skills, it is primarily about alignment in the target domain. This may reduce diversity of states that are reached in the source domain, but this is fine because the end goal is to learn diverse skills for the target domain - anything that is closed off by the $KL$ objective is something we should not learn anyways.\n* One of my comparisons was comparing an unsupervised (no observed reward) method against a supervised (observed reward) method, which was a mistake on my part.\n* After some more consideration, I believe the problem the paper aims to solve is more important than I previously thought.\n\nThese were significant misunderstandings so I am updating from 4 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2hr 15min, additional 3 hours post rebuttal."}, "extra_text": {}}, {"review_id": "FdM4399aJR0", "reviewer": "Reviewer_kXzn", "summary": "The authors address the problem of unsupervised Domain Adaptation in Reinforcement Learning. Combining ideas from previous work, they devise a method able to utilize an easy to sample source environment (e.g., a simulator) in order to learn a set of skills without supervision, while at the same time guarantee that these skills will be suited to the target environment (e.g., a real robotic system). To achieve this, they modify the reward structure of the typical Unsupervised Reinforcement Learning setup to enable the agent to build skills that are compatible with the target environment dynamics, while penalizing exploration in regions of the source environment where the dynamics differ.\nThe proposed approach is benchmarked against state-of-the-art algorithms in a wide variety of environments with increased complexity and manages to outperform previous baselines both in simulation as well as in three walking tasks on a real quadruped robot.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors address the problem of unsupervised Domain Adaptation in Reinforcement Learning. Combining ideas from previous work, they devise a method able to utilize an easy to sample source environment (e.g., a simulator) in order to learn a set of skills without supervision, while at the same time guarantee that these skills will be suited to the target environment (e.g., a real robotic system). To achieve this, they modify the reward structure of the typical Unsupervised Reinforcement Learning setup to enable the agent to build skills that are compatible with the target environment dynamics, while penalizing exploration in regions of the source environment where the dynamics differ.\nThe proposed approach is benchmarked against state-of-the-art algorithms in a wide variety of environments with increased complexity and manages to outperform previous baselines both in simulation as well as in three walking tasks on a real quadruped robot.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The first part of the paper starts with a very clear and well-written introduction in Sections 1 and 2 that introduces the problem and the proposed approach. Figures 1 and 2 provide an excellent overview and help understand the intuition behind the setup and the proposed algorithm. This holds for Sections 3.1 and 3.2 (aided by the visual explanation in Figure 3), where the problem is described in detail along with the derivation of the modified objective in Equation 4.\nUnfortunately, Sections 3.3 – 3.5 which describe the details of the proposed approach are difficult to follow, as the authors omit critical information. The definition and purpose of the discriminator q_\\phi is never explained (or discussed). The same goes for the two classifiers q_\\psi: they are introduced directly in Algorithm 1 and referenced in one line in Section 3.5. Also, it is not clear by reading the text how they are used to shape the reward \\Delta r. If the reader follows the references and reads the relevant papers, these design choices become clear, but the paper should be made more self-contained.\nI would recommend the authors to bring Figure 10 of Section F.1. of the supplementary material in the main text and use it to explain what is the role of the discriminator, how the classifiers are trained and how they are used to calculated \\Delta r. \nThe experiments Section is quite dense and difficult to follow at some points. For example, I would recommend moving the comparisons shown in Figures 8 and 9 in the appendix and expand the discussion there a bit.\n\nPros:\n- Very well written for the most part (see discussion on problematic Sections above)\n- Figures providing intuition on the proposed method, as well as visualization of skills (both trajectories and heatmaps) are very well thought and complement the text in an excellent way\n- Very extensive experimental evaluation and impressive results on real robot\n\nLimitations:\n- One of the main limitations is that the approach requires full knowledge and a-priori access to the target environment, while at the same time assuming that the goal distribution is the same between the two. Nevertheless, the authors discuss these limitations in Appendices B and D providing insights and extensions to the algorithm.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers highly appreciated the author response and the additional details. Reviewer UYK4 didn't update the review but indicated in the discussion to have read the replies.
We had quite an extensive discussion between the reviewers on whether the setting of this paper too narrow and whether that would actually speak against publishing it. In the end they came up with a few examples (please add some motivating examples that have practical relevance to the paper) and everybody was convinced that this is a sound and interesting paper.
Decision: Accept (Poster)