Unsupervised Domain Adaptation with Dynamics- Aware Rewards in Reinforcement Learning
Year: 2021

Abstract:
Unsupervised reinforcement learning aims to acquire skills without prior goal representations, where an agent automatically explores an open-ended environment to represent goals and learn the goal-conditioned policy. However, this procedure is often time-consuming, limiting the rollout in some potentially expensive target environments. The intuitive approach of training in another interaction-rich environment disrupts the reproducibility of trained skills in the target environment due to the dynamics shifts and thus inhibits direct transferring. Assuming free access to a source environment, we propose an unsupervised domain adaptation method to identify and acquire skills across dynamics. Particularly, we introduce a KL regularized objective to encourage emergence of skills, rewarding the agent for both discovering skills and aligning its behaviors respecting dynamics shifts. This suggests that both dynamics (source and target) shape the reward to facilitate the learning of adaptive skills. We also conduct empirical experiments to demonstrate that our method can effectively learn skills that can be smoothly deployed in target.

heading: 1 Introduction
text: Recently, the machine learning community has devoted attention to unsupervised reinforcement learning (RL) to acquire useful skills, ie, the problem of automatic discovery of a goal-conditioned policy and its corresponding goal space [8]. As shown in Figure 1 (left), the standard training procedure of learning skills in an unsupervised way follows: (1) representing goals, consisting of automatically generating the goal distribution p(g) and the corresponding goal-achievement reward function rg; (2) learning the goal-conditioned policy πθ with the acquired p(g) and rg . Leveraging fully autonomous interaction with the environment, the agent sets up goals, builds the goal-achievement reward function, and extrapolates the goal-conditioned policy in parallel by adopting off-the-shelf RL methods [40, 19]. While we can obtain skills without any prior goal representations (p(g) and rg) in an unsupervised way, a major drawback of this approach is that it requires a large amount of rollout steps to represent goals and learn the policy itself, together. This procedure is often impractical in some target environments (eg, the robot in real world), where online interactions are time-consuming and potentially expensive.
That said, there often exist environments that resemble in structure (dynamics) yet provide more accessible rollouts (eg, unlimited in simulators). For problems with such source environments available, training the policy in a source environment significantly reduces the cost associated with interaction in the target environment. Critically, we can train a policy in one environment and deploy it in another by utilizing their structural similarity and the excess of interaction. Considering the navigation in a room, we can learn arbitrary skills through the active exploration in a source simulated
∗Work was done at Westlake University. † Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
room (with different layout or friction) before the deployment in the target room. However, it is reasonable to suspect that the learned skills overfit the training environment, the dynamics of which, dictating the goal distribution and reward function, implicitly shape goal representation and guide policy acquisition. Such deployment would then make learned skills struggle to adapt to new, unseen environments and produce a large drop in performance in target due to the dynamics shifts, as shown in Figure 2 (top). In this paper, we overcome the limitations (of limited rollout in target and dynamics shifts) associated with the (source, target) environments pair through unsupervised domain adaptation.
In practice, while performing a full unsupervised RL method in target that represents goals and captures all of them for learning the entire goal-conditioned policy (Figure 1 left) can be extremely challenging with the limited rollout steps, learning a model for only (partially) representing goals is much easier. This gives rise to learning the policy in source and taking the limited rollouts in target into account only for identifying the goal representations, which further shape the policy. As shown in Figure 1 (right), we represent goals in both environments while optimizing the policy only in the source environment, alleviating the excessive need for rollout steps in the target environment.
Furthermore, we introduce a KL regularization to address the challenge of dynamics shifts. This objective allows us to incorporate a reward modification into the goal-achievement reward function in the standard unsupervised RL, aligning the trajectory induced in the target environment against that induced in the source by the same policy. Importantly, it enables useful inductive biases towards the target dynamics: it allows the agent to specifically pursue skills that are competent in the target dynamics, and penalizes the agent for exploration in the source where the dynamics significantly differ. As shown in Figure 2 (bottom), the difference in dynamics (a wall in the target while no wall in the source) will pose a penalty when the agent attempts to go through an area in the source wherein the target stands a wall. Thus, skills learned in source with such modification are adaptive to the target.
We name our method unsupervised domain adaptation with dynamics-aware rewards (DARS), suggesting that source and target dynamics both shape rg: (1) we employ a latent-conditioned probing policy in the source to represent goals [31], making the goal-achievement reward source-oriented, and (2) we adopt two classifiers [11] to provide reward modification derived from the KL regularization. This means that the repertoires of skills are well shaped by the dynamics of both the source and target. Formally, we further analyze the conditions under which our DARS produces a near-optimal goal-conditioned policy for the target environment. Empirically, we demonstrate that our objective can obtain dynamics-aware rewards, enabling the goal-conditioned policy learned in a source to perform well in the target environment in various settings (stable and unstable settings, and sim2real).

heading: 2 Preliminaries
text: Multi-goal Reinforcement Learning: We formalize the multi-goal reinforcement learning (RL) as a goal-conditioned Markov Decision Process (MDP) defined by the tupleMG = {S,A,P,RG, γ, ρ0},
where S denotes the state space and A denotes the action space. P : S × A × S → R≥0 is the transition probability density. RG , {G, rg, p(g)}, where G denotes the space of goals, rg denotes the corresponding goal-achievement reward function rg : G× S ×A× S → R, and p(g) denotes the given goal distribution. γ is the discount factor and ρ0 is the initial state distribution. Given a g ∼ p(g), the γ-discounted return R(g, τ) of a goal-oriented trajectory τ = (s0, a0, s1, . . . , sT ) is ΣT−1t=0 γ
trg(st, at, st+1). Building on the universal value function approximators (UVFA, Schaul et al. [38]), the standard multi-goal RL seeks to learn a unique goal-conditioned policy πθ : A×S×G→ R to maximize the objective EP,ρ0,πθ,p(g)[R(g, τ)], where θ denotes the parameter of the policy. Unsupervised Reinforcement Learning: In unsupervised RL, the agent is set in an open-ended environment without any pre-defined goals or related reward functions. The agent aims to acquire a repertoire of skills. Following Colas et al. [8], we define skills as the association of goals and the goalconditioned policy to reach them. The unsupervised skill acquisition problem can now be modeled by a goal-free MDPM = {S,A,P, γ, ρ0} that only characterizes the agent, its environment and their possible interactions. As shown in Figure 1 (left), the agent needs to autonomously interact with the environment and (1) learn goal representations (eg, discovering the goal distribution p(g) and learning the corresponding reward rg), and (2) learn the goal-conditioned policy πθ as in multi-goal RL.
Here we define a universal (information theoretic) objective for learning the goal-conditioned policy πθ in unsupervised RL, maximizing the mutual information IP,ρ0,πθ (g; τ) between the goal g and the trajectory τ induced by policy πθ running in the environmentM (with P and ρ0),
max IP,ρ0,πθ (g; τ) = H(g)−H(g|τ) = H(g) + EP,ρ0,πθ,p(g)[log p(g|τ)]. (1)
For representing goals, the specific manifold of the goal space could be a set of latent variables (eg, one-hot indicators) or perceptually-specific goals (eg, the joint torques of ant). In the absence of any prior knowledge about p(g), the maximum ofH(g) will be achieved by fixing the distribution p(g) to be uniform over all g ∈ G. The second term EP,ρ0,πθ,p(g)[log p(g|τ)] in Equation 1 is analogous to the objective in the standard multi-goal RL, where the return R(g, τ) can be seen as the embodiment of log p(g|τ). The objective specifically for learning rg in p(g|τ) is normally optimized by lens of the generative loss [33] or the contrastive loss [42]. With the learned goal distribution p(g) and reward rg , it is straightforward to learn the goal-conditioned policy πθ using standard RL algorithms [40, 19]. In general, optimizations iteratively alternate for representing goals (including both goal-distribution p(g) and reward function rg) and learning the goal-conditioned policy πθ, as shown in Figure 1 (left).

heading: 3 Unsupervised Domain Adaptation with Dynamics-Aware Rewards

heading: 3.1 Problem Formulation
text: Our work addresses domain adaptation in unsupervised RL, raising expectations that an agent trained without prior goal representations (p(g) and rg) in one environment can perform purposeful tasks in another. Following Wulfmeier et al. [54], we also focus on the domain adaptation of the dynamics, as opposed to states. In this work, we consider two environments characterized by MDPs MS (the source environment) andMT (the target environment), the dynamics of which are PS and PT respectively. Both MDPs share the same state and action spaces S, A, discount factor γ and initial state distribution ρ0, while differing in the transition distributions PS , PT . Since the agent does not directly receiveRG from either environment, we adopt the information theoretic IP,ρ0,πθ (g; τ) to acquire skills, equivalently learning a goal-conditioned policy πθ that achieves distinguishable trajectory by maximizing this objective. For brevity, we now omit the ρ0 term discussed in Section 2.
In our setup, agents can freely interact with the sourceMS . However, it has limited access to rollouts in the targetMT with which are insufficient to train a policy. To ensure that all potential trajectories in the targetMT can be attempted in the source environment, we make the following assumption: Assumption 1. There is no transition that is possible in the target environmentMT but impossible in the source environmentMS : PT (st+1|st, at) > 0 =⇒ PS(st+1|st, at) > 0.

heading: 3.2 Domain Adaptation in Unsupervised RL
text: We aim to acquire skills trained in the source environmentMS , which can be deployed in the target environmentMT . To facilitate the unsupervised learning of skills for the target environmentMT
(with transition dynamics PT ), we maximize the mutual information between the goal g and the trajectory τ induced by the goal-conditioned policy πθ over dynamics PT , as shown in Figure 3 (a):
IPT ,πθ (g; τ). (2)
However, since interaction with the target environment MT is restricted, acquiring the goalconditioned policy πθ by optimizing the mutual information above is intractable. We instead maximize the mutual information in the source environment IPS ,πθ (g; τ) modified by a KL divergence of trajectories induced by the goal-conditioned policy πθ in both environments (Figure 3 b):
IPS ,πθ (g; τ)− βDKL ( pPS ,πθ (g, τ)‖pPT ,πθ (g, τ) ) , (3)
where β > 0 is the regularization coefficient, pPS ,πθ (g, τ) and pPT ,πθ (g, τ) denote the joint distributions of the goal g and the trajectory τ induced by policy πθ in sourceMS and targetMT respectively. Intuitively, maximizing the mutual information term rewards distinguishable pairs of trajectories and goals, while minimizing the KL divergence term penalizes producing a trajectory that cannot be followed in the target environment. In other words, the KL term aligns the probability distributions of the mutual-information-maximizing trajectories under the two environment dynamics PS and PT . This indicates that the dynamics of both environments (PS and PT ) shape the goal-conditioned policy πθ (even though trained in the source PS ), allowing πθ to adapt to the shifts in dynamics. Building on the KL regularized objective in Equation 3, we introduce how to effectively represent goals: generating the goal distribution and acquiring the (partial) reward function. Here we assume the difference between environments in their dynamics negligibly affects the goal distribution2. Therefore, we follow GPIM [31] and train a latent-conditioned probing policy πµ. The probing policy πµ explores the source environment and represents goals for the source to train the goal-conditioned policy πθ with. Specifically, the probing policy πµ is conditioned on a latent variable ω ∼ p(ω)3 and aims to generate diverse trajectories that are further relabeled as goals for the goal-conditioned πθ. Such goals can take the form of the latent variable ω itself (Figure 3 c1) or the final state of a trajectory (Figure 3 c2). We jointly optimize the previous objective in Equation 3 with the mutual information between ω and the trajectory τ̃ induced by πµ in source, and arrive at the following overall objective:
maxJ (µ, θ) , IPS ,πµ(ω; τ̃) + IPS ,πθ (g; τ)− βDKL ( pPS ,πθ (g, τ)‖pPT ,πθ (g, τ) ) , (4)
where the context between p(g) and p(ω) are specified by the graphic model in Figure 3 (c1 or c2). Note that this objective (Equation 4) explicitly decouples the goal representing (with πµ) and the policy learning (wrt πθ), providing a foundation for the theoretical guarantee in Section 3.4.

heading: 3.3 Optimization with Dynamics-Aware Rewards
text: Similar to Goyal et al. [16], we take advantage of the data processing inequality (DPI [3]) which implies IPS ,πθ (g; τ) ≥ IPS ,πθ (ω; τ) from the graphical models in Figure 3 (c1, c2). Consequently, maximizing IPS ,πθ (g; τ) can be achieved by maximizing the information of ω encoded progressively to πθ. We therefore obtain the lower bound of Equation 4:
J (µ, θ) ≥ IPS ,πµ(ω; τ̃) + IPS ,πθ (ω; τ)− βDKL ( pPS ,πθ (g, τ)‖pPT ,πθ (g, τ) ) . (5)
2See Appendix D for the extension whenMS andMT have different goal distributions. 3Following DIAYN [10] and DADS [43], we set p(ω) as a fixed prior.
For the first term IPS ,πµ(ω; τ̃) and the second term IPS ,πθ (ω; τ), we derive the state-conditioned Markovian rewards following Jabri et al. [24]:
IP,π(ω; τ) ≥ 1
T T−1∑ t=0 (H (ω)−H (ω|st+1)) = H (ω) + EpP ,π(ω,st+1) [log p(ω|st+1)] (6)
≥ H (ω) + EpP ,π(ω,st+1) [log qφ(ω|st+1)] , (7)
where pP ,π(ω, st+1) = p(ω)pP ,π(st+1|ω), and pP ,π(st+1|ω) refers to the state distribution (at time step t + 1) induced by policy π conditioned on ω under the environment dynamics P; the lower bound in Equation 7 derives from training a discriminator network qφ due to the non-negativity of KL divergence, Epπ(st+1) [DKL(p(ω|st+1)||qφ(ω|st+1))] ≥ 0. Intuitively, the new bound rewards the discriminator qφ for summarizing agent’s behavior with ω as well as encouraging a variety of states.
With the bound above, we construct the lower bound of the mutual information terms in Equation 5, taking the same discriminator qφ:
FI , IPS ,πµ(ω; τ̃) + IPS ,πθ (ω; τ) ≥ 2H (ω) + Epjoint [log qφ(ω|s̃t+1) + log qφ(ω|st+1)] , (8)
where pjoint denotes the joint distribution of ω, states s̃t+1 and st+1. The states s̃t+1 and st+1 are induced by the probing policy πµ conditioned on the latent variable ω and the policy πθ conditioned on the relabeled goals respectively, both in the source environment (Figure 3 c1, c2).
Now, we are ready to characterize the KL term in Equation 5. Note that only the transition probabilities terms (PS and PT ) differ since agent follows the same policy πθ in the two environments. This conveniently leads to the expansion of the KL divergence term as a sum of differences in log likelihoods of the transition dynamics: expansion pP ,πθ (g, τ) = p(g)ρ0(s0) ∏T−1 t=0 [P(st+1|st, at)πθ(at|st, g)], where P ∈ {PS ,PT }, gives rise to the following simplification of the KL term in Equation 5:
βDKL ( pPS ,πθ (g, τ)‖pPT ,πθ (g, τ) ) = EPS ,πθ [β∆r(st, at, st+1)] , (9)
where the reward modification ∆r(st, at, st+1) , logPS(st+1|st, at)− logPT (st+1|st, at).
Combining the lower bound of the mutual information terms (Equation 8) and the KL divergence term pursuing the aligned trajectories in two environments (Equation 9), we optimize J (µ, θ) by maximizing the following lower bound:
2H (ω) + Epjoint [log qφ(ω|s̃t+1) + log qφ(ω|st+1)] − EPS ,πθ [β∆r(st, at, st+1)] . (10)
Overall, as shown in Figure 4, DARS rewards the goalconditioned policy πθ with the dynamics-aware rewards (associating log qφ with β∆r), where (1) log qφ is shaped by the source dynamics PS , and (2) β∆r is derived from the difference of the two dynamics (PS and PT ). This indicates that the learned goal-
conditioned policy πθ is shaped by both source and target environments, holding the promise of acquiring adaptive skills for the target environment by training mostly in the source environment.

heading: 3.4 Optimality Analysis
text: Here we discuss the condition under which our method produces near-optimal skills for the target environment. We first mildly require that the most suitable policy for the target environmentMT does not produce drastically different trajectories in the source environmentMS : Assumption 2. Let π∗ = arg maxπ IPT ,π(g; τ) be the policy that maximizes the (non-kl-regularized) objective in the target environment (Equation 2). Then the joint distributions of the goal and its trajectories differ in both environments by no more than a small number /β > 0:
DKL ( pPS ,π∗(g, τ)||pPT ,π∗(g, τ) ) ≤ β . (11)
Algorithm 1 DARS
1: Input: source and target MDPsMS andMT ; ratio R of experience from source vs. target. 2: Output: goal-reaching policy πθ . 3: Initialize parameters µ, θ, φ and ψ. 4: Initialize buffers B̃S , BS and BT . 5: for iter = 0, . . . ,MAX_ITER do 6: Sample latent variable: ω ∼ p(ω). 7: Collect probing data in source: B̃S ← B̃S ∪ ROLLOUT(πµ,MS , ω). 8: Update discriminator qφ: φ← Update(φ, B̃S) 9: Set reward function for the probing policy πµ:
r̃ = log qφ(ω|s̃t+1). 10: Train probing policy πµ: µ← SAC(µ, B̃S , r̃).
11: Relabel goals: # According to Figure 3 (c1, c2) g ← Relabel(ω, τ̃). 12: Collect source data: BS ← BS ∪ ROLLOUT(πθ,MS , g, ω). 13: if iter mod R = 0 then 14: Collect target data: BT ← BT ∪ ROLLOUT(πθ,MT , g). 15: end if 16: Update classifiers qψ for computing ∆r: ψ ← Update(ψ,BS ,BT ). (Equations 12, 13) 17: Set reward function for πθ: rg ← log qφ(ω|st+1)− β∆r(st, at, st+1). 18: Train policy πθ: θ ← SAC(θ,BS , rg). 19: end for
Given a desired joint distribution p∗(g, τ) (inferred from a potential goal representation), our problem can be reformulated as finding a closest match [29, 28]. Consequently, we quantify the optimality of a policy π by measuring DKL ( pP ,π(g, τ)‖p∗P (g, τ) ) , the discrepancy between its joint distribution and the desired one. With a potential goal representation, we prove that its joint distributions with the trajectories induced by our policy and the optimal one satisfy the following theoretical guarantee.
Theorem 1. Let π∗DARS be the optimal policy that maximizes the KL regularized objective in the source environment (Equation 3), let π∗ be the policy that maximizes the (non-regularized) objective in the target environment (Equation 2), let p∗
PT (g, τ) be the desired joint distribution of trajectory and
goal in the target (with the potential goal representations), and assume that π∗ satisfies Assumption 2. Then the following holds:
DKL ( pPT ,π ∗ DARS (g, τ)‖p∗ PT (g, τ) ) ≤ DKL ( pPT ,π∗(g, τ)‖p ∗ PT (g, τ) ) + 2 √ 2 β Lmax,
where Lmax refers to the worst case absolute difference between log likelihoods of the desired joint distribution and that induced by a policy.
Please see Appendix C for more details and the proof of the theorem. Note that Theorem 1 requires a potential goal representation, which can be precisely provided by the probing policy πµ in Equation 4.

heading: 3.5 Implementation
text: As shown in Algorithm 1, we alternately train the probing policy πµ and the goal-conditioned policy πθ by optimizing the objective in Equation 10 with respect to µ, φ, θ and ∆r. In the first phase, we update πµ with reward r̃ = log qφ(ω|s̃t+1). This is compatible with most RL methods and we refer to SAC here. We additionally optimize discriminator qφ with SGD to maximizing Eω,s̃t+1 [qφ(ω|s̃t+1)] at the same time. Similarly, πθ is updated with rg = log qφ(ω|st+1)− β∆r by SAC in the second phase, where πθ also collects (limited) data in the target environment to approximate ∆r by training two classifiers qψ (wrt state-action qsaψ and state-action-state q sas ψ ) as in [11] according to Bayes’ rule:
max EBS [ log qsasψ (source|st, at, st+1) ] + EBT [ log qsasψ (target|st, at, st+1) ] , (12)
max EBS [ log qsaψ (source|st, at) ] + EBT [ log qsaψ (target|st, at) ] . (13)
Then, we have ∆r(st, at, st+1) = log qsasψ (source|st,at,st+1) qsasψ (target|st,at,st+1) − log q sa ψ (source|st,at) qsaψ (target|st,at) .

heading: 3.6 Connections to Prior Work
text: Unsupervised RL: Two representative unsupervised RL approaches acquire (diverse) skills by maximizing empowerment [10, 43] or minimizing surprise [4]. Liu et al. [31] also employs a latentconditioned policy to explore the environment and relabels goals along with the corresponding reward,
which can be considered as a special case of DARS with identical source and target environments. However, none of these methods can produce skills tailored to new environments with dynamics shifts.
Off-Dynamics RL: Eysenbach et al. [11] proposes domain adaptation with rewards from classifiers (DARC), adopting the control as inference framework [29] to maximize −DKL(pPS ,πθ (τ)‖p ∗ PT
(τ)), but this objective cannot be directly applied to the unsupervised setting. While we adopt the same classifier to provide the reward modification, one major distinction of our work is that we do not require a given goal distribution p(g) or a prior reward function rg. Moreover, assuming an extrinsic goal-reaching reward in the source environment (ie, the potential p∗
PS (τ)), our pro-
posed DARS can be simplified to a decoupled objective: maximizing −DKL(pPS ,πθ (τ)‖p ∗ PS (τ))− βDKL(pPS ,πθ (τ)‖pPT ,πθ (τ)). Particularly, DARC can be considered as a special case of our decoupled objective with the restriction — a prior goal specified by its corresponding reward and β = 1. In Appendix E, we show that the stronger pressure (β > 1) for the KL term to align the trajectories puts extra reward signals for the policy πθ to be ∆r oriented while still being sufficient to acquire skills.

heading: 4 Related Work
text: The proposed DARS has interesting connections with unsupervised learning [10, 43] and transfer learning [55] in model-free RL. Adopting the self-supervised objective [26, 39, 2, 34], most approaches in this field consider learning features [18, 41] of high-dimensional (eg, image-based) states in the environment, then (1) adopt the non-parametric measurement function to acquire rewards [23, 33, 42, 53, 44, 32] or (2) enable policy transfer [23, 16, 17, 13, 22] over the learned features. These approaches can be seen as a procedure on the perception level [20], while we focus on the action level [20] wrt the transition dynamics of the environment, and we consider both cases (learning the goal-achievement reward function and enabling policy transfer between different environments).
Previous works on the action level [20] have either (1) focused on learning dynamics-oriented rewards in the unsupervised RL setting [21, 51, 49, 31] or (2) considered the transition-oriented modification in the supervised RL setting (given prior tasks described with reward functions or expert trajectories) [11, 54, 25, 14, 9, 52, 30]. Thus, the desirability of our approach is that the acquired reward function uncovers both the source dynamics (qφ) and the dynamics difference (β∆r) across source and target environment. Complementary to our work, several other works also encourage the emergence of a state-covering goal distribution [37, 6, 27] or enable transfer by introducing the regularization over policies [45, 15, 46, 47, 36, 48] instead of the adaptation over different dynamics.

heading: 5 Experiments
text: In this section, we aim to experimentally answer the following questions: (1) Can our method DARS learn diverse skills, in the source environment, that can be executed in the target environment and keep the same embodiment in the two environments? Specifically, can our proposed associated dynamics-aware rewards (log qφ − β∆r) reveal the perceptible dynamics of the two environments? (2) Does DARS lead to better transferring in the presence of dynamics mismatch, compared to other related approaches, in both stable and unstable environments? (3) Can DARS contribute to acquiring behavioral skills under the sim2real circumstances, where the interaction in the real world is limited?
We adopt tuples (source, target) to denote the source and target environment pairs, with details of the corresponding MDPs in Appendix F.2. Illustrations of the environments are shown in Figure 5. For all tuples, we set β = 10 and the ratio of experience from the source environment vs. the target environment R = 10 (Line 13 in Algorithm 1). See Appendix F.3 for the other hyperparameters.
Map. We consider the maze environments: Map-a, Map-b, Map-c and Map-d, where the wall can block the agent (a point), which can move around to explore the maze environment. For the domain adaptation tasks, we consider the following five (source, target) pairs: (Map-a, Map-b), (Map-a, Map-c), (Map-a, Map-d), (Map-b, Map-c) and (Map-b, Map-d).
Mujoco. We use two simulated robots from OpenAI Gym [5]: half cheetah (HC) and ant. We define two new environments by crippling one of the joints of each robot (B-HC and B-ant) as described in [11], where B- is short for broken. The (source, target) pairs include: (HC, B-HC) and (ant, B-ant).
Humanoid. In this environment, a (source) simulated humanoid (H) agent must avoid falling in the face of the gravity disturbances. Two target environments each contain a humanoid attacked by blocks from a fixed direction (A-H) and a humaniod with a part of broken joints (B-H).
Quadruped robot. We also consider the sim2real setting for transferring the simulated quadruped robot to a real quadruped robot. For more evident comparison, we break the left hind leg of the realworld robot (see Appendix F.2). We adopt (sim-robot, real-robot) to denote this sim2real transition.

heading: 5.1 Emergent Behaviors with DARS
text: Visualization of the learned skills. We first apply DARS to the map pairs and the mujoco pairs, where we learn the goal-conditioned policy πθ in the source environments with our dynamics-aware rewards (log qφ − β∆r). Here, we relabel the latent random variable ω as the goal g for the goalconditioned policy πθ: g , Relabel(πµ, ω, τ̃) = ω (Figure 3 c1). The learned skills are shown in Figures 2, 6 and Appendix E. We can see that the skills learned by our method keep the same embodiment when they are deployed in the source and target environments. If we directly apply the skills learned in the source environment (without β∆r), the dynamics mismatch is likely to disrupt the skills (see Figure 2 top, and the deployment of DIAYN in half cheetah and ant pairs in Figure 6).
Visualizing the dynamics-aware rewards. To gain more intuition that the proposed dynamicsaware rewards capture the perceptible dynamics of both the source and target environments and enable an adaptive policy for the target, we visualize the learned probing reward log qφ and the reward modification β∆r throughout the training for (Map-a, Map-c) and (Map-b, Map-c) pairs in Figure 7.
The probing policy learns qφ by summarizing the behaviors with the latent random variable ω in source environments. Setting Map-a as the source (Figure 7 (a) left), we can see that log qφ resembles the usual L2-norm-based punishment. Further, in the pair (Map-b, Map-c), we can find that the learned log qφ is well shaped by the dynamics of the source environment Map-b (Figure 7 (a) right):
even if the agent simply moves in the direction of reward increase, it almost always sidesteps the wall and avoids the entrapment in a local optimal solution produced by the usual L2-norm based reward.
To see how the modification β∆r guides the policy, we track three trajectories (with the same goal) and the associated rewards (log qφ − β∆r) in the (Map-b, Map-c) task, as shown in Figure 7 (b). We see that Traj.2 receives an incremental log qφ along the whole trajectory while a severe punishment from β∆r around step 6. This indicates that Traj.2 is inapplicable to the target dynamics (Map-c), even if it is feasible in the source (Map-b). With this modification, we indeed obtain the adaptive skills (eg. Traj.3) by training in the source. This answers our first question, where both dynamics (source and target) explicitly shape the associated rewards, guiding the skills to be domain adaptive.

heading: 5.2 Comparison with Baselines
text: Behaviors in stable environments. For the second question, we apply our method to state-reaching tasks: g , Relabel(πµ, ω, τ̃) = s̃T (Figure 3 c2). We adopt the negative L2 norm (between the goal and the final state in each episode) as the distance metric. We compare our method (DARS) against six alternative goal-reaching strategies4: (1) additionally updating πθ with data BT collected in the target (DARS Reuse); (2) employing DARC with a negative L2-norm-based reward (DARC L2); training skills with GPIM in the source and target respectively (3) GPIM in source and (4) GPIM in target); (5) updating GPIM in the target 10 times more (GPIM in target X10; R = 10 and see more interpretation in [11]); (6) finetuning GPIM in source in the target (GPIM Finetuning in target).
We report the results in Figure 8. GPIM in source performs much worse than DARS due to the dynamics shifts as we show in Section 5.1. With the same amount of rollout steps in the target, DARS achieves better performance than GPIM in target X10 and GPIM Finetuning in target, and approximates GPIM in target within 1M steps in effectiveness, suggesting that the modification β∆r provides sufficient information regarding the target dynamics. Further, reusing the buffer BT (DARS Resue) does not significantly improve the performance. Despite not requiring a prior reward function, our unsupervised DARS reaches comparable performance to (supervised) DARC L2 in (Map-a, Map-b) pair. The more exploratory task (Map-b, Map-c) further reinforces the advantage of our dynamics-aware rewards, where the probing policy πµ boosts the representational potential of qφ.
Behaviors in unstable environments. Further, when we set p(ω) as the Dirac distribution, p(ω) = δ(ω), the discriminator qφ will degrade to a density estimator: qφ(st+1), which keeps the same form as in SMiRL [4]. Assuming the environment will pose unexpected events to the agent, SMiRL seeks out stable and repeatable situations that counteract the environment’s prevailing sources of entropy.
With such properties, we evaluate DARS in unstable environment pairs, where the source and the target are both unstable and exhibit dynamics mismatch. Figure 9 (left) charts the emergence of a
4We do not compare with other unsupervised RL methods (eg. Warde-Farley et al. [53]) because they generally study the rewards wrt the high-dimensional states. DARS does not focus on high-dimensional states. Domain randomization [35, 50] and system (dynamics) identification [12, 7, 1] are also not compared because they requires the access of the physical parameters of source environment, while we do not assume this access.
stable skill with DARS, while SMiRL suffers from the failure of domain adaptation for both (H, A-H) and (H, B-H). Figure 9 (right) shows the comparisons with SMiRL Finetuning, denoting training in the source and then finetuning in the target with SMiRL. With the same amount of rollout steps, we can find that DARS can learn a more stable skill for the target than SMiRL Finetuning, revealing the competence of our regularization term for learning adaptive skills even in the unstable environments.

heading: 5.3 Sim2real Transfer on Quadruped Robot
text: We now deploy our DARS on pair (sim-robot, real-robot) to learn diverse skills (moving forward and moving backward) and balance-keeping skill in stable and unstable setting respectively. We compare DARS with two baselines: (1) training directly in the real world (Full-in-real), (2) finetuning the model, pre-trained in simulator, in real (Finetuning). As shown in Figure 10, after three hours (or one hour) of real-world interaction, our DARS demonstrates the emergence of moving skills (or the balance-keeping skill), while baselines are unable to do so. As shown in Table 1, Fine-
tuning takes significantly more time (four hours vs. one hour) to discover balance-keeping skill in the unstable setting, and the other three comparisons are unable to acquire valid skills given six hours of interaction in the real world. Supplementary material contains videos from this sim2real deployment.

heading: 6 Conclusion
text: In this paper, we propose DARS to acquire adaptive skills for a target environment by training mostly in a source environment especially in the presence of dynamics shifts. Specifically, we employ a latent-conditioned policy rollouting in the source environment to represent goals (including goal-distribution and goal-achievement reward function) and introduce a KL regularization to further identify consistent behaviors for the goal-conditioned policy in both source and target environments. We show that DARS obtains a near-optimal policy for target, as long as a mild assumption is met. We also conduct extensive experiments to show the effectiveness of our approach: (1) DARS can acquire dynamics-aware rewards, which further enables adaptive skills for the target environment, (2) the rollout steps in the target environment can be significantly reduced while adaptive skills are preserved.
Acknowledgments and Disclosure of Funding
The authors would like to thank Hongyin Zhang for help with running experiments on the quadruped robot. This work is supported by NSFC General Program (62176215).

Review summaries:
The paper handles the problem of domain adaptation in RL, specifically the setting where rewards and prior goal representations are not available. They do so by combining a skill learning method (in the source environment) with a transfer learning method that aims to match the joint $(goal, \tau)$ distribution arcoss environments. By doing so they show that skills can both be learned in the source domain, and they transfer more easily to the target domain.

This paper presents an approach for unsupervised learning of a goal-conditioned policy that is robust to changes in dynamics and can be transferred to a target domain that is different from the source domain in terms of the underlying dynamics. There are several components. First, a latent-conditioned probing policy is used to generate goals (either in the source or target domain). This probing policy is trained to generate diverse goals by maximising the information gain between the goals and the resulting trajectories generated by the policy, i.e. we want diverse goals that are predictable given the resulting trajectories. This “predictability” is approximated using a discriminator that scores the likelihood of the goal latent given the resulting trajectory; this is also trained alongside the goal-generating policy based on rollouts in the source domain.   Next, the probing policy is used to generate goals in the source and target domains (these are either the latents themselves or the final states of trajectories from the probing policy), and the goal-conditioned policy (which is conditioned on these goals) is executed in both these domains to generate rollouts. A key assumption here is that it is costly to execute the policy in the target domain (e.g. real world), so the approach tries to minimise this by training the probing policy, discriminator and to a major extent, the policy, primarily on the source domain. The policy is again trained to maximise the log-likelihood of the discriminator, thereby leading to predictable trajectories in the source domain. To ensure that the policy does not overfit to the source domain, and consequently fail on the target domain, an additional KL regularisation term is added to the policy training objective. This regularisation penalises trajectories that provide inconsistent results between the source and target domains, as measured by the KL divergence of the resulting transition distributions when executing the policy for the same goal in both these domains. This effectively ensures that the policy learns trajectories that are executable consistently in both domains, thereby ensuring success even in the presence of differences in dynamics between the two domains. Note that two key assumptions that are necessary for this to work (and the theoretical results) is that there is no transition in the target domain that is not possible in the source domain, and that the goal distribution in the source is also valid in the target (a relaxation of the latter is provided in the supplementary material). This regularisation term is approximated by a pair of classifiers that are trained on data from both domains; these are also trained alongside the policy. The final algorithm alternates training the probing policy & discriminator that parameterise the goal distribution and rewards against training the goal-conditioned policy and the classifiers for approximating the dynamics difference KL.  The approach is tested on several simulated tasks of varying complexity. First, the approach is tested on a 2D point mass with pairs of source and target tasks where the arena has walls of different configurations. Next, the approach is evaluated on continuous control tasks from the OpenAI Gym with the ant, half-cheetah and humanoid embodiments. The source tasks are the standard gym tasks and the target tasks have certain joints not working or additional perturbations in the form of obstacles. Lastly, the approach is also tested in a sim2real setup with a quadruped; the source task is in simulation while the target task is on the real robot. The resulting policy and goal distribution show good performance on all these tasks; the skills and goal distribution learned by the method is able to successfully transfer across changes in wall configuration, joint issues, stability problems and also to real world robots. Several visualisations of different parts of the algorithm (e.g. discriminator, skills) etc. are provided along with comparisons to baselines, namely variants of DARC, SMiRL, DIAYN and GPIM. The supplement also provides additional analysis and proofs of the theoretical guarantees of the near-optimality of the proposed method under the pertinent assumptions.

The authors address the problem of unsupervised Domain Adaptation in Reinforcement Learning. Combining ideas from previous work, they devise a method able to utilize an easy to sample source environment (e.g., a simulator) in order to learn a set of skills without supervision, while at the same time guarantee that these skills will be suited to the target environment (e.g., a real robotic system). To achieve this, they modify the reward structure of the typical Unsupervised Reinforcement Learning setup to enable the agent to build skills that are compatible with the target environment dynamics, while penalizing exploration in regions of the source environment where the dynamics differ. The proposed approach is benchmarked against state-of-the-art algorithms in a wide variety of environments with increased complexity and manages to outperform previous baselines both in simulation as well as in three walking tasks on a real quadruped robot.