Dynamics Adapted Imitation Learning
Authors: Zixuan Liu
Liu Liu
Bingzhe Wu
Lanqing Li
Xueqian Wang
Bo Yuan
Peilin Zhao
Venue: Accepted by TMLR ()

Abstract: We consider Imitation Learning with dynamics variation between the expert demonstration (source domain) and the environment (target domain). Based on the popular framework of Adversarial Imitation Learning, we propose a novel algorithm – Dynamics Adapted Imitation Learning (DYNAIL), which incorporates the dynamics variation into the state-action occupancy measure matching as a regularization term. The dynamics variation is modeled by a pair of classifiers to distinguish between source dynamics and target dynamics. Theoretically, we provide an upper bound on the divergence between the learned policy and expert demonstrations in the source domain. Our error bound only depends on the expectation of the discrepancy between the source and target dynamics for the optimal policy in the target domain. The experiment evaluation validates that our method achieves superior results on high dimensional continuous control tasks, compared to existing imitation learning methods

Published in Transactions on Machine Learning Research (07/2023)
Dynamics Adapted Imitation Learning
Zixuan Liu Liu Liu Bingzhe Wu", Lanqing Li", Xueqian Wang', Bo Yuan Peilin Zhao ex- ax-liu21 @mails. www.ggu...nnggu....om edu.cn, leonlidiuteeneen.coo com, bingzheww@@enneen..com lanqingli wttp:ll199999mmiilloomom @emaill wang. wen.. tsinghuu. edu.cn, boyuan@ieee, org. basonchha@teecent.co
Tsinghua University, 2 TTeentt AT Lab, "Rsearch Institute of Tsinghua University in Shenzhen, Zhefiang Lab
Reviewed on OpenReview: https: openreview. net/ forum? fo=
Abstract
We consider Imitation Learning with dynamics variation between the expert demonstration (source domain) and the environment (target domain). Based on the popular framework of Adversarial Imitation Learning, we propose a novel algoritmm-Dynamics Dynammics Adapted Imitation Learning (DYNAIL), which incorporates the dynamics variation into the stale-action occu- pancy measure matching as a regularization term. The dynamics variation is modeled by pair of classifiers to distinguish between source dynamics and target dynamics. Theoreti- cally, provide upper bound on the divergence between the learned policy and expert demonstrations in the source domain. Our error bound only depends on the expectation of the discrepancy between the source and target dynamics for the optimal policy in the target domain. The experiment evaluation validates that our method achieves superior results on high dimensional contimuuss control tasks, compared to existing imitation learning methods.
1 Introduction
For sequential decision making tasks, recent years have witnessed the success of Reinforcement Learning (RI)) with carefully designed reward functions (e.g. Li et al.
2011 Mnih et al. 2015
Haarnoja et al. 2017
Thomas et al. 2017 Silver et al.] 2018 Vinyals et al.
2019
Kendall et al. 2019
Bellemare et al. 2020
Sutton & Barto
2018). However, it is challenging to define the reward signal in many scenarios, such as
learning "socially acceptable" interactions (Fu et al.
2018 Qureshi et al.
2018
2019
autonomous driving
Kuderer et al.
2015 Kiran et al. 2021), etc.
Recently, modern imitation learning (IL. methods (Ho Ermon 2016 Fu al. 2018 Finn al. 2016
Kostrikov et al.)
2019 Ghasemipour et al. 2020 Orsini al. 2021) ) propose to use adversarial approaches.
More specifically, those Adversarial Imitation Learning (AIL) methods leverage the mechanism of GAN (Goodfellow et al.) 2014h: they alternate between training policy whose trajectories cannot be distinguished from the expert's 's trajectories, and training a discriminator to distinguish between the generated policy and expert trajectories. It has been shown that using diffcrent reward functions (based on the discriminator above) in AIL corresponds to the different choices of divergences between the marginal state-action distribu- tion of the expert and the policy. We refer the readers to Ghasemipour al. (2020) and Orsini et al. (2021) for an in-depth discussion on this topic.
However, in real world scenarios, high quality expert demonstrations are often scarce. For expert demonstra- tion efficiency considerations, limited expert demonstrations are applied to different environments with the same objective. Thus, there may exist dynamics variation between the expert demonstration and the envi- ronment. This issue has already been addressed a line of research (Dulac-Arnold et al. 2020 Mankowitz
let al.
2020
Eysenbach et al. 2021 Liu et al. 2022a| Fu et al. 2018 Qureshi et al. 2019 Kim et al. 2020 Liu
et al.
Kirk al. 2021). g. Under the transfer learning framework (Pan & Yang 2009), we are interested in the problem setting where the transition dynamics in the source domain (expert demonstration) and the
2020
"This work done when Zixuan Liu works intern Tencent Al Lab. i "Corresponding authors.

---

Published in Transactions on Machine Learning Research (07 (022)
(a) Input
(b) Environment
(c) Our Alg.
Source Domain, M'
Target Domain, M
Target Domain, M
ft
Demo
Tt
exp
Start Goal
Discriminator and Classifiers
Figure 1: An illustration of Dynamics Adapted Imitation Learning problems without any reward signal. In (a), we are provided with the expert demonstrations Deemo with the underlying dynamics M'. Texp shows an expert's trajectory from the starting point (round) to the goal (triangle). In (b), an agent aims to make decisions in the target domain (environment) with the same observation space and (unknown) reward function yet different dynamics M. We inject the dynamics variation by adding an obstacle (shaded area). In (c), let T. denote the optimal policy in the target domain. The orange dashed line and red solid line are examples of the trajectories of n²" and f.
target domain (environment) are different. For example, in healthcare treatment, we have the same goal (reward function) for each patients, but they may react differently to treatments (transition dynamics) due to individual differences; in autonomous driving, it is common to encounter different road surfaces different from the expert demonstrations, which also change the transition dynamics. Hence, in this paper, we focus on the imitation learning problem under dynamics variation.
We illustrate the problem sctting Figure [ where we use an obstacle to denote the dynamics variation. The source domain is for expert demonstrations and the target domain shares the same observation space and (unknown) reward function. However, the transition dynamics of the target domain M is different from the source domain. It has been shown (e.g. Xu et al., 2021) that the policy value gap in Imitation Learning directly depends the divergence between the state-action distribution of Texp and denoted by Pexp and Px (please refer to Section 2 for formal definitions). In this paper, we directly use the divergence between Pexp and Pr as prozy for the policy value gap, and we aim to find a policy से in the target domain, which has close performance closed to n²
1.1 Main contributions
(Algorithm) We consider dynamics adaptation in imitation learning where the underlying transi- tion dynamics of the expert demoustration (source domain) and the environment (target domain) are different. To tackle this issue, propose novel algorithm Dynamics Adapted Imitation Learning (DYNAIL,, Algorithm D. The core ingredient of our algorithm is incorporating the dy- namics variation into the state-action occupancy measure matching as regularization term. The regularization for the dynamics discrepancy is modeled by a pair of classifiers to distinguish between source dynamics and target dynamics. Our algorithm builds upon the popular AlL framework, hence it's simple to implement, and computationally efficient.
(Theoretical guarantees) We analyze our algorithm in terms of divergence minimization with a regularization on the discrepancy between the source and target dynamics. We establish a = dependent upper bound on the suboptimality of our algorithm and has no requirement for all other possible policies, where x* is the optimal policy in the target domain. Although n* is unknown, our upper bounds only depends the discrepancy between the source and target dynamics for #*. Our
2

---

Published in Transactions on Machine Learning Research (07/2023)
upper bounds explicitly account for the discrepancy between the source and target dynamics. It offers valuable insights into the relationship between the proposed method and the target domain's optimal policy (Section
(Empirical validations) We validate the effectiveness of DYNAIL on a variety of high-dimensional continuous control benchmarks with dynamics variations. Section and Appendix show that our algorithm achieves superior results compared to saate-of-the-at imitation learning methods.
1.2 Related work
Domain adaptation in IL. Most IL methods consider the unchanged transition dynamics (e.g. Pomerlcau
1988
Ho& & Ermon 2016; Braniley et al. 2019 Sasaki & Yamashina 2021: Rajaraman el al., 2020 Liu et al.
2022b).
Recently, there are different lines of research dealing with domain adaptation in imitation learning. Inverse RL methods AIRL (Fu et al. 2018) and EAIRL (Qureshi et al. 2019) are able to deal with
dynamics variation in imitation learning settings with warm-up in source domain. Kim et al. (2020) and Raychaudhuri et al. (2021) achieve Markov decision process (MDP) alignment by interacting with both source and target domains. However, the source domain inaccessible in most Imitation Learning settings since domains are much harder to store than demonstrations especially in real-world scenarios. Furthermore, (Kim et al. 2020) directly transferring expert policies from source domain to target domain requires online expert policy, which is an additional assumption to imitation learning settings. Liu et al.) (020)) align state sequences to learn to focus on imitating states and the aligument of policy and its prior serves as a constraint in the policy update objective. The policy prior derives from the variational auto-encode encoder and (generally challenging) inverse dynamics model which are hard to train. Fickinger et al. (2021) utilize the Gromov- Wasserstein distance to compare different domains through the whole trajectories. However, sampling the whole trajectories instead of stale-action joint distributions brings high variance. Chae et al. (2022) also deal with dynamics variation in Imitation Learning, however, it requires experiences from multiple source domains to complete the similar task in target domain. In the contrast, our method only needs demonstrations from one source domain to realize domain adaptation.
Off-dynamics RL. Different from IL, RL has direct access to the reward function. recent survey Kirk et al. 2021 provides a thorough introduction on the generalization of RL, which includes state, dynamics, observation or reward function variation. Most relevant to our work, Eysenbach et al. (2021) h and Liu et al. (2022a) consider dynamics variation problems in RT. More specifically, |Eyeebbach et al. 2022 propose reward modification for optimizing the policy in the source domain to obtain the near-optimal policy in the target domain. Liu et al. (022a)) extend the idea of reward modification to the offline setting.
Transfer R.. Transfer RL is aimed at learning robust policy among several domains in RL settings. Unlike Off-dynamics RL, representation is a more popular topic than dynamics in this field. Some previous works focus on learning almain-agnostic representations (Zhang el al.] 2020aa b) Tomar et al.] [2021] to improve the
generalization of representation, while other works (Tirinzoni et al. 2018 Huang et al.) 2021) propose
make use of both domain-shared and domain-specific representations.
2 Background and Problem Setting
2.1 Markov Decision Process and Imitation Learning
We first introduce the background of MDP and IL without considering dynamics variation. An MDP (S,A, r, M, du. consists of state space S, an action space A, a reward function r S x - R, an unknown transition dynamies M S x A A(S), an initial state distribution do E A(S), and a discounted factor y E (0, An agent following the policy (aass) in MDP with dynamics M(s'|s, a) generates the trajectory Tx = (81, (11,01,'' a1,''. 8T, sT,ar), where T is the termination step. We define the probability distribution M (3, a) E A(S x A) to denote the marginal joint stationary distribution for state and action at time-step t. Then define the discounted stationary state-action distribution p a) (1 (1-7) 7) E, 7' (s, a), and the discounted stationary state distribution d= (s) = Eas 2 (s, a). The goal of RL is find policy æ to maximize the expected cumulative reward V~= Eran Ex, Ex7/7(( 7T((8))))))
3

---

Published in Transactions on Machine Learning Research (07/2023)
Without any reward signal, Imitation Learning algorithms (e.g. Osa et al. 2018 aim to obtain policy to mimic the expert's behavior using demonstration dataset Ddemo = {s {*;;
i= where N is the sample size of the dataset. In II, we are interested in the policy value gap Vexp Vxx where Vexp is the value of the
d=))
expert's policy. A classic type of IL methods is Bchavior Cloning [Pomerleau 1988). which uses supervised learning (e.g,, maximum likelihood estimation) to learn the policy n.
Another line of research uses the idea of Inverse RL [Russell 1998 Ng & Russell 2000), which infers the expert's reward function from its demonstrations dataset, and then trains a policy to optimize this learned reward. Early works on Inverse RL (Abbeel & Ng. 2004) rely matching feature expectations or moments between policies and experts. After that, Max Entropy Inverse RL (Ziebart et al. 2008 Zicbart 2010) can handle the ambiguity problem of the policy optimization, where there are many optimal policies that can explain the expert demonstrations. Max Entropy Inverse RL addresses this issues by optimizing the maximum entropy objective to ensure that the solution is unique. More specifically, the following generative model is trained according the trajectory 7:
T x II M (s s,, exp(ro(s,, t=
1 max ET Deamo po()) where Po(T) = 7 p(0))
(1)
where Z is the normalization for the parameterization of po( and n is the temperature parameter. The formulation Equation (1 connects the reward learning problem with maximum likelihood problem.
2.2 Adversarial Imitation Learning
Modern IL methods (Ho & Ermon 2016 Fu et al. 2018 Finn et al., 2016 Ghasemipour et al, 2020
Orsini et al. 2021) use adversarial approaches for generative modeling by casting Equation (1) as a GAN (Goodfelow et al. 2014) optimization procedure, alternating between optimizing the discriminator and updating the policy. The discriminator deals with the binary classification by taking (s, a) as the input, and assigning the expert trajectories positive labels and the generated samples negative labels, and then calculate
the corresponding reward function. For example, AIRL (Fu et al. 2018 uses the following reward function:
ro(s, a) log Do a) log(1 a)))
(2)
It is shown that the minimization of the discriminator function is equivalent to the maximization of the likelihood function Equation ① With the reward function Equation in hand, we can use standard policy learning algorithuss to update the policy.
It has been shown that, without dynamics variation, AIL methods are equivalent to minimizing the diver- gence between the pr(s, a) and (s, Ghhasmiipou et al.) 2020 Xu et al. 2021), which often yield better performance with a few expert demonstrations compared to Behavior Cloning. Under the divergence minimization framework (Ghasemipour et al. 2020), AIRL is equivalent to
AIRL arg D (p, (e))(s,a)).
(3)
which the so-called reverse KL divergence. Since the policy value gap in Imitation Learning directly depends on the divergence between Pexp and Par (Xu al. 2021), we directly use the divergence between Pexp and Psx as prozy for the policy value gap.
2.3 Expert Demonstration Efficiency
Imitation learning, the method of learning from previously collected expert demonstrations, acquires near- optimal policy without the complex reward engineering in standard RL. However, collecting expert demon- stration can be costly and the collected expert demonstrations are usually domain-specific with certain dynamics and policy information. Thus, vanilla imitation learning utilizing expensive expert demonstrations to complete the task in specific domains leads to low expert demonstration efficiency. A natural idea arises: expert demonstrations collected from a single source domain should be used for learning different target domains, which can significantly improve expert demonstration efficiency (Liu al. 2020 Fickinger et al.
2021 Chae et. al. 2022
4

---

Published in Transactions on Machine Learning Research (07 (203))
2.4 Dynamics Adaptation in Imitation Learning
In this section, we consider Imitation Learning methods for dynamics adaplation. We use M' 8,a) to denote the transition dynamics in the source domain, and [s' s,a) to denote the target domain. Directly applying vanilla Behavior Cloning or Inverse RL algorithms requires the same transition dynamics for the environment as the expert demonstrations (uu et al.) 2018), which limits their applicabiliyy. This due the fact. that the learned rewards vanilla. Inverse RL are not transferable to environments featuring different dynamics.
There have been some attempts based Inverse RL to obtain reward function which is transferable and robust to dynamics variation (Fu et al. 2018 Qureshi et al. 2019). By minimizing the KT-divergence, these methods are able to deal with dynamics variation in the setting of imitation learning when the source environment accessible for limited interactions to learn a robust reward function.
Other methods (e.g. Liu et al., 2020 Fickinger et al. 2021 use state distribution matching or trajectory matching rather than state-action distribution matching realize dynamics adaptation. Liu al. (2020) additionally train inverse dynamics model and variational auto-encoder to derive a policy prior to reg- ularize the state distribution matching, Fickinger et al. [2021) introduce the Gromov Wasserstein distance
into trajectory matching to find an isometric policy.
We rntuee the following assumption on the transition dynamics for our problem setting.
Assumption 1. If the transition probability in the torget domain is positive, then the transition probability the source domain is positive, i.e., Ms's 8,a) > 0 = M' 8, a) > 0, Vs, ES
This assumption is modest and common in previous literature (e.g. Koller & Friedman 2009 Eysenbach et al. 2021). If it does not hold, the trajectory induced by the optimal policy in the target domain x² might incur invalid behaviors in the source domain.
3 Our Algorithm
To better identify the role of the transition dynamics in the distribution of the trajectories, we directly compare the distribution over the trajectories. In the source domain with transition dynamics M', the goal of Max Entropy Inverse RL (Ziebart et al. 2008 is to maximize the likelihood given the expert demonstrations data Demo by the following parameterization.
max Eew [lg poo; sourcce))
(4)
1 II M' (sr+1 8t, az) exp(ra(st,a.)/)).
pa(:; pa(r:source) 7 p((5) so)
(5)
where is the normalization of Po(T;source), and n) is the temperature parameter. On the other hand, for the target domain with transition dynamics M, the policy To(a; | s:) yields the distribution over trajectory
P= target)) p(so) II M( I st, (e) 8t).
(6)
3.1 Divergence minimization with dynamics shift
Based the divergence minimization framework for imitation learning (Ho & Ermon 2016 Finn al. 2016 Fu et al. 2018 Ghasemipour et al. 2020 Eysenbach et al.) 2021), our motivation is to learn policy running under the target domain dynamics whose behavior has high likelihood based on the expert demonstrations. Hence we minimize the reverse KL divergence"
min ID (px (T; target) (T: source))
The KT. divergence has been widely used in the RL literature, and we leave the discussion the Appendix
5

---

Published in Transactions on Machine Learning Research (07 (27)))
Expanding Equation
7 yields the policy objective Equation (8)
max B M.
Σ (a(s,,a)) a,) 1og ((a 8x) +
1yD O2 S8+1)
(8)
Discrepancy Regularizatinn
where the discrepancy regularization equals
(st, (1,,111 = log (8441 11 81, a.) - log M (8(41 86; (2).
Note that neglect the normalization term in Equation Section 2.2 the reward function term training discriminator. The ((, sz) term is considered to be the entropy regularizer. These two terms are actually equivalent to the vanilla AIRL Fuu et al. 2018).
Following the ATL framework rntuedd in be obtained from generating the policy in the environment and
Importantly, the regularization term (S,, at 1t+1) represents the dynamics shift from M' to M by calculating log M' (8 8t, -log (8++1 8,, a+). When summnd over the trajectory induced by a under the dynamics M, this equivalent regularization term
(s.a)=p0 (M 8,a) M' |8,))]]
in the reward maximizing objective. When the source and larget dynamics are equal, the regularization term & is zero and our method reduces to the vanilla ALL methods (Ho & Ermon
2016 Fu et al. 2018
This discrepancy between the source and target dynamics has been previously applied to dynamics adaptation in RT literature (e.g., online RI (Fysenbach et al., 202T) and offline RIĄLiu et al. 2022a We provide novel and to the best of our knowledge, the first algorithm to incorporate the dynamics discrepancy in the state-action occupancy measure matching for Imitation Learning problems. In the next subsection, we propose practical methods (see Equation [1 and Equation
12 to estimate a(s,a,s').
3.2 Dynamics Adapted Imitation Learning
To maximize the likelihood function over the trajectories of the experts Equation
44 we use adversarial
approaches for generative modeling by casting Equation [ as GAN (Goodfellow et al. 2014] optimization. As introduced in Section 2 we leverage the reward function formulation r = log( (- which equivalent to distribution matching under KL divergence as in Equation (3 We summarize our algorithm Dynamics
Adapted Imitation Learning (DYNAIL)) Algorithm [ The overall procodure of our algorithm follows from
the GAN (Goodfellow et al.] 2014) which alternates between optimizing the discriminator/classiiii and updating the policy.
Training the discriminator. The discriminator Do is designed to distinguish expert policy and current running æ. The training of the discriminator Dg follows from
min Do
Er-Dacne [log Du(Se, ar)] Erwn, 1og(1 - Da(se, ux))
(10)
exp(ra/n-4) where we train the discriminator, Dg Tog (=ooxpps///--00 by labelling the expert demonstrations Ddemo from souree domain as positive. The negative samples are the trajectory from running policy in the target domain under the dynamics M. As justified in Fu et al. (2018), training on Do results in optimizing the likelihood function in Equation 4
Training the classifiers.
The discrepancy regularization term obtained by training two classifiers,
((st, 01, S141) log M' (st+1 a.) log M st, ae)
Pr(sourco|s.a,5 a, Pr(source|s,
log Pr(target|s, s')
log Pr(target|s, (s
where (i) follows from the fact that M'(s'1s, = Pr(s'18, source) =
Pr(sourcel Pr(s,a,s)) Pr(source|ss s.a) Pr(s,a)
and
( a) Pr(s'ls, a, target) Pr(target|s.aa.' Pe(e.a,d') Pr(target]s.a) Pr(s.o)

---

Published in Transactions on Machine Learning Research (07/2023)
Algorithm Dynamics Adapted Imitation Learning (DYNAIL)
Input: Expert demonstrations Ddemo from the source domain, hyperparameter 7/.
Output: Policy ă in the target domain with different dynamics. Randomly initialize π.
for t = 0 to T do
5: Sample trajectories T = ((st, oz, s by rumning æ in the target domain's environment.
6: Train the discriminator Do: the expert demonstrations Deemo from source domain are positive and the trajectory from policy π are negalive via Equation (10).
7: Train the classifier qsa and Jsss with cross-entropy loss via Equation and Equation
12
Update the reward function F(s, s')) via the following equation,
=(sga, =lgss))=oo((D log(1 Dn) + 17 log
Isas
log
Usa
(9)
Isas
9sm
Update the policy æ with respect to {ss at &o)T using standard RL algorithm. 10: end for
11: Return: Policy
Hence we parameterize Pr(source|s, s') and Pr(source|ss a) with the two classifiers (sns (8, s') and (s, a) respectively. We optimize the cross-entropy loss for the binary classification by labelling the trajectories Ddemo from source domain as positive. The negative samples are the trajectory from running policy x in the target domain under the dynamics M.
min Ey
[log Jsns (8, a, s')] E [log(1 - (sas(s, a, s'))
(11)
Boar
min E demo [log asa (s, a)] EY [log(1 Isa (s, (s)
(12)
Omn
1= Compared to the vanilla AIRL (Fu al.) 2018), we only incur computational overhead by training two additional classifiers. We note that only learn the classifiers rather than the transition models. Hence method is suitable to handle high-dimensional tasks, which is validated on 111-dimensional ant environment in Section 5
Hence the discrepancy regularization ((s,, at, 81+1) log (ee (4. log qea (s)
Updating the policy. With Equation ( in hand, we can update the reward function for the collected trajectory {(s,, az, st. f(),7) We note that is the hyperparameter for the regularization term in the reward function. Besides the entropy term which considered for policy regularization, optimizing the policy objective π w.r.t. Equation shares the same structure as Equation (8)
4 Theoretical Analysis
Let Texp denote the expert policy in the source domain, and (s, a) denote its corresponding distribution of (s, As introduced in Equation 3). AIL methods can be interpreted as minimizing the divergence between Pa (8, a) and (x, a) (Ho & Ermon 2016 Ghasemipour et al.] 2020 Xu el al. 2021), , and this serves a proxy for the policy value gap. Our theoretical analysis shows that by imposing the discrepancy regularization the objective function,
error bound for the KL divergence compared to x² only depends the discrepancy between the source and target dynamics for the expert policy in the target domain.
To distinguish the expert demonstrations and the generated trajectory of the policy n, the optimal discrim- inator for Equation (10) (line 6 of Algorithm I is achieved at
Pexpp (s a) D' = Pexp(s, a) + pdd (3,
We consider the asymptotic results and assume that the optimal Da be achieved.
7

---

Published in Transactions on Machine Learning Research (07/2023)
We note that a simple proof was included in Goodfellow et al. (201]). We plug in this D* a) to the reward function log( (D)) and clip the logits of the discriminator to the range [-C, CJ (C is universal constant such as 10). The Equation (9) in Algorithm with D is equivalent to the following optimization problem
with r as a regularization term
max B M
Σ log Pexp (sx a+) - log Po
A (se, ar) + 1 at s(ti)]
(13)
x
where the temperature 7 > 0 is a tuning regularization hyperparameter to be chosen. According to KKT condition, there exists an & > O, such that regularized formulation Equation (13) is equivalent to the constrained formulation (The equivalence is straightforward and we leave the details to the Appendix A
Equation [4], where the deviation in KT. divergence with different dynamics for any policy x E ITg is
controlled by some
max B(sa))=pf/ [log Pexp a) log pM a)] subject to π E 1lg, x
(14)
where IIg ( E (s)))))p D (M s, (. s, a))) =
(15)
By leveraging the regularization/ satnlaarizatioo/cnnsaaatt constraint to the reward maximizing optimization, we show that, surpris-
ingly, our theoretical guarantees only depend on the smallest positive & of Equation 15 for the optimal policy in the target domain.
Definition 4.1. Let nt be the optimal policy the target domain, and e be the smallest positive solution of Equation
15 for n², i.e.,
E (s.0)~ M DD (M(.
We note that the ex in Definition 1.1 can always be achieved, since 1) is a tuning hyperparameter. And for a fixed pair of M and M", smaller hyperparameter 7) corresponds to a larger Then there always exists a pair of y and such that Definition 4.1 is satisfied for π², Although the n' is unknown, this critical quantity ex only depends on the discrepancy between the source and target dynamics for nt, rather than any other policy candidates.
Our Theorem 4.1 bounds the KL divergence between AM (s, a) and Pexp(s, a). We focus on the term A (s, a), which denotes the state-action distribution while executing the policy मे in the source domain with the transition dynamics M'. Since Pexp the provided expert demonstration, the upper bound for the quantity (x.." || Pexp) directly relates to the sub-oplimality of the value function, which is a natural metric as introduced in Section 2.2. Compared to executing x² (the optimal policy in the target domain) in the source domain, the KL divergence with the optimal (s) only incurs an additive error of 0 ((=Vee))
Theorem 4.1. Let Pexp(8, a) denote the distribution of (8, a) for the expert policy in the source domain, and nt be the optimal policy in the target domain. Under Assumption 1 and Definition 4.1 we have
S DD (pM. |l P(xD) + +0 7 VE*) =(v--_
ll Pexp)
(16)
1-7
Remark 4.1. The theorelical analysis is restricted to the source domain. Due to the nature of Imiiation Learning, we provide the upper bound for the quantity DKL(p! Pexe)) If the reward signal is provided, we can exlend the bounds to the larget domain. The proof is deferred to the Appendir A In Equation (6), we provide theoretical upper bounds by the "x" "d"-dependent" quantity e", and has no requirement. for all other possible policies. Following the previous statistics literabure (Donoho & Johnstone) 1994 Wainwright 2001)), we refer to such bounds as the oracle property - the error upper bounds can automatically "adapt" " to the dynamics discrepuncy of the trajectory induced by x² Although the optimal policy x² in the target domain is unknown, our upper bound identifies the discrepancy of KL divergence for nt via the "oracle" property. Furthermore, the linear dependency on --m (which is the effective horizon for the discounted MDP) matches the best known results [Rajaroman et at., 2020 Xu et al., (022]) for IL methods. When the dynamics in the source domain and target domain are the same, i.c., E + O, and we recover = = which is also the expert policy in the source domain.
8

---

Published in Transactions on Machine Learning Research (07/202) 2022)
CustomAnt-vo
DisabledAnbvv
ledAnt
EXIRL
nomalized eeturn
DYNAIL
AIRL
GAIEO
SOWILL
02
steps(le6)
steps:1 e6)
Steps(led)
(a) The normalized return vs. the iterations on different target domains. The dashed line stands for the performance of x², which the optimal results in each larget domain. BC is straight line for interaction with the larget domain. The shaded area stands for one standard deviation for 20 trials. In the left plot, the target domain is the same the source domain, and all the methods have good convergence properties. When there exists dynamics variations (middle and right plots) DYNAIL has superior performance compared to AIRL, SOAIRL (the shorthand for state-only AIRL), EAIRL, GAIFO, GWIL and BC. The performance gap is widening as Perippledlegs grows (from left to right).
(b) The snapshot in the corresponding target environments. Different levels of dynamics variations are represented by different Rerippledlegs in CustomAnt-vO, DisabledAnt-vl and DisabledAnt-v2 (from left to right).
Figure 2: Performance on different target domains for ant environment. For all the experiments, we use the same pre-collected 40 expert trajectories on source domain (CustomAnt-vo) as expert demonstrations.
5 Experiments
In this paper, we propose an imitation learning method which deals with the dynamics variation between demonstrations and intoractions. Our experiments aim to investigate the following questions: (1) Does our method outperform prior work in dynamics adaptation? (2) What degree of dynamics shift can our method address? We briefly present the performance of our algorithm in this section, and defer the ablation study to the Appendix B
Experimental setup. We compare our algorithm DYNAIL with number of baselines introduced in Section 2.4 A natural baseline is to use Behavior Cloning (BC) the expert demonstrations, and then directly apply it to the target domain without the consideration of dynamics variation. We investigate AIRL (uu et al. 2018), its variant State-only AIRL (SOAIRL), Empowered AIRL (EAIRL) (Qureshi et al., 2019) and GAIFO (Torabi et al. 2018). We also evaluate our algorithm by comparing performance against the state-of-the-art bascline GWIL Fickinger et al. 2021).
Since these baselines and our algorithm share the similar training procedure, use the same net architecture. We use PPO (Schulman et al. 2017) for the generator in AIL framework except for humanoid task where we use SAC (Haarnoja et al. 2018 for the generator, to optimize the policy and use 10 parallel environments to collect transitions on larget domains. The discriminator Do, classifiers qa and Gsss have the same structure of hidden layers, 2 layers of 256 units each, and a normalized input layer. We use ReLU as activation after each hidden layer. In all experiments, discounting factor is considered as 0.99. A key hyperparameter for our method 7,, which serves as tuning regularization. and we defer the full ablation study on 1j to the A Appendix
9

---

Published in Transactions on Machine Learning Research (07/202) 2023)
BrokenHallKheettah33
#eirCheetanGestaccleee
EAIRL
112
random
normalzed return
GNII
AIRL
GAIFO
normalized
SOARL
Stepstlef
steps(le6)
BrokenRescher-v0
Lewfrictionussirupe
Brokenillumenoide
normalized return
normelized return
normaliion
-2.
steps(1o6)
Figure 3: The normalized return the iterations different target domains for continuous control tasks. The baselines are the same with Figure hence the descriptions are omitted. For all the experiments, we use the same pre-collected 40 expert trajectories source domain (HalCChetaah-v3, Reacher-vO, Quadruped and Humanoid-v3) as expert demonstrations, and the snapshots for different environments are leaved to Figure 5 in the Appendix.
For all the experiments, the expert demonstrations are collected by using RL algorithms in Stable Baselines3 (Raffin et al. 2019) by interacting with the source domains without access to target domains. We use 40 trajectories collected by expert as demonstrations. For all the environments used in our experiments have fixed horizon, which is the maximum length of episode in the environment.
5.1 Different levels of dynamics variation
For the environment sctup, we follow the simulator previously used in Qureshi et al. (2019), whore we use a custom and as source domain and several differently disabled ant as target domains.
Custom ant. Custom ant basically the same ant from OpenAl Gym (Brockman al. 2016 except for joint gear ratios. With lower joint gear ratios, the robot flips less often and the agent learns fast. We refer this environment as CustomAnt-vO. The dimension of observation space is 111 and that of action space is 8.
Differently disabled ant. Disabled ant differs custom ant in crippled legs whose impact on the dynamics of robot is bigger than crippled joint in broken ant domain. The degree of disability depends on the number of crippled legs, Perippledlegs 1,2, which corresponds Ant-vi and Ant-v2
We present the experimental results in Figure 2 The experts represented by dashed lines in the plots are RL agents trained with reward in the target domain. The shaded area of all the curves stands for standard deviation for 20 trials. BC does not require interaction with the target domain, hence its performance are straight lines in the plots. In the left. plot, the target domain CustomAni-vo is the same as the source domain, and all the methods have good convergence properties. With the increase of nerippledlegs: we observe that the performance of basclines decreases, yet our method still outperforms the basclines in different target domains. This demonstrates the effectiveness of our method for imitation learning the target domain with dynamics variation.
10

---

Published in Transactions on Machine Learning Research (07 (023))
5.2 Different continuous control tasks
We extend our experiments to four conlinuous control lasks with crippled bodies and obstacles, and a real- world scenario F
The target environments derived from standard environments are widely studied in previous works (Eysenbach et al., 2021; Dulac-Arnold et al. 2020). Due to space limit, we defer the detailed setup of the target environments to the Appendix B
We present the experimental results in Figure 3 In all the experiments with dynamics variations, our algorithm DYNAIL achieves superior performance compared to the baselines. We note that in HalfCheetahOsstacley
method even obtains higher reward than the "optimal" policy in the target domain which is trained via standard RL algorithm in the target domain. This is due to that it is easier to run backwards in MalCChootahhbsttclee obstacle with limited reward. However, our algorithm DYNAIL learns to run forwards and gets more reward with the assistance of demonstrations provided by expert running forwards in the source domain. This phenomenon for the environment also observed in Eysenbach (2021).
the expert tends to run backwards and is prevented by the
6 Conclusion and future work
We've proposed a novel algorithm to incorporate dynamics variation into imitation learning and learns a superior near-optimal policy to the baselines. Compared with prior work (Liu et al. 2020 Kim et al. 2020 Raychaudhuri et al. 2021 Fickinger et al.] 2021). is achieved without access to source domain or any assistance of the online expert and complex dynamics model. There are several avenues for the future work: since our work is actually a Meta Algorithh designed for dynamics variation, which builds upon Adversarial Imitation Learning. We can leverage previous work ALL to improve the sample complexity (Kostrikov ct al. 2019). Any improvement in the time/sample complexity of AIL methods would benefit from the robustness to dynamics variation by our algorithm. Also, we can leverage a Real world RI. control suite Aallac-Annod et al.) 2020), Grid2Op (Donnot 2020) and CityLearn VVazquez-Canteli Canteli et al. 2020) for the experiments, it would be of interest to apply our algorithm with real data, such as autonomous driving, girdpower operation and demand response.
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty- first international conference on Machine learning, pp. 1, 2004.
Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using rein- forcement learning. Nature, (188(836):77-888 2020.
Kianté Brantley, Wen Sun, and Mikael Henall. Disagreement-regglaaiieedd imitation learning. In International Conference on Learning Representations, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Jongseong Chae, Seungyul Han, Whiyoung Jung, Myungsik Cho, Sungho Choi, and Youngehul Sung. Robust imitation learning against variations in environment dynamics. In International Conference on Machine Learning, pp. 2828 2852. PMLR, 2022.
B. Donnot. Grid2op- A testbed platform to model sequential decision making in power systems. comtHuub.com/ree. fereeffrance/eee france/gridooop 2020.
https:
David L Donoho and Jain M Johnstone. Ideal spatial adaptation by wavelet shrinkage. biometrika, 81(3): 425 455, 1994.
3To further demonstrate the efficacy our methods, we provide experiment videos http:///tthuub.ccooaaaaa----a
DYNATL
11

---

Published in Transactions on Machine Learning Research (07/2023)
Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learning. 2020.
Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhutdinov, Off- dynamics reinforcement learning: Training for transfer with domain classificrs. In International Conference on Learning Representations, 2021. URL
Arnaud Fickinger, Samuel Cohen, Stuart Russell, and Brandon Amos. Cross-domain imitation learning via optimal transport. In International Conference Learning Representations, 2021.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversar- ial networks, inverse reinforcement learning, and energy-basod models. arXiv preprint arXiv:1611.08855 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018. URL https::/openreviee..
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. divergence minimization perspective imitation learning methods. In Conference Robot Learning, pp. 1259 1277. PMLR, 2020.
Tan Goodfellw, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-poliey maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pp. 1861-1870 PMLR, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in nearul information processing systems, 29:4565-4573, 2016.
Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. In International Conference on Learning Representations, 2021.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in day. In 2019 International Conference Robotics and Automation (ICRA), pp. 9248-8254. IEEE, 2019.
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning, pp. 1286-5295 PMLR, 2020.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Almad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation. Systems, 2021.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of generalisation in deep reinforcement learning. arXio preprint 0rXim:2111.0999,, 2021.
Daphne Koller and Nir Friedman. Probabilislic graphical models: principles and techniques. MIT press, 2009.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Diseriminator-attoorritt::: Addressing sample inefficiency and reward bias in adversarial imitation learn- ing. In International Conference on Learning Representations, 2019. URL Forum?idd-H44fpoo5kk
12

---

Published in Transactions on Machine Learning Research (07/2023)
Markus Kuderer, Shilpa Gulati, and Wolfram Burgard. Learning driving styles for autonomous vehicles from demonstration. In 2015 IEEE International Conference Robotics and Aulomalion (ICRA), pp- 2641 2646. IEEE, 2015.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint trXii:1805.0000 00909,, 2018.
Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual- based news article recommendation algorithms. In Proceedings of the 4th International Conference on Web Search and Data Mining (WSDM), pp. 297-306, 2011.
Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-based imitation learning. In In- ternational Conference Learning Representations, 2020. URL http:://pppprreeeeeeeeeteee/rrrs rylrdxHFDr
Jinxin Liu, Zhang Hongyin, and Donglin Wang. DARA: Dynamics-aware reward augmentation in oflline
reinforcement learning. In International Conference on Learning Representations, 2022a. URL https:
Liu Liu, Ziyang Tang, Lanqing Li, and Dijun Luo. Robust imitation learning from corrupted demonstrations. arXiv preprint ar Xic:2201 12594, 2022b.
Daniel J. Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Yuanyuan Shi, Jackie Kay, Todd Hester, Timothy Mann, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In International Conference on Learning Representations, 2020. URL httts::/ htt.://oooommmm000.CC00000
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 -3332 2015.
Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In ICML, 2000.
Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowicz. What matters for adversarial imitation learning? Advances in Neural Information Processing Systems, 34, 2021.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and TrendsS in Robotícs, 7(1-2):1-179, 2018.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. JEEE Transactions on knowledge and dala engineering, 22(10):1345 13599, 2009.
D. Pomerlean. Alvinn: An autonooouus land vehicle in a neural network. Tn Advances in Newral Information Processing Systems, 1988.
Ahmed H. Qureshi, Byron Boots, and Michael C. Yip. Adversarial imitation via variational inverse re- inforcement learning. In International Conference on Learning Representations, 2019. URL https: //openreview.c wwt.:e/oorrrmmiidnmmmeeeeee
A Ahmed Hussain Qureshi, Yutaka Nakamura, Yuichiro Yoshikawa, and Hiroshi Ishiguro. Intrinsically moti- vated reinforcement learning for human-robot interaction the real-world. Neural Networks, 107:23-33 33,, 2018.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3, 2019.
Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 33, 2020.
13

---

Published in Transactions on Machine Learning Research (07/2023)
Dripta S Raychaudhuri, Sujoy Paul, Jeroen Vanbaar, and Amit K Roy-Chowdhury. Cross-domain imitation from observations. In International Conference on Machine Learning, pp. 8902-8912 PMTR, 2021.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Compulational learning theory, pp. 101 101-103, 1998.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In International Conference on Learning Representations, 2021.
John Schulman, Sergey Levine, Picter Abbecl, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889 1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint (xX::11777 777.06347, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Joannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Gracpel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140 1144, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Philip S. Thomas, Georgios Theocharous, Mohammad Ghavanzadeh, Ishan Durugkar, and Emma Brunskill. Predictive off-policy policy evaluation for nonstationary decision problems, with applications to digital marketing. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pp. 4740 4745, 2017.
Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, and Marcello Restelli. Importance weighted transfer of samples in reinforcement learning. In International Conference on Machine Learning, pp. 4936 4945. PMLR, 2018.
Manan Tomar, Amy Zhang, Roberto Calandra, Matthew Taylor, and Joelle Pincau. Model-invariant state abstractions for model-based reinforcement learning. ar preprind 2XXw:2102.098500 2021.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. anXiv preprint axXic:1807.06158, 2018.
Jose R Cazquez--Canteli Sourav Dey, Gregor Henze, and Zoltan Nagy. Citylearn: Standardizing research in multi-agent reinforcement learning for demand response and urban energy management. arXie preprint arXiv:2012 10504 2020.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement. learning. Nature, 575(7782):350 354, 2019.
Martin J Wainwright. High-dimensional statistics: non-usymptotic viewpoint, volume 48. Cambridge
University Press, 2019.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policics and environments for reinforcement learning. IEEE Transactions on Pallern Analysis and Machine Intelligncee 2021.
Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta. Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. In International Conference on Machine Learning, pp. 11214-11224. PMLR, 2020a.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant represental ions for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2020b.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of mazimum cousal entropy. Carnegie Mellon University, 2010.
Brian Ziebart, Andrew Maas, Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Agai, volume 8, pp. 1433 1438. Chicago, IL, USA, 2008.
14

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
A Proofs
A.1 Facts about KL divergence
We introduce Assumption on the transition dynamics, which assumes that if M(' s,a) 0, then M' 8, 0, Vs, 8' E S.a E A. If it does not hold, then the trajectory induced by the optimal policy in the target domain x might involve invalid behaviors in the source domain. The reverse of Assumption | is not necessary to be truc.
The usage of KT. divergence in Imitation Learning. Reinforcement Learning and Control is ubiquitous (Fu et al. 2018 Levine 2018 Ghasemipour al., 2020). Similar assumptions on transition dynamics are also widely used in Koller & Friedman (2009) and Eysenbach al. [2021) We note the Assumption 1 is also
consistent with the definition of KL divergence in Equation
min ID) (7; target) pe (Ti source))
However, such assumptions may not hold for simulation environments with deterministic transition dynamics such MuJoCo, due to mathematical definition. Hence, is found important for the practical implemen- tation to use the combination of clipped discriminator logits, and tuning the gradient penalty for effective training of Adversarial Imitation Learning methods (Ghasemipour et al. 2020 Orsini et al. 2021
A.2 Proof of Theorem
Based on the constrained optimization formulation Equation 14 we first build a uniform bound (Temma A.1 for the deviation of all policies in the constraint set ITg-
Lemma A.1 (Uniform Bound). Under Assumplion 1 for all the policy in the constraint set ITe we have
[R [log Pexp log M1
E R
[og Pexp log Px A5
0 1-9
TT. Vo),,, ~y)',
(17)
where I1g = {TT E(na))a! (D( s,a) M'( s,)) se}.
Proof of Lemma A.1 For the LHS of Equation
17 we consider the following function
(ssa)so Jog Pexp Pexp(s, (s. -D (Pot ll Pexp),
Pr (s.
which is actually the KL divergence function between Per and Pexp. The KL divergence Lipschitz continuous with respect to Px (s, a) in & norm. This is due to the constant C' for the upper and lower bound the logits of the discriminator.
We then present the perturbation bound on & norm for the stationary function of state d=l The techniques are from the perturbation theory (Schulman et al. 2015 Xu et al., (021). Let PM 8) = Eas (s' 8, (a]s) as the state transition function for a given policy æ. Then the stationary distribution of d can written
2
dMM = (1 77) Σ 7 Pr(s: = 8; M, do)
= (1 - 7T 7 P.M D
where do is the initial state distribution.
Then we calculate the difference in the distribution
MM' dM 7(1 - 7PP' y (PM -PM )
15

---

Published in Transactions on Machine Learning Research (07 2022))
Based on the bound in Lemma 7 of Xu et al.
2021 we have
d
11 S 7
-ppa')- I - PM p.pppdd dd 111
(18)
2
7
|1(par- phl Jd
(19)
1
(1)
7
B((aa) [IM s, M1 [,,))
(20)
1-
(11)
V2y
VE,
(21)
7-7
where (i) follows from the fact |1- -DM')p''-- )ー S Ei = 1 (ii) follows from the definition of PM (s') s), and (iii) follows from the Pinsker's inequality for bounding the & norm via KL divergence, and the definition for c. Similar techniques are also used in Schulman et al. (0055) and Xu et al. 2021
The last step is bounding the | (s, p (s. (ss.a)), by laai - a |
in Equation (0
1p (s,0) p (s, S ||d d=d 111
V2y
Ve,
where (i) follows from the definition of ( a) and d,.
Putting together the pieces, we have
Bppr [log Pexp - log Pst M
E Reye [log Pexp - log Px M'
01 -1(0 v8)
for all the policies in the constraint set glg-
Theorem A.1 (Theorem 4.1). Let Pexp a) denote the distribution of (s, a) for the expert policy in the source domain, and n² be the optimal policy in the target domain. Under Assumplion and Definition 4.1
we have
D
|| Poxp)
S D (pxt
7 Pexp) + +01 1- + VE*)
(22)
Proof of Theorem 4.1 To distinguish the expert demonstrations and the generated trajectory of the policy
n, the optimal discriminator for Equation [10 (line of Algorithm achieved at
Pexp (s, a)
(* (exp a) + p (s,a)*
(23)
The simple proof was included in Goodfellow et al. (2014), and the logits of the discriminator is clipped to be within the range, for example, 10, 10) Ghasemipour et al. 2020
Plugging the Equation 23 and we are solving the RL problem with the discrepancy regularizer
Σ log (st, at) - log log (st, at) + 1 at (1t1
max EYNT.
(24)
Since we have (33,, ar, 8t41) = log M'(44+ St, ar) - log M (st+1 St, az), the regularization term is actually equivalent to the constraint set
II. f ((aa))=o [D (M( s, ||
16

---

Published in Transactions on Machine Learning Research (07 (07/202) 2023)
Hence we have the following optimization in the domain of p.
max (( [log (s a) log P (s, a)] subject to и IIg.
(25)
The next step analyzing the objective function of Equation 25 basod on the optimality condition of / in Algorithm 1 and the feasibility of x² in the constraint formulation Definition 4.1 we have
E p
log llggeex Pexp
log p
M
B
[log Pexp log P-
(26)
We then apply Lemma A.1 (wice.
log Pexp - log A!l E pi1
log Pexp - log Mr
0 7
Ve*).
1-7
Finally, we have
( p || PoxD SD (p POXD) + +0 +0( 7 - Vo²).
--7
B
Full Experiments
B.1
Hyperparameters
To use the popular framework of Adversarial Imitation Learning, we use the regularization method and their hyperparameters according previous papers (e.g. Brantley al., 2019 Orsini al. 2021). Besides the policy learning algorithm and the discriminator in AIL framework, our algorithm DYNAIL also includes two classifiers (sa and (sns, and use the same architecture and hyperparameter the discriminator. For the parameter we explain our methods in the next subsection. The hyperparameters are shown in Table [
Table 1: Hyperparameters in Algorithm
!
Hyperparameter
Values considered
Final Value
Parallel Environments
8, 10
10
fo regularization
0
Entropy cocfficient
0.01
0.01
Gradient clipping
0.1, 1.0
1.0
Policy learning rate
10-
3 10
Generator batchsize
100, 200
100
Discriminator learning rate
1 10
1 10
Discriminator batchsize
800, 1000
1000
Discriminator updates
5, 10
5
Discriminator weight decay
0.01, 0.1
0.01
Discriminator logits clipping
5, 10, 20
20
Classifier learning rate
1 10
1 10
Classifier batchsize
800, 1000
1000
Classifier updates
5, 10
5
Classifier weight decay
0.01, 0.1
0.01
Discrepancy Regularization & clipping
5, 10, 20
5
17

---

Published in Transactions on Machine Learning Research (07 /023)
B.2 Ablation study on parameter 7)
The workhorse of Algorithm is incorporating the dynamics variation into the stale-action occupancy measure matching as a regularization term. When we optimize the policy via the reward function
a(s,a,s') = log - log(1 DD) + log 1-
log
Usa
qsus
1 qua
it is important to select the tuming parameter Also, to achieve the ex in Delinition 4.1 and Theorem we need to use an appropriate 7). In this section, we describe our method to select this hyperparameter, and present the empirical performance.
Recall that our goal is to minimize the reverse KL divergence via Equation which
min D (T; target) (ource)))))))
When use Algorithm with different hyperparameter 7, obtain the learned policy Ry. We use the "reverse KL divergence" between the 7 and the expert demonstrations as a metric to seleet the hyperpa- rameter 7/- More specifically, we train a classifier Q, for the (s, a, a,, 8') between the expert demonstrations Ddemo and uq. Let i denote the stationary distribution (s, a, s'). Similar to the analysis of discriminator introduced in Section the classifier Qn achieves its optimum at
Mexp a, 8') Qn(s,a,s')= Mexp (8, 8²) + MGz (8, a,,')
Then the average of negative logit value on the generated trajectory equals
log Muxp N (8, (8,0,88)
- -E Dupp
ID ( (oXD)
(27)
Hence we choose the hyperparameter 7), which yields the smallest the D (/S, study of n and its corresponding performance for the target domain Ant- in Figure More specif- ically, we use the expert demonstrations from the source domain, apply different ry in (0.5, 1, 1,2,4,8... 1024), and run Algorithm respectively. For different 7,, we calculate the Equation (27 for each 7.,, and present it on the left plot of Figure 4 In the right plot of Figure 4 we present the normalizee reward vs. ilerations for different 77. We observe that smaller KL divergence for मn in the left plot leads better performance of the normalized return in the right plot. Consistent to our theory, this provides accurate method for model selection on different hyperparameter 7).
m, Mexp We present the ablation
B.3
Experiments Details
In Section 5.2
compare algorithm with several baselines different simulation platforms, which are previously studied in Eysenbach et al. (2021) and Dulac-Arnold et al. (020)). Details for these target environments are as follows, and the snapshots are provided in Figure 5
This environment based on the source environment HalfCheetah--3 from AT Gym (Brockman al.) 2016).. The goal of this environment to make the agent move forwards. Episode length is 1000 steps long. This environment differs source environment from the Oth joint (0-indexed) which is broken: the input torque at this joint is ignored.
BrokenHalFCheetahvv..
This environment is based on the source domain HalfChoctah-v3 from Ope- AAT Gym (Brockman et al. 2016). The goal of this enviromment is to make the agent move forwards. Episode length is 1000 steps long. key modification is that the agent rewarded for running both forwards and backwards. Because it is casier learn to run backwards, an obstacle behind the agent used in this target environment. The agent bounces off the obstacle when running backwards and thus runs forwards for more reward.
HalCheetahOOsstaleev..
18

---

Published in Transactions on Machine Learning Research (07 (02)))
DisabledAnt-v1
KL-divergence vs. in DisabledAnt-v1
1.0
espert
nommllized return
1.0
KL-divergence
0.
16.0
0.
17-
256.0
512.0
16-
-02
1024.0
400
1000
iterations
log2n)
Figure 4: Ablation study results for the DisabledAnt-vl enviromment. The left plot shows the IB (M Mexp) for different a... The right plot shows the normalized reward vs. iterations for different 17. The shaded area stands for one standard deviation for 20 trials. We observe a smaller KT. divergence for R, does lead to better performance of the normalized return. Consistent to our theory, this provides an accurate method for model selection on different hyperparameter 7).
Figure The snapshot in the corresponding targel eneeiromment (BrokenHallChhettahhvo) HalfChettaOObstaclee BrokenReacher-vl), LowFrictionQuudnpeed and BrokenHumanoid-vz: are pre- sented from left right)..
The reacher in this environment is constructed by the 7DOF robot arm in Pusher environment from OpenAI Gym (Brockman et al. 2016). The goal of this enviroument is move the end effector close to the target. Episode length is 100 steps long. This environment differs source environment from the 2nd joint (0-indexed) which is broken: the input torque at this joint is ignored.
BrokenRcacher-vv..
This environment is based on the source domain "quadruped' with "real- walk" task from realworldrl-suite Dulacc Arnold et al. 2020). The goal of this environment is to make the agent walk forwards. Episode length 1000 steps long. The contact friction of the source domain ranges from 0.1 4.5 with the standard deviation of 0.5, while we lower the friction 0.1 as constant in the larget domain. With low contact friction, it is harder for the agent to walk forwards.
www.Frictionuadruuppee
19

---

Published in Transactions on Machine Learning Research (07/2023)
BrokenHHumanoidkv.. This environment is based the source environment Humanoid-v3 from OpenAI Gym [Brockman et al. 2016). The goal of this environment is to make the agent move forwards. Maximum episode length is 1000 steps long, however, episode may terminate early due to unhealthy conditions. This environment differs source environment from the Oth joint (0-indexed) which stands for the red broken abdomen in Figure the input torque at this joint ignored.
C Societal Impact
Real world Reinforcement Learning and Imitation Learning are closely related to our work (Dulac-Arold
ctt al. 2020 Kirk al., 2021). Recent years witness the success of using RI. and
a series of artificial
domains. Nevertheless, many state-of-the-ar Imitation Learning and Inverse RL algorithms are hard deploy in real-world scenarios since the underlying assumptions are rarely satisfied in practice (Dulac-Arnold) et al. 2020). Our research is aligned with previous papers study the generalization performance for Imitation Learning algorithms (more specifically, under dynamics variation). We mainly focus on the algorithm design and do not consider specific applications.
20


Reviews:
[{"review_id": "eh5aGkQmwe", "reviewer": "Reviewer_hbCj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe paper shows good empirical evidence showing that the proposed method performs well compared to previous baselines. The DYNAIL model proposes to close the gap between different environment dynamics between train and testing by using \n\n\nWeakness:\nHowever, this paper does not provide experimental validation on discrete domain environments. Superior performance on such environments would make the claim of the paper stronger and more general. Additionally, the problem of adapting to target environments with different dynamics can be also handled by imitation learning from observation methods like Behavior Cloning from Observations (BCO)[1] and Generative Adversarial Imitation from Observations (GAIfO)[2]. Since these methods learn from observations, they can handle dynamics model differences between source and target. Furthermore, some previous works (such as [3]) also attempts adversarial imitation learning for different target environments. Discussion related to the differences from these works might be useful.\n\n[1] Torabi, F., Warnell, G. and Stone, P., 2018, July. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (pp. 4950-4957).\n\n[2] Torabi, F., Warnell, G. and Stone, P., 2018. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158.\n\n[3] Desai, S., Durugkar, I., Karnan, H., Warnell, G., Hanna, J. and Stone, P., 2020. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems, 33, pp.3917-3929.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes DYNAIL - an adversarial imitation method that accounts for changes in transition dynamics from the source to the target environment. The paper proposes a practical algorithm that consists of the discrepancy regularizer to compensate for the dynamics model difference. The paper also establishes an upper bound for divergence from expert policy which depends on transition dynamics differences between source and target. The paper showed empirical validation on multiple continuous domain tasks showing that DYNAIL outperforms previous baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could you please discuss how the proposed approach compares to Imitation Learning from Observation kind of methods like GAIfO and BCO mentioned above? These methods learn directly from the observations in the demonstration that can transfer to target environments with different dynamics.\n\n2. Why was this method not applied to any discrete action domain tasks? For example, a FlappyBird environment with higher/lower gravity in the target setting can be a good experiment or something else with discrete actions.\n", "broader_impact_concerns": "No major ethical concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "v8hTfPHNpz", "reviewer": "Reviewer_tc73", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The description of the method is sufficiently clear\n- The paper tackles an important problem\n- The method shows reasonable performance in the problem setting\n- penalizing the dynamics mismatch using the reverse KL is sound, as it will bias the agent to state-action pairs that produce similar transitions compared to the source domain. \n\nWeaknesses:\n- As far as I can tell, the additional term of the reward function was not derived but heuristically chosen. Although the paper states that\n\"Expanding equation (7) yields the policy objective Equation (8)\", this derivation is not shown in the paper, Furthermore, it seems quite clear that Equation (8) can not directly be derived from the distribution matching objective (Eq. 7), as it contains an additional hyperparameter $\\eta$ for weighting the cost for the dynamics mismatch. The approach is only justified by showing that the price (imitation gap) of this additional cost term can be bounded. \n\n- The theoretical analysis is restricted to the source domain. However, on the source domain, prior imitation learning methods (that do not use the addition cost for biasing the policy) enjoy better theoretical guarantees. Instead, it would be much useful to bound the imitation gap on the *target* domain.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles the problem of imitation learning when the expert demonstrations were collected under different system dynamics. The paper proposes a new method, DYNAIL, to better imitate the expert demonstrations despite the mismatch of the system dynamics. DYNAIL uses adversarial imitation learning to minimize the reverse Kullback-Leibler divergence, similar to AIRL (However, DYNAIL does not seem to use the special form of discriminator used by AIRL). Different from previous adversarial imitation learning methods, DYNAIL introduces an additional cost given by the expected KL-divergence $D(p(s'|s,a) || p'(s'|s,a))$ between the transition dynamics on the target domain and those on the source/expert domain. This expected KL divergence is estimated using two additional discriminators, one trained on the state transitions (s',s,a) and the second one on the system inputs (s,a) [A similar procedure was proposed by EAIRL]. The paper also provides a bound on the imitation gap on the *source* domain. The approach is evaluated on Mujoco-Ant where the dynamics by changing the kinematics (e.g. leg length) of the robot. Compared to related work, such as EAIRL and GWIL that also tackle imitation learning under dynamic mismatch, the proposed method achieves better final performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The claim that \"Expanding equation (7) yields the policy objective Equation (8)\" needs to be substantiated. The additional cost term should be derived from the distribution matching objective.\n\nIdeally, the theoretical analysis should be extended to the *target* domain. The limitations of the current analysis should be clearly stated.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ni2HvGvIVx", "reviewer": "Reviewer_PYce", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well written and correctly placed with respect to the literature. It is easy to follow and the main idea is simple, neat and well explained.\n\n\n**Weaknesses**\n\nOn the fundamental part, I see one main weakness which is the following: I think it would be worth discussing much more the chosen objective (Eq.7). I know the reverse KL objective is widely used but in this specific case, I had to think quite a lot to try to make sure the objective made sense. This is also the case for Assumption 1, which I believe is much stronger than what the authors say in the paper. One extremely simple example breaking this assumption can be derived directly from the example present in Fig.1: what if, instead of “adding a wall”, we “move a wall”? The assumption is broken. \n\nOn the experimental part, I thus miss experiments that actually break this assumption. What happens in this case? Does DYNAIL still work in practice or does it fail? It would be a strength of the paper to highlight failure modes of the proposed algorithm.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors are interested in Imitation Learning with a dynamic shift between the demonstrations (given in a source domain) and the actual environment (the target domain).\nThe proposed algorithm (DYNAIL), builds upon the adversarial imitation learning (AIL) framework. The authors propose a modified version of the divergence minimization objective that takes into account the dynamics shift. They then propose an algorithm that requires training two other networks and is a modification of the “inverse RL reward” classically used in AIL using the confusion of the discriminator.\nThey propose theoretical guarantees and experiments validating their algorithms on a set of simulated tasks for MuJoCo.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see:\n\n- more detailed justification of the assumption and objective choice.\nexperiments actually breaking the assumption and highlighting if DYNAIL still works or fails.\n- an experiment in the lower data regime (see [1], or [2], for which higher dimensional tasks like Humanoid are successful with very little trajectories that are subsamples 20 times). I am not saying the subsampled setup is the most meaningful, but then what happens if you only have a handful of demonstrations?\n- for the reason previously mentioned, I would have liked to see one experiment with a higher -dimensional action space like a humanoid  OR one that requires navigation (like AntMaze) with dynamic shift as presented in figure 1. \n- for all experiments, please don’t use “iterations” for the x-axis which is not meaningful at all in RL. Please rather use the number of environment steps, as you are working with online imitation learning.\n- Ideally, I would like to see results in terms of divergence too, as optimizing the reward is only a proxy to performance, as suggested in [2] or [3].\n\n\n[1] What Matters for Adversarial Imitation Learning?\n\n[2] Primal Wasserstein Imitation Learning\n\n[3] Hyperparameter Selection in Imitation Learning\n", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the important problem of imitation learning when source and target environments differ in dynamics. Reviewers agree the method is solid, and empirical evaluation is convincing (after adding new ones in the revised version). The theoretical justification is limited, but the revised version added an explicit discussion about the limitations. Overall, the work is an interesting and well-supported contribution.
Decision: Accept (poster)