Imitation Learning from Observations under Transition Model Disparity
Authors: Tanmay Gangwani
Yuan Zhou
Jian Peng
Venue: ICLR 2022 Poster (2022)
Keywords: Imitation Learning
Deep Reinforcement Learning

Abstract: Learning to perform tasks by leveraging a dataset of expert observations, also known as imitation learning from observations (ILO), is an important paradigm for learning skills without access to the expert reward function or the expert actions. We consider ILO in the setting where the expert and the learner agents operate in different environments, with the source of the discrepancy being the transition dynamics model. Recent methods for scalable ILO utilize adversarial learning to match the state-transition distributions of the expert and the learner, an approach that becomes challenging when the dynamics are dissimilar. In this work, we propose an algorithm that trains an intermediary policy in the learner environment and uses it as a surrogate expert for the learner. The intermediary policy is learned such that the state transitions generated by it are close to the state transitions in the expert dataset. To derive a practical and scalable algorithm, we employ concepts from prior work on estimating the support of a probability distribution. Experiments using MuJoCo locomotion tasks highlight that our method compares favorably to the baselines for ILO with transition dynamics mismatch.

Under review as a conference paper at ICLR 2022
IMITATION LEARNING FROM OBSERVATIONS UNDER TRANSITION MODEL DISPARITY
Anonymous authors Paper under double-blind review
ABSTRACT
Learning to perform tasks by leveraging a dataset of expert observations, also known as imitation learning from observations (ILO), is an important paradigm for learning skills without access to the expert reward function or the expert ac- tions. We consider II.O in the setting where the expert and the learner agents operate in different environments, with the source of the discrepancy being the transition dynamics model. Recent methods for scalable ILO utilize adversarial learning to match the state-transition distributions of the expert and the learner, an approach that becomes challenging when the dynamics are dissimilar. In this work, propose algorithm that trains an intermediary policy in the learner environment and uses it as a surrogate expert for the learner. The intermediary policy is learned such that the state transitions generated by it are close the state transitions in the expert dataset. To derive practical and scalable algorithm, we employ concepts from prior work on estimating the support of a probability dis- tribution. Experiments using MuloCo locomotion tasks highlight that our method compares favorably to the baselines for ILO with transition dynamics mismatch.
INTRODUCTION
Imitation Learning (IL) framework that trains an agent perform desired skills by leverag- ing expert demonstrations of those skills. Compared to the standard Reinforcement Learning (RL) approach, IL offers the benefit of not requiring reward function, that can be difficult to specify for complicated objectives. Recent II. methods that integrate efficiency with deep-RI. and are per- formant in high-dimensional state-action spaces include sehavioral-cloning-haee et al., 2011; Brantley et al., 2019), and adversarial II. algorithms inspired by maximum entropy inverse-RL (Ho & Ermon, 2016; Finn et al., 2016). Imitation Learning from Observations (ILO) refers to the setting where the expert demonstrations consist of only observations (or states), while the expert actions are unavailable. ILO beneficial when the measurement of the expert action is difficult, e.g., in kinesthetic teaching in robotics or when learning with motion capture datasets.
algorithms (Ross
Adversarial methods frame the problem as the minimization of an -divergenee between the state- action visitation distributions of the expert and the learner (Ke et al., 2019). Since expert actions absent in IIOO, the analogous methodology here to minimize the f-divergence between the state- transition distributions of the expert and the learner (Arumugam al., 2020). A state-transition dis- tribution for policy is the joint distribution over the current state and the next state, and is defined formally in Section 2. Choosing the f-divergence be the Jensen-Shannon (JS) divergence has en- abled successful imitation using algorithms such as GAIL (Ho & Ermon, 2016) and GAIfO (Torabi al., 2018b). for IL and ILO, respectively.
The transition dynamics model of an environment governs the distribution over the next state, given the current state and action. this paper, we focus on II.O in the scenario of transition dynamics mismatch between the expert and the learner environments. Such discrepancy could manifest in real-world applications of imitation learning when there are subtle differences in the physical at- tributes of the system used to collect the demonstrations and the system where the learner policy is run. Adversarial ILO methods such as GAIfO, that attempt to train the learner by matching its state- transition distribution with that of the expert, perform very well when the expert and learner operate in a shared environment, under the same dynamics. When the dynamics differ, however, matching the sate-transition distributions becomes challenging since the state transitions provided in the

---

Under review as a conference paper at ICLR 2022
pert demonstrations could be infeasible under the dynamics in the learner's 's environment. To form some intuition, consider ILO in a 2D grid with discrete states, where the path demonstrated by the expert follows the main diagonal from the bottom-left grid-cell to the top-right grid-cell. Suppose that the transition dynamics for the learner environment are such that only horizontal and vertical moves are permitted on the grid, L.e, no diagonal motion for the learner. In this case, an optimal learner's movement is still in the same general direction as the expert, and it covers all the expert states (along with some adjacent states). However, matching the trae-transition distributions is an ineffective strategy since the learner cannot reproduce the (one-step) state transitions of the expert.
To alleviate the challenges of ILO under dynamics mismatch, we propose an algorithm that trains an intermediary policy in the learner environment, and hope to use it as surrogate expert for training the learner (imitator). We refer to this policy as the advisor. For the advisor to be effective, the state transitions generated by it in the learner environment should be as close as possible to the state transitions in the expert dataset. We formalize this concept in terms of the cross-entropy distance between state-conditional next-state distributions of the expert and the advisor. To convert this into a practical and scalable algorithm for training the advisor, we incorporate ideas from distribution support estimation (Wang et al., 2019). Simultaneous to the advisor training, the learner agent is updated to imitate the advisor. Crucially, the advisor operates in the same environment as the learner, making the Instribution-matchine II objective amenable.
We evaluate the efficacy of our II.O algorithm using five locomotion control tasks from OpenAl Gym where we introduce mismatch between the dynamies of the expert and the learner by changing different configuration parameters. We demonstrate that our approach compares favorably to the baseline ILO algorithms in many of the considered scenarios.
PRELIMINARIES
We model the RL environment as a discounted, infinite horizon Markov Decision Process (MDP). At every discrete timestep, the agent observes a state (s E S), generates an action (a E AA) from stochastic policy [a]ss), receives scalar reward r(s, a), and transitions to the next state (5') sampled from the transition dynamics model p(s'|s, a). In the infinite horizon setting, the future rewards are discounted by factor of y E [0, 1). Let (s) denote the distribution induced by ı over the state-space at a particular timestep t. The stationary discounted state distribution of π is then defined as Pr (s) = (1 7) Σi y=d² (s). The RL objective of maximizing the cumulative discounted sum of rewards can be framed as max, (x. (r(s, a)), where px(s, a) Pa (s()) (a]s) the state-action distribution (also known as the occupancy measure). Lastly, we define the state- transition distribution for policy as ( -nn p(d pn
2.1 GAIL AND GAIFO
Generative Adversarial Imitation Learning (Ho & Ermon, 2016) is a widely popular model-free IL. method that builds the Maximum Causal Entropy Inverse-RL (MaxEnt-IRL) framework (Ziebart, 2010). MaxEnt-IRL models the expert behavior with a policy that maximizes its 7-discounted causal entropy, H(T)) E--[-lop log T[a]a) r[a|s)]. while satisfying feature matching constraint. GAIL consid- ers a regularized version of the dual to this primal problem. It shows that RL with the reward function recovered as the solution of the regularized dual is equivalent to directly learning a policy whose state-action distribution is similar to that of the expert. For specific choice of the regu- larizer, this similarity is quantified by the IS divergence between the two state-action distributions, Dus (8, a) || Pa. (s, a) Based on these ideas, GAIL seeks to learn policy with the objective:
min max E,,, [log D(s, a)] + Ep.. [log(1 D(s, a))] - XH((T) D
where D: S x X (0, 1) is the discriminator that provides the rewards for training the learner pol- icy π. and the inner maximization over D) approximates (-) similar GANs (Goodfellow al., 2014). To empirically estimate the expectation under Pr, (s, a). state-action pairs are sampled from the available expert demonstrations. In the IL.0 setting, however, expert actions are not included in the demonstrations. To mitigate this challenge, Torabi et al. (2018b) propose GAIIO, which adapts
With slight abuse of notation, we use the symbol Pr for state, state-action, and state-transition distributions of policy म. We would provide context around their usage avoid any confusion.
2

---

Under review as a conference paper at ICLR 2022
to ILO by modifying the GAIL objective to match the transition distributions of the expert and the learner, i.e. DJs(pr (s. s') |I Pr. (s)s Correspondingly, the discriminator in GAlfO is a function of the state transitions D(s,s')
2.2 SUPPORT ESTIMATION RED
We summarize recently proposed method for estimating the support of a distribution in high di- mensions (RED; Wang et al. (2019)), since it forms a core ingredient of our final algorithm. Let X denote set and p be probability distribution on X. Denote by supp(p) = {ar E X p(x) A 0), the support of the distribution p. Given any x X, the task to know if x E supp(p). Towards this goal, Wang et al. (2019) combine ideas from kernel- support estimation (De Vito et al., 2014) and RND (Burda et al., 2018), and consider the following objective:
-" arg arg min E (l)) fz(x))
8
where fo : : R is a trainable function parameterized by 6, while fø is a fixed function with randomly initialized parameters 0. Define the score function as (with constant positive scalar X):
(RED(Z) = exp(-All - (z)
Wang et al. (2019) conclude that the score TRED (a) is high if E supp(p), and is low otherwise. With neural network function approximators, we thus obtain a smooth metric whose value decreases (increases) as we move farther from (closer to) the support of the distribution p.
METHODS
We begin by defining some notations for our setup. We denote the MDP in which the expert policy (Te) operates the e-MDP, while the learner (or imitator) policy is run in the I-MDP. The two MDPs share all the attributes, except for the transition dynamics function, which we symbolize with Ps(s's,, a) and pi(s'(s, ss, a), for the e-MDP and (-MDP, respectively. Pe (s),,, Pe (s, a), and Pe s') are the state, state-action, and sateetrannsiion distribution for the expert policy. These distributions depend the e-MDP dynamics Pe(8') [s, a). The corresponding distributions for the learner denoted by P= (-) and they depend on the 1-MDP dynamics (s"(ss.))
Adversarial ILO methods, such as GAIfO, that learn the imitator policy by matching the state- transition distributions of the expert and learner aim to solve the following primal problem:
max H(T)) s.t. Po(s, (((,,)) 4 (8, 8 E S
If the expert and the learner operate under different transition dynamics, depending on the extent of the mismatch, it is possible that some (or all) of the one-step state transitions of the expert are infeasible under the dynamics function in I-MDP. Said differently, given (Se, 8xx) pair sampled from the expert demonstrations, there could be no action in the 1-MDP from the state Se that results in s as the next state, as per the dynamics (s'((s ss a). This renders the tnte-trransiion matching objective hard to optimize in practice. For instance, in the practical implementation of GAIfO, the rewards for the imitator policy are computed from a discriminator that is trained with binary classification on (s,8') pairs from the expert and the learner. If the expert transitions can't be generated in I-MDP by the learner, then a high-capacity discriminator could achieve perfect accuracy and thus fail to provide informative rewards for imitation.
Consider a sequence of states {81, 8 generated by the expert policy in the e-MDP, as shown in Figure la. Given an expert state s,, may not be possible reach $,+1 in single timestep in the (-MDP. Instead, we would like to find an alternative state $;+1 that is reachable from s; in one step (i.e., feasible under the dynamics pi) and close to the desired destination 81+1- For instance, in Figure 1a, starting from the expert state 81, 82 is more desirable next state compared to 82.
3.1 GUIDANCE VIA AN ADVISOR POLICY
To discover such feasible states 8, we introduce an advisor policy म. that operates in the I-MDP. n. is invoked only for action selection the expert states, rather than being run in a closed feedback
3

---

Under review as a conference paper at ICLR 2022
MDP
HMDP
MDD
JWPP
Pg
Topar SUM
Agei
Doant For 8-
Apu
Maxi Mieeraaiioo desibattoss
Madsoniliool
dleritation
Of pr. (rhe))
Plec En
(a) State transitions
(b) Comparison between GAIIO (left) and approach (right)
Figure I: (a) A sequence of states 8; from the expert dataset. The states &; are reached by sampling an action from the advisor policy Ta from every expert state. State $2 less desirable than $2 since the latter is closer to the expert state $2. The squiggly lines show the path that learner, that is optimized to match the state-action distribution of the advisor, may take. (b) A high-level overview of our approach. While GAIfO directly matches the state-transition distributions of the expert and the learner, we learn an intermediary policy (advisor) in the 1-MDP that acts as the surrogate expert for the learner. D is used to denote the corresponding distance metric.
loop the learner environment. In this way, To akin to a contextual bandit policy. The goal with No is produce action a; (-(ss))) from the expert state s, such that the next state in the I-MDP, 5++1 ~ pr(-,s,, a,). is close to the next state 8,+1 in the e-MDP under the expert policy. We expand on the measure of closeness and the methodology to train the advisor in the next subsection.
When the expert and learner dynamics are the same, the optimal advisor would take the same actions as the expert policy. In the presence of dynamics mismatch, however, the advisor actions provide reasonably good guidance on how to stay close to the expert's 's state trajectory (Figure la). Therefore, the advisor Te could act as surrogale expert for the learner and the learner's imitation learning objective could be suitably modified. The main advantage of the learner and the ad advisor operating in the same I-MDP that we could now attempt match their visitation distributions, which is much more structured than matching distributions across dynamics (Eq. 1). We consider the IL objective:
max H(TT) s.t. Pr. (s, (s) A (s.a) E π
where Pr. (s, a) is the state-action distribution of Ta. Since the advisor is invoked only on the expert states, effectively, Pr. (8,a) pe(s)TT (als). We can convert the above objective to an unconstrained optimization by using a parametric function fu a) as the Lagrange multiplier:
min max wTT == H(T) + (ap. a)] Ep (x.a) (f
(2)
We note that this objective bears resemblance to entropy-reae regularregl apprenticeship learn- ing (Abbeel & Ng, 2004; Syed al., 2008; Syed & Schapire, 2008), modulo the use of advisor policy that contributes the favorable actions, instead of the expert policy.
3.2 TRAINING THE ADVISOR
The advisor policy generates an action (deterministic Ta), or a distribution over actions (stochastic Ta). given an expert state sampled from the demonstration data. One suitable objective for training the advisor is to minimize the dissimilarity between the destination state achieved with the expert policy in e-MDP and that achieved with the advisor in I-MDP. For instance, consider the scenario where the dynamics functions are deterministic and known, denoted by fe and fi for the e-MDP and I-MDP, respectively. Also, the policies me and Fa are deterministic. Then, the advisor could be optimized with the following loss:
min I (TTa) = Ex-ps (oorccsss((((ffee [[] (T())) fi(s, [s(s)]]]
where D is a distance measure, e.g., the L2-norm in the state space. However, we are interested in the setup with stochastic, unknown dynamics functions and stochastic policies. To compute distance metric in this setting, we first define the state-conditional next-state distributions for the expert and the advisor, by marginalizing over the actions:
Pe,x. (arx.ss100 (s'1s) = Σ Pe (a)s))
Pl.aa (5'1s) pp(i8'ss,aTToll
4

---

Under review as a conference paper at ICLR 2022
We then the cross-entropy distance (H) between these distributions as measure of closeness between the expert and the advisor:
T = arg min [(Ta) = Bo) H [Pt,n (s'1s), Pei (d'1s)]
%a
arg max [(Ta) = Beppe (a) Barna (p llog ne (s'[s)]
(3)
(e arg max (TTa Eerpe(s) (ala) (p loog Pe s')]
where, in the last equation, we have multiplied the term inside the log with Pe(s), a quantity inde- pendent of Ta. Figure lb provides high-level overview of our approach.
3.2.1 1
AN APPROXIMATION BASED ON RED
The objective in Eq. presents couple of challenges. Firstly, is infeasible to evaluate the den- sity of any state transition (s, s') under the expert's state-transition distribution (e(s, s'), since this distribution is unknown and only few samples from this distribution are available us in the form of the expert demonstrations. Secondly, and more importantly, note that the state transition that ought to be evaluated under (e s') is generated by the advisor policy the I-MDP. If no advisor policy can replicate the state transition behavior of the expert, as is likely when there is dynamics mismatch, then the objective becomes degenerate with an optimal value of -Cc.
Our approach to mitigate these issues is to replace the log-density term in Eq. 3 with an estimated value that quantifies the proximity of a given state transition (s, 8') to the manifold of the expert's support in this space, i.e., (s, To get this value, we leverage RED (Wang al., 2019), which pre-trains a deep neural network using data samples from the distribution of interest. Then, given test sample, the network outputs continuous value that provides estimate of how far the test sample is from the distribution's ' support. Section 2.2 provides a short background on RED.
In our instantiation, we pre-train RED network TRED (s, s') with state transitions from the expert demonstrations. The network is then frozen and utilized the objective to train the advisor:
max [(a)) = (arnr.ccllla...taaaaaaa
(s,
(4)
For any advisor-generated state transition, (RD s') provides smooth metric in the range (0, 1]. whose value increases (decreases) as the transition moves closer to (farther from) the support of the distribution Pe (s, s'). Thus, training Te to maximize this value yields an advisor that provides guidance on how to stay close to the expert's state trajectory when operating in I-MDP (Figure lb),
To provide further intuition on the use of RED, Table 1 shows the TEE (s,ss) s') values obtained from a trained RED network two environments-aalkee is trained with 50 state transitions {8e, ) sampled from (s(s,8') and then evaluated under several noisy transitions (se: á
and HalfCheetah. The RED network
+n) where
Increasing noise
Walker2d
HallCheetah
denotes zero-mean Gaussian noise. We show the 'RED values aver- aged across the transitions for different settings of the ratio (N/S), i.e., the ratio of the standard deviation of the noise to the standard deviation of the states. We observe that ( s') is close to 1.0 when the transitions are on the support the expert, and it
N/S
-o--gnall
N/S
0.80
0.71
NIS - 0.5
0.44
0.23
NS=1
N/S
0.001
Table 1: Averaged riED s',
gradually decreases as the transitions drift away from the support as a result of a larger amount added noise.
values for different noise levels.
3.3 ALGORITHM AND IMPLEMENTATION
is possible to use a two-stage training procedure for the overall algorithm- learn the advisor in I-MDP with Eq. 4, and then use it in the IL objective laid out in Eq. 2 to optimize for the reward network fu and the learner policy л. In the first stage of advisor learning, although the RED network is trained offline, computing the gradients for optimizing aa still requires environment interaction. More importantly, since the advisor is only trained on the expert states, the objective in Eq. requires the capability to reset the environment to the expert states in the 1-MDP.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1: AILO (Advisor-augmented Imitation Learning from Observations) Input: A dataset of expert state transitions D. = {se, s'y} collected in e-MDP
/" initialization and offline pre-training" //
Initialize: advisor Ta. learner x, reward network (s, a), RED network o (R, s')
Pre-train (RD s') with D. using the algorithm in Wang et al. (2019)
for ier in (1...., N do
/" data collection 7/ Roll out trajectories using π
Update reward network fa using T, na and De (Eq. 2)
/" update learner '/
Compute reward (s, a) for each transition in Use to update with MaxEnt RL
/" update advisor 7/
Compute reward r RED s') for each transition in r. Use t to update Ta with Eq.
end
To alleviate this problem, we propose to train the advisor (Ta). the reward network (fw) and the learner (T)) jointly, in an iterative manner, and reuse the environment interaction data generated with π for training parameters of T.. Then the gradient of the objective function J (ma) is:
well, using the importance sampling trick. Specifically, let B denote the
Eoippits s.a) [ (RED s')]
Esmpe(s) Bawns(al=) (Es' (TED (s, s')] V log na (alss))
(5)
= -Ep (s(s) To (ala(s)
E, (Esrpp( (RED s')l. log la
Px (s) (als)
18 ratios
where the second equation uses the score function estimator (Kleijnen & Rubinstein, 1996), and the third employs two importance sampling ratios. These ratios are easy to estimate for our set- ting (please see Appendix A.1 for details). Crucially, since the gradient is now computed with state-action data from Pa (s, a). we no longer require environment resets. Further, we observe that approximating the inner expectation with single sample is sufficient for our tasks.
We abbreviate our method as AILO, short for augisor-augmented Imitation Learning from Obser- vations. Algorithm provides outline. We start with the offline pre-training of the RED network using a dataset of expert state transitions collected in the e-MDP. Then, we perform iterative opti- mization in the 1-MDP where each iteration involves generating trajectories with the current learner т, followed by gradient updates to the reward network fa, and to the advisor and learner poli- cies. Following Eq. 2. the learner policy is trained with MaxEnt RL algorithm with per-timestep entropy-regularize reward given by fa (st, a,) a log π ((a))))) where a is the entropy coefficient. In our experiments, we use the clipped-ratio PPO algorithm (Schulman al., 2017) and adaptively tune a as suggested in prior work (Haarnoja et al., 2018).
4 RELATED WORK
There is vast amount of literature on IL since it is powerful framework to train agents to perform complex behaviors without a reward specification. ILO, where no expert action labels are available, presents several benefits well some unique challenges, and thus, has garnered significant at- tention from the community in recent times (Torabi et al., 2a;;;; Liu et al., 2018; Edwards et al., 2019; Sun et al., 2019; Yang al., 2019; Zhu al., 2021). ILO methods adapted from GAIL have been proposed for one-shot imitation of diverse behaviors (Wang et al., 2017), training policies to generate human-like movement patterns using motion-capture data (Peng al., 2018; Merel al.,

---

Under review as a conference paper at ICLR 2022
2017), and for locomotion control from visual data (Torabi al., 2018b). Arumugam et al. (2020) introduce a framework that casts adversarial ILO as f-divergence minimization and provide insights on the design decisions that impact performance.
Several approaches have been proposed to handle the differences between the expert and the learner environments in terms of viewpoints, visual appearances, presence of distractors, and morphology changes (Stadie et al., 2017; Gupta et al., 2017; Liu et al., 2018; Sermanet et al., 2018). They typ- Aonainninvarrint representation and matching features in that space. Learning such a representation is not required in our setup since the state-space is shared between the (-MDP and the e-MDP. Methods for robustness to shifts in the action-space and the dynamics model have also been researched. Zolna et al. (2019) propose to match state-pai distributions of the expert and the learner, where the states in a pair are sampled with random time gaps, rather than be- ing consecutive. Liu et al. (2019) learn an inverse action model to predict deterministic actions in the 1-MDP that could generate expert-like state transitions and use it to regularize policy updates. Gang- wani & Peng (2020) filter the trajectories generated in (-MDP based on their similarity to the states in the expert dataset and perform (self-) imitation on these. The key methodological difference be- tween these methods and our work is that learn intermediary stochastic policy (advisor) in the I-MDP by bringing its state-conditionll next-state distribution closer to that of the expert, and propose an instantiation of this idea using approximation based on support estimation.
ically proceed by learning
5 EXPERIMENTS
We evaluate the efficacy of AILO using continuous-control locomotion environments from OpenAl Gym (Brockman et al., 2016), modeled using the MuJoCo physies simulator (Todorov et al., 201 We include a description of the tasks, the baselines, and the learning curves. Further details on the architectures and the hyperparameters are provided in Appendix A.2.
Environments. We consider five tasks {Half- [Half-Cheetah, Walker, Hopper, Ant, Humanoid). To create discrepancy between the expert and the learner transition dynamics, we modify one physical property in the learner environment from the set- {density, gravity, soint-friction] Specifically, for a task T from the task-set, we denote:
T (heavy) - learner agent has 2 the mass of the expert agent
T (light) + gravity in the learner's environment is half the value in the expert's
(drag) + the friction coefficient for all the joints in the learner is 2 the value in the expert
Baselines. We contrast AlLOO with two baselines a.) GAIfO strives to minimize the IS divergence between the state-transition distributions and is briefly described in $2.1; b.) VAIL (Peng et al., 2018) applies the concept of variational information bottleneck (Alemi et al., 2016) to the GAIL discrimi- nator for improved regularization and has been successfully used for ILO with motion-capture data. To limit the effect of confounding factors during comparison, share modules across AILO and the two baselines to the best of our ability. Concretely, all discrininator/rewardd networks use the same architecture and the gradient-penalty regularization (Mescheder al., 2018) and thus exhibit the same reward biases. Furthermore, the MaxEnt-RU RR PPO module is the same for all algorithms.
Performance. Figures 2a-2c plot the learning curves for all the algorithms across the different tasks (heavy, drag, light). We show the average episodic returns achieved by the learner in the I- MDP, normalized the returns achieved by the expert in e-MDP. The plots include the mean and the standard deviation of returns over 6 independent runs with random seeds.
We observe that AILO provides a noticeable improvement in learning efficiency in several situations, such as Chaf-Cheetah (heavy, drag, light), Walker (heavy). Hopper (heavy), Ant (drag). Humanoid (drag, light); while being comparable to the best baseline in other cases. For GAIfO, we find that does not learn any useful skill for Walker (heavy) and exhibits training instability for Half-Cheetah (heavy, drag). We attribute this to the difficulty of matching the state-transition distributions across different dynamics. VAIL, which matches state distributions, instead of state-transition distributions. is a stronger baseline and works well in several cases. Lastly, we highlight a few failure modes of AILO in Ant (light) and Humanoid (heavy), note that AILO (and baselines) do not make much progress towards imitating the demonstrated behavior within our time budget, motivating the need for future enhancements to enable efficient skill transfer in these challenging setups.

---

Under review as a conference paper at ICLR 2022
lours)
HalfChcetah (hcavy)
Walker (hcavy)
Hopper (heavy)
Ant (heavy)
Humanoid (heavy)
Average Epkodi Returns Coonaalizdddt Exper))
0.6
0.6
0.6
0.06
0.4
0.4
0.4
0.4
0.04
0.2
0.2
0.2
0.0
0.0
0.0
2.5 Timestops (millions]
2.5
Timestops (millions)
Timesteps (millions)
Timestops Imil lons)
Timestaps [millionst
(a) Performance on 7 (heavy) environments
AILO (eursi
GAMD
VAL
HalfCheetah (drag)
Walker (drag)
Hopper (drag)
Ant (drag)
Humanoid (drag)
Average Epizodic Returns (Nommlizedd
0.8
0.8
0.8
0.6
0.2
0.6
0.1
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
00.2
2.5
2.5
2.5
2.5
0
2.5
Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Timesteps (mill
(b) Performance T (drag) environments
AILO (ours)
CATO
WIL
HalfCheetah (light)
Walker (light)
Hopper (light)
(light)
Humanoid (light)
1.0
Episodic Returns (Nommainn Expert)
0.6
0.8
0.25
0.6
0.6
0.20
0.15
0.4
0.10
0.2
0.2
0.05
0.0
0.0
-0.2
2.5
2.5
Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
Timesteps (millions)
(c) Performance T (light) environments
Figure Learning curves for AILO and the baselines for different environments with discrepancy dynamics
Ablation the degree of dynamics mismatch. For the empirical results in Figure 2, the variation in mass, gravity, or friction, between the e-MDP and the I-MDP was kept at a constant factor. In Figure consider the Walker task and plot the system- atic degradation in the performance of AILO and the base- line GAIFO, as the I-MDP parameters drift further from the e-MDP parameters. We show the final average episodic returns achieved by the learner in the I-MDP, normalized to the returns achieved by the expert in e- c-MDP, as function of the degree of dynamics mismatch. In the plots, the expert parameters (mass, friction) are kept constant and the learner parameters are varied such that the ratio increases from a starting value of 1 (no mis- match). We observe that although imitation naturally becomes more challenging as the dynamics become more different, the degradation with AILO is graceful compared GAIfO.
Kuber boau!
20
20
Figure
Results with different
amount of dynamics mismatch
8

---

Under review as a conference paper at ICLR 2022
6 CONCLUSION
In this paper, we present AILO, our algorithm for imitation learning from observations under transi- tion model disparity between the expert and the learner environments. Rather than directly matching the state-transition distributions across environments, we train an intermediary policy (advisor) in the learner environment and use it as surrogate expert for the learner. Towards learning an advi- sor that acts an effective surrogate, we propose minimize the cross-entropy distance between the state-conditional nexr-state distributions of the advisor and the expert. realize this idea into scalable ILO algorithm, we leverage prior work support estimation (RED). Our experiments on five MuJoCo locomotion tasks with different types of dynamics discrepancies show that AlLO compares favorably to the baseline ILO methods in many cases.
REPRODUCIBILITY STATEMENT
The authors pledge make the source code for reproducing all the experiments of this paper pub- lic upon the de-anonymization of the paper. The hyperparameters used for the results and some implementation details are included Appendix A.2.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Alexander A Alemi, lan Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information
bottleneck. arXiv preprint 161 2016.
Dilip Arumugam, Debadeepta Dey, Alekh Agarwal, Asli Celikyilmaz, Elnaz Nouri, and Bill Dolan. Reparameterized variational divergence minimization for stable imitation. arXiv preprint arXiv:2006. 10810, 2020.
Kiante Brantley, Wen Sun, and Mikael Henaff. Disagroement--egulliiize imitation learning. In International Conference on Learning Representations, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, arXiv preprint 16X6vv1606.0/55 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. 12894, 2018.
Ernesto De Vito, Lorenzo Rosasco, and Alessandro Toigo. A universally consistent spectral estima- for the support of a distribution. Appl Comput Harmonic Anal, 37:185-2177 217, 2014.
Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International Conference on Machine Learning. pp. 1755-1763. PMLR, 2019.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852. 2016.
Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. arXiv preprint arXiv:2002. 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint arXiv: 1703.0299,, 2017.

---

Under review as a conference paper at ICLR 2022
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- on Machine Learning, pp. 1861-1870. PMLR, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 0065--473, 2016.
Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. arXiv preprint arXiv: 12888, 2019.
Jack PC Kleijnen and Reuven Y Rubinstein. Optimization and sensitivity analysis of computer simulation models by the score function method. European Journal of Operational Research, 88 (3)74134427, 1996.
Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-based imitation learning. arXiv preprint arXiv:
2019.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing to imitate behaviors from raw video via context translation. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 1118-1125 IEEE, 2018.
Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201 2017.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pp. 3481-3490, PMLR, 2018.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational dis- criminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining infor- mation flow. arXiv preprint arXiv: ww..iv:880...0 1810.0082/, 2018.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635, 2011.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 0rXiv:1707.06347 2017.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1134-1141. IEEE, 2018.
Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. arXiv preprint 1rXiv:1703.01703. 2017.
Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient imitation learn- ing from observation alone. In International Conference on Machine Learning, pp. 6036-6045. PMLR. 2019.
Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. In Advances in neural information processing systems, pp. 1449-1456, 2008
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear pro- gramming. In Proceedings of the 25th international conference on Machine learning, pp. 1032- 1039, 2008.
10

---

Under review as a conference paper at ICLR 2022
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. IEEE, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint 1805.00005,, 2018a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: 1807.06/75 2018b.
Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, and Yiannis Demiris. Random expert dis- tillation: Imitation learning via expert policy support estimation. In International Conference Machine Learning, pp. 6536-6544. PMLR, 2019.
Ziyu Wang. Josh Merel, Scott Reed, Greg Wayne, Nando de Freitas, and Nicolas Heess. Robust imitation of diverse behaviors. arXiv preprint arXiv:1707.02244 2017.
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. arXiv preprint a9Xiv:1900.0e 044 2019.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou Off- Off-policy imitation learning from observa- tions. arXiv preprint 138. 2021.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
Konrad Zolna, Negar Rostamzadeh, Yoshua Bengio, Sungjin Ahn, and Pedro O Pinheiro. Reinforced imitation in heterogeneous action space, arXiv preprint arXiv:1904.0344 2019.

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 IMPORTANCE SAMPLING FOR ADVISOR TRAINING
Following Eq. 5, the importance sampling factor to be estimated is P=(a) sulale), Pr(a) =(als) which is a product of ratios. The second ratio N=(als) is easily computable since we have both the advisor policy [aas]]
(Ta) and the learner policy (T) as parameterized Gaussian distributions. The first ratio dele) pe(s) can be approximated by training a binary classifier to distinguish the states sampled from the distributions (e(s) and Pa (s). Concretely, consider the objective:
D' = arg max (s) log [(s)] + E, (4) [log D(s))]
(05-+
Then, estimate the ratio of the state distributions (c(s) D'(+)
A. (5) I-D'
A.2 HYPERPARAMETERS AND IMPLEMENTATION DETAILS
Hyperparameter
Value
Discriminator Reward network (Fa)
architecture
3 layers, 128 hidden-dim, tanh
# updates per iteration
optimizer
Adam = x 10-4)
RL agent (shared)
policy arch.
layers, 64 hidden-dim, tanh layers, 64 hidden-dim, tanh 0.2
critic arch.
PPO clipping
PPO epochs per iteration optimizer discount factor (7)
Adam (Ir = 1 x 10 10-4)
0.99
GAE factor (Schulman et al., 2015) entropy target
0.95
-|A|
Table 2: Hyperparameters for AILO and the baselines
The hyperparameters used for experiments mentioned in Table 2. The discriminator network used by the baselines and the reward network employed in AlLOO share a common structure. The same RI agent is used across AILO and all the baselines. Additionally, for the baselines, tested the hyperparameters mentioned in the respective papers and tuned them with grid-search.
Expert data collection. We train the expert policy in e-MDP using environmental rewards with the PPO algorithm. We then generate a few rollouts with the trained policy and sub-sample state transitions from these rollouts. We create the expert dataset with 50 such transitions for (Half- Cheetah, Walker, Hopper, } and 1000 transitions for Humanoid.
Absolute value for the expert scores. In Figure we plot the average episodic returns achieved by the learner in the 1-MDP, normalized to the returns achieved by the expert in the e-MDP. For completeness, Table reports the absolute value of the expected returns obtained by the expert per episode (max-episode-lengthh 1000 timesteps) in the e-MD
Task
Returns
Cheethh
6441
Walker
4829
Hopper
3675
Ant
5765
Humanoid
9678
Table 3: Expert's performance in c-MDP
12

---

Under review as a conference paper at ICLR 2022
$5
$5
S4
S4
$3
$3
$2
$2
31
$1
$1
5
Advisor actions
5;
Expert states
Path taken by the learner
Favorable next state
Unfavorable next state
Figure 4: Illustration the expert states, the actions by the advisor policy, and the path taken by the learner policy in a grid- world environment.
A.3 ILLUSTRATION OF THE ADVISOR AND THE LEARNER GRID- WORLD
We discuss the differences between the advisor policy and the learner policy with the help of an illustration in the deterministic grid-world environment (Figure 4). The leftmost plot shows the demonstrated expert states {s1, $2, $3, $4, $5). The e-MDP and the I-MDP the same grid- world but with the following transition dynamics mismatch while the e-MDP allows diagonal hops such that the shown expert transitions consecutive, in the 1-MDP, the agent is restricted only horizontal and vertical movements to the nearby cells.
As discussed in Section 3.2, we seek train an advisor such that the dissimilarity between the destination state achieved with the expert policy in e-MDP and that achieved with the advisor in I-MDP minimized. The middle plot in Figure 4 shows the actions (marked with red arrows) that an optimal deterministic advisor would take. Note that the advisor is trained only on the expert states to optimize for the next state; not trained for long-ten term behavior and thus struggles with decision-making
the non-expert states. At the state 83. the green states are the favorable states (minimum distance to 64), while the red states are the unfavorable states (maximum distance 84). In the I-MDP, the dataset {st, Ta (s:) is then used to train a learner that runs in a closed feedback loop with the environment.
The learner is trained with the standard adversarial imitation learning objective to match its station- ary discounted state-action distribution with that of the generated dataset {s,, (s;)) in the I-MDP. With adversarial imitation learning, the learner reproduces the demonstrated action the demon- strated state, while also being trained to navigate back to the manifold the demonstrated states when it encounters non-demonstrated states, thus enabling long-horizon imitation. The advisor, that trained to optimize just the immediate action at each expert state (similar principle to behavior cloning), is incapable of oonghhorizon imitation. The rightmost plot in Figure 4 shows the trajectory that an optimal learner may produce.
A.4 COMPARING THE ADVISOR AND THE LEARNER PERFORMANCE
Table 4, we report the average episodic returns achieved by the advisor and the learner in the I-MDP at the end of the training, normalized to the returns achieved by the expert in e-MDP. We show data for three environments and all the types of transition dynamics mismatch considered in the paper Heavy (H), Drag (D), Light (L). We note that the learner outperforms the advisor by a substantial margin for all the scenarios. This because, unlike the learner, the advisor is not optimized for long-horizon performance (ef. Appendix A.3).
13

---

Under review as a conference paper at ICLR 2022
AlLO-advisor
AlLO-learner
(H)
0.19
0.57
HalfCheetah
0.27
0.82
0.18
0.63
(H)
0.07
0,50
Walker
(D)
0.14
0.78
(L)
0.17
0.81
(H)
0.53
0.67
Hopper
(D)
0,48
0.77
(L) 0.66
0.84
Table 4: Normalized average episodic returns achieved by the advisor and the learner in the (-MDP.
A.5 COMPARISON TO BEHAVIORAL CLONING AND RL BASELINES
In this section, compare AILO with BCO (Torabi al., 2018a). BCO is an ILO algorithm that first learns an inverse dynamics model p(a]s, s') ) in the 1-MDP, and then uses this model to infer the expert actions from the given expert teaeetrranstion dataset. These inferred actions are used train the learner agent using the standard behavioral cloning strategy.
Table reports the average episodic returns achieved by BCO and AIL.O, normalized the returns achieved by the expert in e-MDP. We show data for all the environments and all the types of tran- sition dynamics mismatch considered the paper Heavy (H), Drag (D), Light (L). We note that BCO performs much worse than AILO in all scenarios, with the exception of Ant (H). This can be at- tributed to two factors: 1.) the compounding errors problem in the behavioral cloning approach, and 2.) BCO does not explicitly account for the transition dynamics mismatch while inferring actions from the learned inverse dynamics model.
Although the agent does not receive rewards in the I-MDP in the imitation learning setup, we sider an oracle RI. baseline that trains the agent to maximize the rewards in the I-MDP using the clipped-ratio PPO algorithm (Schulman et al., 2017). This baseline provides us with estimate the upper bound of the performance of AILO. Similar to the AILO numbers, the RL baseline scores normalized the returns achieved by the expert in e-MDP. The data in Table 5 shows that there is a lot of scope to optimize AILO further, especially for environments such as Walker and Ant.
BCO
AILO
RL I-MDP
(H) 0.28 0.02
0.57 + 0.08
0.67
HalfChectah
(D)
0.47 0.03
0.82 082:100 0.04
1.06
(L)
0.09 1
0.63 0.05
0.78
(H) 0.03 + 0.02
0.50 0.14
0.72
Walker
0.04 + 0.01
0.78 + 0.03
1.13
0.04 0.03
0.81 = 0.08
0.93
(H) 0.10 0.01
0.67 0..55
0.65
Hopper
DD) 0.09 0.01
0.77 = 0.04
0.76
LL) 0.11 = 0.05
0.84 = 0.03
0.86
(H) 0.73 0.01
0.65 1 0.04
1.04
Ant
(D) 0.02 + 0.00
0.19 + 0.13
0.83
(L) 0.06 + 0.02
0.02 = 0.14
0.75
(H) 0.02 + 0.00
0.07 + 0.00
0.48
Humanoid
()) 0.01 0.00
0.78 0.05
0.71
(L) 0.01 = 0.00
0.26 - 0.02
0.32
Table 5: Normalized average episodic returns for BCO and RL with rewards in I-MDP.
A.6 COMPARISON WITH I2L
In this section, we compare AILO with 12L (Gangwani Peng. 2020). 12L is an algorithm that builds top of adversarial imitation learning and introduces specific components to handle the dy-
14

---

Under review as a conference paper at ICLR 2022
namics mismatch between the e-MDP and the I-MDP. AILO and I2L share the principle of curating an intermediate dataset in the I-MDP, which is then used to train the learner with adversarial imita- tion learning. In the case of I2L. the intermediate dataset is created by applying filtration step the trajectory generated in the I-MDP, by using a learned Wasserstein critic to rank the trajectories. Differently, in AILO, the intermediate dataset is obtained by the application of the advisor policy on the demonstrated expert states. These are two quite distinct and orthogonal approaches that could potentially be blended to create an effective intermediate dataset for the learner agent.
Table 6 reports the average episodic returns achieved by 12L, along with the performance of the other algorithms considered in this paper. We normalize all the scores the returns achieved by the expert in e-MDP and highlight the best algorithm in each row with bold. The table verifies that I2L works very well for IL.O under dynamies model disparity. Among the 15 Senvroommentmmismann GAIfO is the best method for 3. VAIL. for 1, I2L. for 5, and AILO for 7. This clearly indicates that a single approach may be insufficient solve large variety of ILOO problems and opens up an interesting future direction of understanding the unique characteristics of these algorithms (e.g. the contrasting approaches to create the intermediate dataset 12L and AILO) towards designing efficient ILO algorithms.
scenarios,
GATFO
VAIL
12L.
AILO
(H) 0.34 + 0.17
0.49 + 0.09
0.59 + 0.04
0.57 + 0.08
HalfCheetah
(D) 0.41 0.19
0.65 + 0.07
0.80 0.22
0.82 + 0.04
(L) 0.52 = 0.03
=.20
0.48 0.19
0.63 = 0.05
(H) 0.13 = 0.07
0.34 = 0.23
0.56 0.09
0.50 0.14
Walker
(D) 0.50 0.29
0.78 1 0.03
0.64
0.24
0.78 1 0.03
(L) 0.86 1 0.07
0.81 1 0.05
0.76 + 0.05
0.81 1 0.08
(H) 0.44 0.04
0.50 + 0.10
0.67 0.07
0.67 0.05
Hopper
(D) 0.79 + 0.03
0.78 + 0.04
0.78 + 0.05
0.77 + 0.04
(L) 0.70 = 0.06
0.75 1 0.07
0.70 0.08
0.84 = 0.03
(H) 0.69 = 0.05
0.55 + 0.06
0.48 0.22
0.65 = 0.04
Ant
(D) 0.09 + 0.12
0.14 + 0.06
0.17 + 0.10
0.19 + 0.13
(L) 0.03 0.14
0.02 + 0.14
0.13 0.11
0.02 + 0.14
(H) 0.06 + 0.01
0.06 + 0.00
0.07 + 0.01
0.07 + 0.00
Humanoid
(D) 0.57 0.08
0.65 + 0.10
0.33 0.22
0.78 0.05
(L) 0.18 = 0.02
0.15 + 0.02
0.45 = 0.21
0.26 0.02
Table 6: Normalized average episodic returns for different IL.0 algorithms.
15


Reviews:
[{"review_id": "_i6GVCnFNr1", "reviewer": "Reviewer_E4NG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using RED for estimating the joint density for consecutive states for imitation learning from observation is interesting, and the overall structure of the paper is clear and easy to follow. \n\nEq. (3) seems confusing: $J(\\pi_a)$ is a function of $\\pi_a$, but on the RHS of the equation it is taking a maximum over $\\pi_a$. I am not sure what are taking maximum over since $\\pi_a$ should be fixed in this situation as a input of $J(\\pi_a)$. Also, the last equation does not hold if it is taking $\\max$, it only hold if it is taking $\\arg\\max$. I do understand what the authors are trying to show here but this may need some revision. Similarly to Eq. (4).\n\nAlso the usage of the intermediate policy still lacks some intuition: if the advisor policy is able to recover the expert state trajectory, then optimizing the expected return s.t. the state distributions are the same should solve the problem. If the advisor policy is not able to fully recover the expert state trajectory, how would we expect the learner policy to recover the state distribution of the expert policy since the learner policy is matching the state-action distribution of the advisor policy (Eq. above Eq. (2))? For example, how is the learning path (the squiggly line) in Fig. 1 achieved by the proposed objectives?\n\nThe setting itself is not new: learning from expert and environments with different dynamics has also been proposed in many papers with different names: policy transfer, policy adaptation, domain transfer, sim2real, etc.\n\nThe main major issue is the experiment: the current baselines are weak since they are just LfO methods but do not specifically deal with dynamics mismatch. Even in this situation the proposed method is outperformed or evenly matched by the baselines. This is not the main issue, the main issue is the lack of comparison with baselines under the same setting as the proposed method, such as [1]. (Possibly, plus works in the above papers (policy transfer, policy adaptation, domain transfer, sim2real, ...) that work in the same setting.) Only comparing with baselines that do not specifically deal with dynamics mismatch does not seem very fair, and thus the experiment is not thorough enough for such an empirical paper. \n\n[1] Tanmay Gangwani and Jian Peng. State-only imitation with transition dynamics mismatch. arXiv\npreprint arXiv:2002.11879, 2020.", "summary_of_the_paper": "The paper proposes a new algorithm for solving LfO task where the dynamics are different in the expert environment and the learner environment. The algorithm first trains an intermediate policy in the learner environment that mimics the expert's state trajectory and then the learner can just mimic this intermediate policy in the learner environment. The authors further show in the experiment that the proposed method outperforms previous LfO algorithms in several locomotion tasks. ", "summary_of_the_review": "Some mathematical representations still need some improvement. The experiment is not convincing enough. I would recommend a weak reject for the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L7JGjOFQVsE", "reviewer": "Reviewer_hbsF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strengths}$:\n- Empirically, they demonstrate their algorithm is more robust than baselines in the mismatch setting. \n- The approach is quite innovative\n- The problem they tackle is very interesting, as in realistic scenarios, their will usually be such dynamics mismatch.\n\n$\\textbf{Weaknesses}$:\n- I found the paper slightly hard to parse, i.e. I had a hard time understanding for instance why we could not directly leverage the advisor policy, and why the learner policy trained so its occupancy would match the advisor's would be better suited than the advisor's. It would help if this could be further clarified, maybe with an intuitive visual example showing the expert, the advisor and the learner policies?\n", "summary_of_the_paper": "This paper tackles the problem of imitation learning from observations only (no access to expert actions), in the setting where expert and agent are in different MDPs. More precisely, transition dynamics are different. The proposed approach consists in instantiating two agents, an advisor and a learner, such that the advisor is trained for its next state distribution to be close to the one of the expert, while the learner is trained so its state-action occupancy matches the one of the advisor.\n\nThe reason they instantiate this framework is that otherwise, next states may be unreachable by the agents hence the trained discriminator (if directly optimizing a GAIL loss between agent and expert occupancies) may perfectly classify and learning may then be challenging. Instead, in their setup the agent learns through a proxy living in the same MDP - the advisor.\n\nContributions:\n- They introduce an algorithm aiming to mitigate the issue of agent and expert having dynamics mismatch\n- Their demonstrate improved robustness to such mismatch in continuous control environments\n", "summary_of_the_review": "I find the paper in this current form to be borderline due to the fact that it is hard to parse and to understand why the framework is instantiated in the way it is. It is still an interesting contributions, and empirical results are promising. Upon such clarifications to be made (along with potentially visual examples providing intuitions), I would be considering raising my score to a weak accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7YO5uXmgKne", "reviewer": "Reviewer_6eKr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Summary++:\nThe setup for this paper is super interesting.  The domain is a realistic one: there is some limited data collected by simply watching an expert interact with the environment, but, where the learner is different in some critical way.  The question is then how do we leverage the observations of the expert to expedite learning an agent.  If solved generally (see 5 + 6 below) this would be a brilliant addition to the literature. \n\nSections two and four are _very_ well written.  Section one motivates the problem well, but doesn’t provide much detail on what content will be provided in the paper.  Section three is not well written, does not clearly explain the method, and is instead a bit of a drawn-out narrative that is hard to follow.  The empirical validation presented is lacking.  Finally, there is only a very short supplement that only provides hyperparameter / architectural settings and no further insight.  \n\nThe crux of my critique is that I believe the empirical validation, to justify a method with this many moving parts, is lacking; and that the clumsy exposition of the methodology obfuscates why each component is necessary, and how they work in conjunction to create a performant policy.  While I believe there is a good method and a good conference submission in this line of work, it is not currently in that state.  \n\n### Major comments:\n1 - My main complaint with this work is the lack of clarity in how the advisor actually works and what it provides.  The advisor is this magic box that encapsulates the expert policy _and_ the constraints/limitations of the learner.  The advisor is defined in learner space (as that is the only space we tractably have access to) and so how is the policy learned by the learner different to that of the advisor?  Why can’t the learner just imitate the advisor exactly (using, for instance, DAgger)?  If the learner can imitate the advisor exactly, then what is the point of the advisor?  Why not just learn the learner by whatever method you use to learn the advisor?  It is then also poorly explained how the learner is subsequently recovered from the expert demonstrations and the advisor policy.  Eq (2) is pointed to, but without much more detail as to the intuition that underlies this step.  \n\n2 - The method itself is also _very_ complicated.  As far as I can tell, there are at least four trained elements: the RED network, a reward network, an advisor policy and a learner policy.  That is a lot of stuff to specify, train, tune etc.  There is also no indication of how robust to hyperparameter/architecture settings the performance is.  i.e. If I happen to use a RED network with too-lower capacity, how is performance affected and how easy is it to diagnose that that is where the problem is?  I understand that all things RL are inherently a bit magic-boxey and can be sensitive, but even still, this method seems very convoluted/involved and is not sufficiently empirically validated to convince me that this complexity is warranted.\n\n3 - Following on from 1 & 2, is the advisor and/or RED even strictly required?  It seems, and I may have gotten it wrong here, that you are essentially pre-training some policies to match the expert as well as it can, and then refining the policies in places where the expert can’t be matched.  I can just about believe that RED provides a tractable method for learning a policy that keeps as close to the experts state evolution as possible.  The advisor policy then steers the learner towards those actions?  Why not just weight the ILO loss by the prediction from RED?  Or just initialize the learner to match the state distribution where it can, and then refine based on reward?  Or something more akin to THOR [Sun, Bagnel & Boots, 2018]?  If the authors can comment on and clarify this.  \n\n4 - I will say upfront that this may be a little unfair as a critique, so take this with a pinch of salt.  The absolute performance of the method (and even the baselines) is disappointing.  Although I don’t know this for sure, I am relatively confident that something like regular SAC applied directly to the environment would achieve better performance in fewer environment interactions.  Therefore, why are we even bothering to use the expert demonstrations if regular RL in the learner could achieve competitive performance?  I was desperately hoping that this method would leverage the “imperfect” demonstrations to obtain a learner policy with dramatically fewer environment interactions, even if the absolute/final performance was slightly lower than the best-possible performance.  As I mentioned above, there is a TON of internal mechanics in this method, and while I believe there is an algorithm that can do ILO in this mismatched domain, I am as of yet unconvinced that the AILO method presented is _the_ method for doing it -- both in terms of absolute performance and technical overhead.  \n\n4.1 - The results in this paper have not convinced me that this method successfully, expediently and robustly extracts information from demonstrations from a slightly different environment to expedite learning a policy.  \n\n4.2 - I think the following “baselines” are missing:  \n- SAC/RL on the learner (also to provide the true upper bound on performance of the learner).\n- DAgger on the learner using the misspecified expert policy. \n- Behavioural cloning (BC) on just state-actions from the expert.  \n- Initializing a learner to the BC learned-policy and then performing RL to refine that policy. \n\nSome of these are not practically applicable methods in real-life given the domain, but are, in my opinion, required to elucidate the performance of the method (bearing in mind that baselines are also there to help the _reader understand_ the performance and limitations of the method, as opposed to simply whether or not it works).  \n\n5 - I would also have liked to have seen the method applied to a smaller-scale toy example.  You even motivate the problem with a simple gridworld example -- why wasn’t the method applied to such a gridworld?  A simple example would help you pull apart the method and verify that all the components are working as intended, and would also allow instructive and intuitive visuals, graphs, diagrams, heatmaps etc to be generated to further help the reader _understand_ the method and its components, as opposed to simply posting a grid if graphs showing that it achieves half-way reasonable performance.  \n\n6 - I think there may also be a missed opportunity exploring (at least in principle) connecting IL and sim-to-real domains.  Sim-to-real seems to me to be a domain where this method (with some tweaking) could have some utility, as that is entirely derived from mismatch between two domains.  The advisor seems in-part to be a more flexible representation of an expert policy (learned in sim) that accommodates for the imperfections of the learner (learned at test-time in real).  There may be nothing in this, and it is just a suggestion, but the setups are so similar that _any_ comparison is conspicuous by absence -- even just commenting and discounting.  \n\n### Minor comments:\n\nA - I would like to see a more specific explanation of the method earlier in the paper.  The advisor is loosely mooted early on, but is only really introduced with any specificity at the bottom of page 3.\n\nB - Only proper nouns should be capitalized (“Reinforcement Learning” -> “reinforcement learning”;  “infinite horizon Markov Decision Process” -> “infinite horizon Markov decision process”).\n\nC - Figure 1 should be improved to improve its clarity.  I have no idea what the “squiggly” lines represent.  Are these lines something we will learn?  The second part of that figure also offers no helpful content as it is too high-level.  I would also move it up to be a banner figure at the top of page two.  This would help frame your approach more succinctly and earlier on.  I would also consider (even more simply) pseudocoding the algorithm and placing it alongside the diagram on page two to drive home the method early. \n\nD - Figure 2 could easily be compressed to save space if required.\n\nE - The algorithm block is totally unhelpful.  It should either be made more precise using full mathematical notation, or, made more intuitive using pure pseudocode and high-level objects.\n\nF - Releasing code for this is ultra important because I do not believe I could reproduce the results in this work from the prose and limited supplement alone.\n", "summary_of_the_paper": "This paper tackles imitation learning from observations under model mismatch.  This is a realistic consideration when tackling more complex or real environments where methods such as IL (and its variants) may see dramatic improvements over more conventional RL-based approaches (to the point of being the only feasible approach).  Model mismatch makes existing ILO methods unstable (in theory, at the very least).  To ameliorate this, the paper introduces an _advisor_ policy, which learns from the expert demonstrations but under the constraints implicitly imposed by the capacity of the learner or the learners dynamics.  The advisor is then used to train the learner.  The method is empirically validated on some standard benchmarks and appears to perform reasonably well.   \n", "summary_of_the_review": "This work is interesting and shows promise, and if it were to work, would be a useful addition to the literature.  However, I am unconvinced by the method itself.  The method is quite involved, with lots of moving parts.  Furthermore, the presentation of the core of the method (the advisor) is remarkably poor, considering that it is _the_ contribution and that the rest of the paper is so well written.  The empirical validation is also a little lacklustre, both in terms of range and actual performance.  I also have more subjective issues with aspects of the paper (see comments above) that, while it would be unfair to reject solely on those issues, dissuade me from giving the authors the benefit of the doubt here.  Therefore, I recommend that this paper is not accepted for inclusion in this review cycle;  but note that with some reworking this paper could make a good submission to a good conference in the near future.  \n\nGood luck.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submitted paper considers the very interesting problem of imitation learning from observations under transition model disparity. The reviewers recommended 2x weak accept and 1x weak reject for the paper. Main concerns about the paper regarded clarity of the presentation, complicatedness of the proposed method, and experimental validation. During the discussion phase, the authors addressed some of the comments and provided an update of the paper providing additional details. While some of the reviewers' concerns still stand, I think the addressed problem is very relevant and the proposed method can be (with clarifications and improvements of the presentation) be interesting to parts of the community. Hence I am recommending acceptance of the paper. Nevertheless, I strongly urge the authors to carefully revise their paper, and taking the reviewers' concerns carefully into account when preparing the camera ready version of the paper.
Decision: Accept (Poster)