MobILE: Model-Based Imitation Learning From Observation Alone
Authors: Rahul Kidambi
Jonathan Daniel Chang
Wen Sun
Venue: NeurIPS 2021 Poster (2021)
Keywords: Imitation Learning from Observation Alone
Imitation Learning
Imitation Learning Theory

Abstract: This paper studies Imitation Learning from Observations alone (ILFO) where the learner is presented with expert demonstrations that consist only of states visited by an expert (without access to actions taken by the expert). We present a provably efficient model-based framework MobILE to solve the ILFO problem. MobILE involves carefully trading off exploration against imitation - this is achieved by integrating the idea of optimism in the face of uncertainty into the distribution matching imitation learning (IL) framework. We provide a unified analysis for MobILE, and demonstrate that MobILE enjoys strong performance guarantees for classes of MDP dynamics that satisfy certain well studied notions of complexity. We also show that the ILFO problem is strictly harder than the standard IL problem by reducing ILFO to a multi-armed bandit problem indicating that exploration is necessary for solving ILFO efficiently. We complement  these theoretical results with experimental simulations on benchmark OpenAI Gym tasks that indicate the efficacy of MobILE. Code for implementing the MobILE framework is available at https://github.com/rahulkidambi/MobILE-NeurIPS2021.

MobILE: Model-Based Imitation Learning From Observation Alone
Rahul Kidambi*
Jonathan D. Chang
Amazon Search & Al Berkeley CA 94704.
CS Department, Comeell University Ithaca NY 14853.
rk773@cornell.edt
jdc396@cornell.. edu
Wen Sun
CS Department, Cornell University Ithaca NY 14853.
vB4558corne11 edu
Abstract
This paper studies Imitation Learning from Observations alone (ILFO) where the learner is presented with expert demonstrations that consist only of states visited by an expert (without access to actions taken by the expert). We present a provably efficient model-based framework MobILE to solve the ILFO problem. MobILE in- volves carefully trading off strategic exploration against imitation - this is achieved by integrating the idea of optimism in the face of uncertainty into the distribu- tion matching imitation learning (IL) framework. We provide unified analysis for MobILE, and demonstrate that MobILE enjoys strong performance guaran- tees for classes of MDP dynamics that satisfy certain well studied notions of structural complexity. We also show that the ILFO problem strictly harder than the standard IL problem by presenting an exponential sample complexity separation between IL and ILFO. We complement these theoretical results with experimental simulations ficacy of MobILE. Code for implementing the MobILE framework is available https: / /gi thub. com
benchmark OpenAl Gym tasks that indicate the ef-
ILEE -ewe-TLE--eeuuPPe IPS222.
Introduction
This paper considers Imitation Learning from Observation Alone (ILFO). In ILFO, the learner is presented with sequences of states encountered by the expert, without access to the actions taken by the expert, meaning approaches based on a reduction to supervised learning (e.g., Behavior cloning (BC) [49], DAgger [50]) are applicable. ILFO is more general and has potential for applications where the learner and expert have different action spaces, applications like sim-to-real [56, 14] etc.
Recently, [59] reduced the ILFO problem to sequence of one-step distribution matching problems that results in obtaining a non-stationary policy. This approach, however, is sample inefficient for longer horizon tasks since the algorithm does effectively reuse previously collected samples when solving the current sub-problem. Another line work considers model-based methods to infer the expert's s actions with either inverse dynamics [63] forward dynamics [16] model; these recovered actions are then fed into an II. approach like BC to output the final policy. These works rely on stronger assumptions that only satisfied for Markov Decision Processes (MDPs) with injective transition dynamics [68]; we return to this in the related works section.
Work initiated when RK was post-doc at Cornell University; work done outside Amazon.
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

We introduce MobILE Model-baser Imitation Learning and Exploring, a model-based framework, to solve the ILFO problem. In contrast to existing model-based efforts, MobILE learns the forward tran- sition dynamics model-a quantity that is well de- fined for any MDP. Importantly, MobILE combines strategic exploration with imitation by interleaving a model learning step with a bonus-based, optimistic distribution matching step perspective, to the best of our knowledge, that has not been considered in Imitation Learning. MobILE has the ability to au- tomatically trade-off exploration and imitation. It simultaneously explores to collect data refine the model and imitates the expert wherever the learned model is accurate and certain. At a high level, the- oretical results and experimental studies demonstrate that systematic exploration is beneficial for solving ILFO reliably and efficiently, and optimism is both theoretically sound and practically effective approach for strategic exploration in ILFO (see Figure for comparisons with other ILFO algorithms). This paper extends the realm of partial information problems (e.g. Reinforcement Learning and Bandits) where optimism has been shown to be crucial in obtaining strong performance, both in theory (e.g.. E [30], UCB [3]) and practice (e.g., RND [10]. This paper proves that incorporating optimism into the min-max IL. framework [69, 22, 59] is beneficial for both the theoretical foundations and empirical performance of ILFO.
Expert Performance Normalized
12
1.0
0.8
0.6
014
0.2
0.0
Reacher-p
Hegporee?
Whlkarldr22
Figure 1: Expert performance normalized scores of ILFO algorithms averaged across seeds in environments with discrete action spaces (Reacher-v2) and continuous action spaces (Hopper-v2 and Walker2d-v2).
Our Contributions: We present MobILE (Algorithm 1), a provably efficient, model-based frame- work for ILFO that offers competitive results in benchmark gym tasks. MobILE can be instantiated with various implementation choices owing to its modular design. This paper's 's contributions are:
1. The MobILE framework combines ideas of model-based learning, optimism for exploration, and adversarial imitation learning. MobILE achieves global optimality with near- vear-optimal regret bounds for classes of MDP dynamics that satisfy certain well studied notions complexity. The key idea of MobILE is to use optimism to trade-off imitation and exploration.
2. We show exponential sample complexity gap between ILFO and classic IL where one has access to expert's actions. This indicates that ILFO is fundamentally harder than IL. Our lower bound on ILFO also indicates that to achieve near optimal regret, one needs to perform systematic exploration rather than random or no exploration, both of which will incur sub-optimal regret.
3. We instantiate MobILE with a model ensemble of neural networks and disagreement-based bonus. We present experimental results on benchmark OpenAl Gym tasks, indicating MobILE compares favorably to or outperforms existing approaches. Ablation studies indicate that optimism indeed helps in significantly improving the performance in practice.
1.1 Related Works
Imitation Learning (IL) is considered through the lens of types of approaches: (a) behavior cloning (BC) [45] which casts IL as reduction to supervised or full-information online learning [49, 50], or, (b) (adversarial) inverse RL [40, 1, 69, 17, 22, 29, 18], which involves minimizing various distribution divergences to solve the II problem, either with the transition dynamics known (e.g.. [69]), or unknown (c.g.. [22]). MobILE does not assume knowledge of the transition dynamics, is model-based, and operates without access to the expert's actions.
Imitation Learning from Observation Alone (ILFO) [59] presents a model-free approach FAIL that outputs a non-stationary policy by reducing the ILFO problem into a sequence of min-max max problems, one per time-ster step. While being theoretically sound, this approach cannot share data across different time steps and thus is not data efficient for long horizon problems. Also FAIL in theory only works for discrete actions. contrast, our paper learns a stationary policy using model-based approaches by reusing data across all time steps and extends to continuous action space. Another line of work [63, 16, 66] relies on learning an estimate of expert action, often through the use of an inverse dynamics models, p (als,, s'). Unfortunately, an inverse dynamics model is not well defined

---

in many benign problem instances. For instance, [68, remark 1, section 9.3] presents an example showing that inverse dynamics isn't well defined except in the case when the MDP dynamics is injective (i.e., no two actions could lead to the same next state from the current state. Note that even deterministic transition dynamics doesn't imply injectivity of the MDP dynamics). Furthermore, ILPO [16] applies to MDPs with deterministic transition dynamics and discrete actions. MobILE, on the other hand, learns the forward dynamics model which is always unique and well-defined for both deterministic and stochastic transitions and works with discrete and continuous actions. Another line of work in ILFO revolves around using hand-crafted cost functions that may rely on task-specific knowledge [44, 4, 53]. The performance of policy outputted by these efforts relies on the quality of the engineered cost functions. In contrast, MobILE does not require cost function engineering.
Model-Based RL has seen several advances [61, 36, 13] including ones based on deep learning (e.g., (3.g..[34, 19, 38, 24, 37, 65]). Given MobILE's modularity, these advances in model-based RL can be translated improved algorithms for the IIFFO problem. MobILE bears parallels provably efficient model-based RL approaches including E [31, 27], R-MAX [7], UCRL [23], UCBVI (5), Linear MDP [67], LC3 [25], Witness rank [58] which utilize optimism based approaches trade-off exploration and exploitation. Our work utilizes optimism to trade-off exploration and imitation.
2
Setting
We consider episodic finite-horizon MDP M (S, 80). where S,A are the state and action space, P S x A + S is the MDP's transition kemel,, H is the horizon, so is a fixed initial state (note that our work generalizes when we have a distribution over initial states), and c is the state-dependent cost function e S + [0, 1]. Our result can be extended to the setting where e S x S + [0, 1], i.c., the ground truth cost c(s, s') depends on state and next state pairs. For analysis simplicity, we focus on e S - [0, 1].
We denote dz. E ASS x A) as the average state- action distribution of policy x under the transition ker- o als0, T,, P), where Pr(s; = 8,0, als0; π,, P) the probability of reaching (s at time step t starting from so by following π under transition d' to denote a state is sampled from the state-wise distribution which marginalizes action over dī i...
nel P. i.e., (7,(s,a) a) == if Em Pr(s;
kernel P. We abuse notation and write
i.e., ds) == में Σ Pr(s, = s|so, म, P). For a given cost function ʃ S - [0, 1]. V denotes the expected total cost of ग under transition P and cost function f. Similar to II. setting, in ILFO, the ground truth cost c is unknown. Instead, we can query the expert, denoted n² S A(A). Note that the expert n² could be stochastic and does not have to be the optimal policy. The expert, when queried, provides state-only demonstrations P* (-|s,, 0z) and a, (-[se).
{so, 81 .... $H) ), where 8441
The goal is to leverage expert's ' state-wise demonstrations to learn a policy æ that performs as well as Tf in terms of optimizing the ground truth cost with polynomial sample complexity on problem parameters such as horizon, number of expert samples and online samples and underlying MDP's complexity measures (see section 4 for precise examples). We track the progress of any (randomized) algorithm by measuring the (expected) regret incurred by a policy defined as E[V*) - V** as function of number of online interactions utilized by the algorithm to compute π.
2.1 Function Approximation Setup
Since the ground truth cost c is unknown, we utilize the notion of function class (i.e., discriminators) F C + [0, 1) to define the costs that can then be utilized by a planning algorithm (e.g. NPG (26]) for purposes of distribution matching with expert states. If the ground truth depends (s, s'), use discriminators F C x S + [0, 1]. Furthermore, we use a model class P C x A + A to
capture the ground truth transition p* For the theoretical results in the paper, we assume realizability:
Assumption 1. Assume F and P captures ground truth cost and transition, i.e.
We will use Integral probability metric (IPM) with F as our divergence measure. Note that E F and c: S + [0, 1]. then IPM defined as max Eg--a= - Egngr f(s) directly upper
"Without any additional assumptions, in ILFO, learning to optimize action- deoonneeeendent cost c(s, a) (or c(s, a, s') is not possible. For example, if there are two sequences of actions that generate the same sequence of states, without seeing expert's preference over actions, we do not know which actions commit to.
3

---

Algorithm MobILE: The framework of Model-basee Imitation Learning and Exploring for ILFO
1: Require: IPM class F, dynamics model class P. policy class II, bonus function class B, expert dataset D. { fi==1
2: Initialize policy E II, replay buffer D-1 = 0.
3: for t = 0,- T do
H-1 U 8n Append replay 4: Execute in true environment Pa to get samples 71 {8k, as )
buffer D; 1 Ur.
5: Update model and bonus: P.+1 8 x A + S and 1-11 8 A R+ using buffer Dr.
6: Optimistic model-based min-max IL: obtain Te+11 by solving equation (1) with P++1. b De.
7: end for
8:
Return TT.
bounds sub-optimality gap V V**,, where V is the expected total cost of ग under cost function This justifies why minimizing IPM between two state distributions suffices [22, 59]. Similarly, e depends on 8, we can simply minimize IPM between two state-next state distributions, i.e., maxy Bs,s' -d= f(s, =''""d"" fs, s') where discriminators now take (s, s') as input.
To permit generalization, we require P have bounded complexity, For analytical simplicity, we assume F is discrete (but exponentially large), and we require the sample complexity of any PAC algorithm to scale polynomially with respect to complexity In([F)). The In [F complexity can be replaced to bounded conventional complexity measures such as Rademacher complexity and covering number for continuous F (e.g., F being Reproducing Kernel Hilbert Space).
Algorithm
We introduce MobILE (Algorithm 1) for the ILFO problem. MobILE utilizes (a) function class F for Integral Probability Metric (IPM) based distribution matching, (b) a transition dynamics model class P for model learning, (c) bonus parameterization B for exploration, (d) policy class Il for policy optimization. At every iteration, MobILE (in Algorithm 1) performs the following steps:
1. Dynamics Model Learning: execute policy in the environment online to obtain state-action-next state (s, a, 8') triples which appended the buffer DD Fit transition model P on DD
Bonus Design: design bonus incentivize exploration where the learnt dynamics model is uncertain, i.e. the bonus b(x, a) large at state where P(-Is.a) uncertain terms of estimating (*(+ss,a), while b(s, a) is small where P(-|s, a) is certain.
3. Imitation-Explorattin tradeoff: Given discriminators F, model P, bonus and expert dataset De, perform distribution matching by solving the model-based IPM objective with bonus:
Tt+1 f- arg min max L(T, f: f.P,b,D.) B(s,a)=33 [f(s) b(s, a)) EEwD. [f(s)), (1)
sell JEF
where ffwD.f(s):= F(ep))))
Intuitively, the bonus cancels out discriminator's power in parts of the state space where the dynamies model P is not accurate, thus offering freedom for MobILE to explore. We first explain MobILE's components and then discuss MobILE's ss key which is to trade-off exploration and imitation.
3.1 Components of MobILE
This section details MobILE's components.
Dynamics model learning: For the model fitting step in line 5, we assume that we get a calibrated model in the sense that: [-|s a) (=|ss, a)ll < (r a), Vs, for some uncertainty measure ar(s, a), similar model-based RL works, e.g. [12]. We discuss ways to estimate o (s, a) in the bonus estimation below. There are many examples (discussed in Section 4) that permit efficient
we slightly abuse notation here and denote d' as the average state-next state distribution of क. i... d' (s, == (s) J. (sls) (s)

---

estimation of these quantities including tabular MDPs, Kernelized nonlinear regulator, nonparametric model such as Gaussian Processes. Consider general function class & CSS X A + S, one can learn De via solving regression problem, i.e.,
gt
argmin DEG
Ig(5,a) a) - s'12, 5,0,"'ED,
Σ
and setting P:(-|s,a) a) = N as, where, a is the standard deviation of error induced by Ge In practice, such parameterizations have been employed in several settings in RL with & being multi-layer perceptron (MLP) based function class (e.g..[48)). In Section 4, we also connect this with prior works provable model-base RL literature.
Bonus: We utilize bonuses means to incentivize the policy to efficiently explore unknown parts of the state space for improved model learning (and hence better distribution matching). With the uncertainty measure 0x(88 a) obtained from calibrated model fitting, we can simply set the bonus (, a) = O(Hor(s, a)). How do we obtain (x a) in practice? For a gen- eral class G, given the least square solution 9t, can define version space Gr as: Gr {9 Eg: E' h=
- llg(s%, af,)) (5% 4/11 S #t with being a hyper parameter. The version space 9t is an ensemble of functions g E 9 which has training error on D; almost as small as the training eтгоr of the least square solution It- In other words, version space G, contains functions that agree on the training set D.. The uncertainty measure at (s, a) is then the maximum disagreement among models in Gr, with or (s, a) x fuee, Il [s, a) f2(s, a))l2. Since g E Gr agree on DD a large dz a) indicates (s, a) is novel. See example 3 for more theoretical details.
Empirically, disagreement among an ensemble [41, 6, 11, 43, 37] is used for designing bonuses that incentivize exploration. We utilize an neural network ensemble, where each model is trained on D, (via SGD on squared loss Eq. 2) with different initialization. This approximates the version space Gr. and the bonus is set as a function of maximum disagreement among the ensemble's predictions.
Optimistic model- min-max IL: For bosel imitation (line 6), MobILE takes the current model P, and the discriminators F as inputs and performs policy search to minimize the divergence defined by Pn, and F: de( x²) max [Bs.add, (f()) (s,, a)) E.-a-= f(s) Note that, for a fixed ग, the arg f is identical with or without the bonus term, since E8 Ls be a) is independent of f. In our implementation, we use the Maximum Mean Discrepancy (MMD) with Radial Basis Function (RBF) kernel to model discriminators F. We compute argmin, de (TT,, T") by iteratively (I) computing the argmax discriminator f given the current म, and (2) using policy gradient methods (e.g.. TRPO) to update inside P. with f - by as the cost. Specifically, to find m (line 6), we iterate between the following two steps:
f(8) - Eo-D- f(),, PG Step: = - V V.. 1. Cost u = argmax Esdd p. JCF
P..f
where the PG step uses the learnt dynamics model P; and the optimistic IPM cost f(s) by bs(s,a). Note that for MMD, the cost update step has closed-form solution.
3.2 Exploration And Imitation Tradeoff
We note that MobILE is performing an automatic trade-off between exploration and imitation. More specifically, the bonus is designed such that it has high values in the state space that have not been visited, and low values in the state space that have been frequently visited by the sequence of learned policies so far. Thus, by incorporating the bonus into the discriminator f E F (e.g.. f(s.a) = fs) (s(s, a)), we diminish the power of discriminator f at novel state-action space regions, which relaxes the state-mat matthemmtching constraint (as the bonus cancels the penalty from the discriminators) at those novel regions so that exploration is encouraged. For well explored states, we foree the learner's 's states to match the expert's 's using the full power of the discriminators. Our work uses optimism (via coupling bonus and discriminators) to carefully balance imitation and exploration.
FFor MMD with kernel k. F= (w" ó(s, ()s,a))llllll 1) where di (6(s, a), ((5' ,a)) ((((s,),,
5

---

4 Analysis
This section presents general theorem for MobILE that the notion of information gain [57], and then specializes this result to common classes of stochastic MDPs such as discrete (tabular) MDPs, Kernelized nonlinear regulator [28], and general function class with bounded Eluder dimension [51].
Recall, Algorithm generates one state-action trajectory +f := {s/, af, S at iteration t and estimates model Pt based on D, = - ,(-1, We present our theorem under the assumption that model fitting gives model P and confidence interval of the model's prediction.
Assumption (Calibrated Model). For all iteration t with E N, with probability 1- 8, we have a model P, and its associated uncertainty measure o, S x A + Rt, such that for all s, a E x A
PM (-1s, ||
< min ())
Assumption 2 has featured in prior works (e.g., [12]) to prove regret bounds in model-based RI. Below demonstrate examples that satisfy the above assumption.
Example (Discrete MDPs). Given Dr denote Nss,a) a) as the number of times (8, appears in De and ((s, s') number of times (s,a,8') appears in D. We can set ())) = N ( a, ss)/N(s, a), Vs, a's . We can set orls, a) = Ô (v (S/N(s,)))
Example 2 (KNRs (28)). For KNR, we have PM P*(+|s,a) = - (W M(s 0²1) where feature mapping ((s, a) E IRd and [] a)lla S for all 8, a. We can learn P. via Kernel Ridge regression, i.e, Ge(s, = Weo(s, a) where
W. argmin
Σ
- -
W
S.a,s'ED:
3: "Iol(s, a)ll; 3t = [ 1 + 802 Ide (n() + 2 ln(( + In(4) + In (det(E))) det(A))))) and, H- - d(st, at) ( a A with À O.See Proposition 12 for more details.
where . || is the Frobenius norm. The uncertainty measure 01(s,a)
E, -E-
Similar to RKHS, Gaussian processes (GPs) offers a calibrated model [57]. Note that GPs offer similar regret bounds as RKHS; so we do not discuss GPs and instead refer readers to [12].
Example 3 (General class 9). In this case, assume we have (* = [x) o2) with 9* E G. Assume G is discrete (but could be exponentially large with complex- ity measure, In((g))), and supgeg.e,a (l a)g(s, I E RR. Suppose model learning step is done by least square: Dt = argmitl yeg
8 1 Com- pute a version space G. : [ at) - To (0) at) < zt.. where = (o
1 att)
and use this for uncertainty computation. In particular, set uncertainty (s)s.a) = max 93 (8, g2(s, a)ll2. ie., the maximum disagreement between any two functions in the version space G. Refer to Proposition 14 for more details.
The maximum disagreement above motivates our practical implementation where use ensemble of neural networks to approximate the version space and use the maximum disagreement among the models' predictions as the bonus. We refer readers to Section 6 for more details.
4.1 Regret Bound
We bound regret with the quantity named Information Gain I (up to some constant scaling factor) [57]:
T-1 H-1
IT = max EAI
ΣΣ
Σ
min (o 1).1
(3)
Alg
1=O
'the uncertainty measure (s a) will depend the input failure probability 8, which drop here for notational simplicity. When we introduce specific examples, we will be explicit about the dependence on failure probability 8 which usually is in the order of (n(1/8).
'The covariance matrix be generalized to any PSD matrix with bounded condition number.

---

where Alg is any adaptive algorithm (thus including Algorithm 1) that maps from history before iteration to some policy ne II. After the main theorem, we give concrete examples for Zr where we show that Zr has extremely mild growth rate with respect to T (i.e., logarithimie). Denote V as the expected total cost of under the true cost function e and the real dynamics Pa.
Theorem (Main result), Assume model learning is calibrated (i.e., Assumption 2 holds for all and Assumption holds. In Algorithm I, set bonus (e(s, a) == H min(or (0, ( There exists set of parameters, such that after running Algorithm for iterations, we have:
H VIT VT
In(TH[FI) N
E8
min
V
Vx 00
Appendix contains proof of Theorem 3. This theorem indicates that as long as Zy grows sublinearly o("), find policy that is at least good as the expert policy when T and N approach infinity. For any discrete MDP, KNR [28], Gaussian Processes models [57], and general g with bounded Eluder dimension ([52, 42]), can show that the growth rate of IT with respect to T is mild.
Corollary (Discrete MDP). For discrete MDPs, IT 5(HS 22 A) where S = [8], |A|. Thus:
V V - Vx = SASA VT
In(FFF)
E
min
H
IE LE......T-111
Note that Corollary 4 (proof in Appendix A.1) hold for any MDPs (not just injective MDPs) and any stochastic expert policy. The dependence on A, T is tight (see lower bound in 4.2). Now we specialize Theorem 3 to continuous MDPs below.
Corollary (KNRs (Example 2)). For simplicity, consider the finite dimension setting c: S x A + Rdd We can show that IT = Ô (Hd + Hdd, + Hd?) (see Proposition 13 for details), where d is the dimension of the feature (88, a) and d, is the dimension of the state space. Thus, we have?
V= = H3 /dd. + de
In([FF) N
min AE(O....,
VT
+ H
VT
Corollary (General G with bounded Eluder dimension (Example 3)). For general G, assume that has Eluder-dimensien (c(c) (Definition in [42]). Denote de dg(1//TTT The information gain is upper bounded IT = (Hdg + dg in(TTH|9 (THH)) (see Proposition 16). Thus,
va Ō H V de
In((Fl) N
E
min
+ H
IE (O.....
VT
Thus as long as G has bounded complexity in terms of the Eluder dimension [52, 42), MobILE with the maximum disagreement- based optimism leads to near-optimal guarantees.
4.2 Exploration in ILFO and the Exponential Gap between IL and ILFO
To show the benefit of strategic exploration over random exploration in ILFO, we present a novel reduction of the ILFO problem to bandit optimization problem, for which strategic exploration is known to be necessary [9] for optimal bounds while random exploration is suboptimal; this reduction indicates that benefit of strategic exploration for solving ILFO efficiently. This reduction also demonstrate that there exists an exponential gap in terms of sample complexity between ILFO and classic IL that has access expert actions. We leave the details of the reduction framework in Appendix A.4. The reduction allows us to derive the following lower bound for any ILFO algorithm.
Theorem 7. There exists an MDP with number of actions A 2 2, such that even with infinitely many expert data, any ILFO algorithm must occur expected commutative regret D( ((VAT).
Specifically we rely on the following reduction where solving ILFO, with infinite expert data, is at least as hard as solving an MAB problem with the known optimal arm's 's mean reward which itself
We use Ö to suppress log except the In([9)) and In(FFl) which present the complexity of JF and G.

---

occurs the same worst case VTT cumulative regret bound as the one in the classic MAB setting. For MAB, it is known that random exploration such as e-greedy will occur suboptimal regret (1T2/3).
Thus to achieve optimal VT rate, one needs to leverage strategic exploration (e.g., optimism).
Methods such BC for IL have sample complexity that scales as poly (n(A), e.g.. see [2, Theorem 14.3, Chapter 14] which shows that for tabular MDP, BC learns a policy whose performance is O(H² (n(()///// away from the expert's 's performance (here S is the number of states in the tabular MDP). Similarly, in interactive IL setting, DAgger [50] can also achieve poly In(AA) dependence in sample complexity. The exponential gap in the sample complexity dependence on A between IL and ILFO formalizes the additional difficulty encountered by learning algorithms in ILFO.
5 Practical Instantiation of MobILE
We present a brief practical instantiation MobILE's components with details in Appendix Section C. Dynamics model learning:Wa We employ Gaussian Dynamics Models parameterized by an MLP [48, 32], i.e., P(s, a) == (hg (s, a), ob where, ho(s, a) = 8 + as. MMPP (Se, ac), where, are MLP's trainable parameters, 8. (s ps)/ 4g = (a pa)/aa with ps: Pa (and o., aa) being the mean of states, actions (and standard deviation of states and actions) in the replay buffer D. Next, for (s, (,) 8') E D. A. = s' and da. is the standard deviation of the state differences A, D. We use SGD with momentum [60] for training the parameters of the MLP.
Discriminator parameterization: We utilize MMD as our choice of IPM and define the discriminator as (+) = w T (s), where, (s) are Random Fourier Features [46].
Bonus parameterization: We utilize the discrepancy between predictions of pair of dynamics models he, (s, a) and he (s, a) for designing the bonus. Empirically, we found that using more than two models in the ensemble offered little to no improvements. Denote the disagreement at any (s, a) as 0(s, a) = ||o,, (s, a) - heg (s, a)l. and &p = max(s,a)<<2) 5(s, a) is the max discrepancy of a replay buffer D. We set bonus as b(s, a) = X min(6(s, a) /0p where À >> 0 is a tunable parameter. PG oracle: We use TRPO [54] to perform incremental policy optimization inside the learned model.
Experiments
This section seeks to answer the following questions: (1) How does MobILE compare against other benchmark algorithms? (2) How does optimism impact sample efficiency/final performance? (3)
How does increasing the number of expert samples impact the quality of policy outputted by MobILE?
We consider tasks from Open AI Gym [8] simulated with Mujoco [62]: Cartpole-vi, Reacher-v2 v22 Swimmer- v2, Hopper-v2 and Walker2d-v2. We train an expert for each task using TRPO [54] until we obtain an expert policy of average value 460, 10, 38, 3000, 2000 respectively. We setup Swimmer-v2 v2, Hopper-v2, w2w.aller2dd v2 similar to prior model- based RL works [33, 39, 38, 48, 32].
We compare MobILE against the following algorithms: Behavior Cloning (BC), GAIL [22], BC- O [63]. ILPO [16] (for environments with discrete actions), GAIFO [64]. Furthermore, recall that BC and GAIL utilize both expert states and actions, information that is not available for ILFO. This makes both BC and GAIL idealistic targets for comparing ILFO methods like MobILE against. As reported by Torabi et al. [63], BC outperforms BC-O in all benchmark results. Moreover, our results indicate MobILE outperforms GAIL and GAIFO in terms of sample efficiency. With reasonable amount of parameter tuning, BC serves as a very strong baseline and nearly solves deterministic Mujoco environments. We use code released by the authors for BC-O and ILPO. For GAIL we use an open source implementation [21], and for GAIFO, we modify the GAIL implementation as described by the authors. We present our results through (a) learning curves obtained by averaging the progress of the algorithm across 5 seeds, and, (b) bar plot showing expert normalized scores averaged across seeds using the best performing policy obtained with each seed. Normalized score refers to ratio of policy's score over the expert score (so that expert has normalized score of 1). For Reacher-v2, since the expert policy has a negative score, we add an constant before normalization. More details can be found in Appendix C.
8

---

CartPole-vl (10 traj.)
Reacher-v2 (10 traj.)
Swimmer-v2 (40 traj.)
.08
(Value)
Return (Valuc)
Return (Value)
244
MoMTLE
BC:0
Expert
GAIL
14
to
ua
10
Online Samples
Online Samples
Online Samples
Hopper-v2 (10 traj.)
Walker2d-r2 (10 traj.)
2408
1.35
Normalized Score
Return (Valuo)
Return (Value)
2000
8:75
6:25
Online Samples
Online Samples
0:25
1.00
1.25
lc6
CariPoless
Beadhe-122
Swimmer-
Bopper-e2
Waller
Figure 2: Comparing MobILE (red) against BC (orange), BC-O (green), GAIL (purple), GAIFO (periwinkle), ILPO (green olive). The learning curves are obtained by averaging all algorithms over seeds. MobILE outperforms BC-O, GAIL and matches BC's behavior despite MobILE not having access to expert actions. The bar plot (bottom-right) presents the best performing policy outputted by each algorithm averaged across seeds for each algorithm. MobILE clearly outperforms BC-O, GAIFO, ILPO while matching the behavior of IL algorithms like BC/GAIL which use expert actions.
6.1 Benchmarking MobILE on MuJoCo suite
Figure compares MobILE with BC, BC-O, GAIL, GAIFO and ILPO. MobILE consistently matches or exceeds BC/GAIL's performance despite BC/GAIL. having access to actions taken by the expert and MobILE functioning without expert action information. MobILE, also, consistently improves upon the behavior of ILFO methods such as BC-O, ILPO, and GAIFO. We see that BC does remarkably well in these benchmarks owing determinism in the transition dynamics; in the appendix, consider a variant of the cartpole environment with stochastic dynamics. Our results suggest that BC struggles with stochasticity in the dynamics and fails solve this task, while MobILE continues to reliably solve this task. Also, note that we utilize 10 expert trajectories for all environments except Swimmer- v2; this is because all algorithms (including MobILE) present results with high variance. We include learning curve for Swimmer- with 10 expert trajectories in the appendix. The bar plot Figure shows that within the sample budget shown in the learning curves, MobILE (being model-basee algorithm), presents superior performance in terms of matching expert, thus indicating is more sample efficient than GAIFO, GAIL (both being model-free methods), ILPO and BC-O.
6.2 Importance of the optimistic MDP construction
Figure 3 presents results obtained by running MobILE with and without optimism. In the absence of optimism, the algorithm either tends to be sample inefficient in achieving expert performance or
CartPole-vl (10 traj)
Reucher-22 traj.)
Swimmer-x2 truj.)
Hopper-v2 2 traj.)
Walker2d-v2 (10 traj.)
Rotee Vale
Chilla Sangles
Tel
Calre Sungles
1et
Dallee Sumple
1A5
Driilo Samelee
1e5
Oollee Sarepies
Tes
Figure 3: Learning curves obtained by running MobILE with (red) and without (green) optimism. Without optimism, the algorithm learns slowly does not match the expert, whereas, with optimism, MobILE shows improved behavior by automatically trading off exploration and imitation.

---

completely fails to solve the problem. Note that without optimism, the algorithm isn't explicitly incentivized explore only implicitly exploring due noise induced by sampling actions. This, however, is not sufficient to solve the problem efficiently. In contrast, MobILE with optimism presents improved behavior and in most cases, solves the environments with fewer online interactions.
6.3 Varying Number of Expert Samples
Table 1:
Expert normalized score and standard de- viation policy outputted by MobILE when vary- ing number of expert trajectories Ej and Eg (specific values represented in parentheses)
Table I shows the impact of increasing the num- ber of samples drawn from the expert policy for solving the ILFO problem. The main takeaway that increasing the number of expert samples aids MobILE in reliably solving the problem (i.e.
Environment
E
Eg
Expert
Cartpole-vi
1,77 0.15
1.14 0 (10)
1.25
Reacher-v2
0.05 (10)
0.907
(20)
++1.11
with lesser variance).
Svinner-12
111 (10)
1.25 0.15 (10)
1.00
Ropper-v2
0.01
1,16
Valker2d-v2
0.973 0.12 (10)
0.91 2 0.038 (50)
0:25
Conclusions
This paper introduces MobILE, model-based ILFO approach that is applicable MDPs with stochastic dynamics and continuous action spaces. MobILE trades-off exploration and imitation, and this perspective is shown be important for solving the ILFO efficiently both in theory and in practice. Future works include exploring other means for learning dynamics models, performing strategic exploration and extending MobILE to problems with rich observation spaces (e.g. videos).
By not even needing the actions to imitate, ILFO algorithms allow for learning algorithms to capitalize on large amounts of video data available online. Moreover, in ILFO, the learner is successful if it learns to imitate the expert. Any expert policy designed by bad actors can naturally lead to obtaining new policies that continue to imitate and be negative influence to the society. With this perspective in mind, any expert policy must be thoroughly vetted in order to ensure ILFO algorithms including MobILE are employed in ways that benefit the society.
Acknowledgements
Rahul Kidambi acknowledges funding from NSF TRIPODS Award CCF 1740822 at Cornell University. All content represents the opinion of the authors, which is necessarily shared or endorsed by their respective employers and/or sponsors.
10

---

References
[1] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML. ACM, 2004.
[2] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 2019.
[3] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, (7(22)235-2566 2002.
[4] Yusuf Aytar. Tobias Pfaff, David Budden, Tom Le Paine. Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In NeurlPS, pages 2935-2945, 2018.
[5] Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272,
2017.
[6] Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep q-networks. In ITA, pages 1-9. IEEE, 2018.
[7] Ronen I. Brafman and Moshe Tennenholtz. R-max a general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res., 3:213-231 2001.
[8] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai Gym. arXiv preprint arxiv:1606.01556 2016.
[9] Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret analysis of stochastic and non-stochastic multi-armed bandit problems. Found. Trends Mach. Learn, 5(1):1-122, 2012.
[10] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12899 2018
[11] Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR. OpenReview.net, 2019.
[12] Sebastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient model-based reinforcement learning through optimistic policy search and planning. arXiv preprint arXiv:2006. 08684, 2020.
[13] Mare Deisenroth and Carl E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning, pages 465-472, 2011.
[14] Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, Peter Stone, and AI Sony. An imitation from observation approach transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems, 33, 2020.
[15] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https:// //gi thub. www.huu..oooooooooooooeoooeeooeo happeaaibaaseeeeeeeee 2017.
[16] Ashley D. Edwards, Himanshu Sahni, Yannick Schroecker, and Charles L. Isbell Jr. Imitating latent policies from observation. In /CML, 2019.
[17] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In ICML, 2016.
[18] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence mini- mization perspective imitation learning methods. In Conference Robot Learning, pages 1259-1277. PMLR, 2020.
[19] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration, 2016.
[20] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, abs/1 0011881..00 201.01290. 2018.
11

---

[21] Ashley Hill, Antonin Raffin, Maximilian Emesstss, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert. Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https:
//github. wwt.uuu..commiilaaaaaaaaaaaaaaaa wwwpaaaaaaaabllleeeeeeeeeeeeeeee 2018.
[22] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. abs/1606.03376, 2016.
CoRR,
[23] Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 1 11(Apr):153311600, 2010.
[24] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. CoRR, abs/1906.08225 08253,, 2019.
[25] Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Infor- mation theoretic regret bounds for online nonlinear control. arXiv preprint arXiv:2006. /2466, 2020.
[26] Sham M. Kakade. A natural policy gradient. In NIPS, pages 1531-1538, 2001.
[27] Sham M. Kakade, Michael J. Kearns, and John Langford. Exploration in metric state spaces. In ICML, 2003.
[28] Sham M. Kakade, Akshay Krishnamurthyy Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Information theoretic regret bounds for online nonlinear control. In NeurIPS, 2020.
[29] Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha Srini- vasa. Imitation learning as f-divergence minimization. arXiv preprint arXiv:1905. 12888, 2019.
[30] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning. (00233):099-233 2002.
[31] Michael Kearns and Satinder Singh. Near optimal reinforcement learning in polynomial time. Machine Learning, 49(2-3):209-233 2002.
[32] Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. CoRR, abs/2005.05951, 2020.
[33] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. ICLR. OpenReview.net, 2018.
[34] Thomas Lampe and Martin A. Riedmiller. Approximate model-assisted neural fitted q-iteration. In IJCNN, pages (698-2704. IEEE, 2014.
[35] Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.
[36] Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In ICINCO, pages 222-229, 2004.
[37] Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control. In International Conference on Learning Representations (ICLR), 2019.
[38] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees. 2018.
arXiv preprint
[39] Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In IEEE International Conference on Robotics and Automation, pages 7559-7566, 2018
[40] Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning. Proc. ICML, pages 663-670, 2000.
12

---

[41] Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep rein- forcement learning. CoRR, abs/1806.03333. 2018.
[42] Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the Eluder dimension. In Advances in Neural Information Processing Systems, pages 1476,1474, 2014.
[43] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagree- ment. In ICML, pages 5062-5071, 2019.
[44] Xue Bin Peng. Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: example- guided deep reinforcement learning of physics-banee bassec character skills. ACM Trans. Graphics, 2018.
[45] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. Technical report, CMU, 1989.
[46] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, pages 1177-1184, 2008.
[47] Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham Kakade. Towards General- ization and Simplicity in Continuous Control. In NIPS, 2017.
[48] Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. A game theoretic framework for model based reinforcement learning. ArXiv, abss/2004007844 2020.
[49] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Yee Whye Tch and D. Mike Titterington, editors, AISTATS, JMLR Proceedings, pages 661-668, 2010.
[50] Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, pages 627-635, 2011.
[51] Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In NIPS, pages 2256-2264 2013.
[52] Daniel Russo and Benjamin Van Roy, Learning to optimize via posterior sampling. Mathematics of Operations Research, 39(4): 2014.
[53] Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Re- inforcement learning with videos: Combining offline observations with interaction. CoRR, 1.06507, 2020.
[54] John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CORR, abs/1502.05477,, 2015.
[55] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. CoRR, foss/1777.0633 2017.
[56] Yuda Song, Aditi Mavalankar, Wen Sun, and Sicun Gao. Provably efficient model-based policy adaptation. In International Conference on Machine Learning, pages 9088-9098. PMLR, 2020.
[57] Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias Seeger. Gaussian pro- cess optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.39995 2009.
[58] Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-based rl in contextual decision processes: Pac bounds and exponential improvements over model-free approaches. In Conference Learning Theerr, pages 2898-2933 PMLR, 2019.
[59] Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient imitation learning from observation alone. In ICML, volume 97. PMLR, 2019.
[60] Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In ICML, volume 28, 2013.
13

---

[61] R. S. Sutton. First results with dyna, an integrated architecture for learning, planning, and reacting. In Neural Networks for Control, pages 179-189. The MIT Press: Cambridge, MA, USA, 1990.
[62] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. IEEE International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[63] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In IJCAI, pages 4950-4957, 2018.
[64] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprint arXiv: 1et.10877000555 2018.
[65] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint arXiv: XXX::/907.00057 2019.
[66] Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagree- ment. In NeurLPS, 2019.
[67] Lin F Yang and Mengdi Wang. Reinforcement leaning in feature space: Matrix bandit, kernels, and regret bound. arXiv preprint arXiv: 19Xiv:1905./035 10389, 2019.
[68] Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. In NeurIPS, 2020.
[69] Brian D Ziebart, Andrew L. Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008
14


Reviews:
[{"review_id": "xjKmNOe1vQ", "reviewer": "Reviewer_eB6p", "summary": "This paper investigates the problem of imitation learning from observation alone (ILFO). The authors leverage the idea of optimism in the face of uncertainty and propose a novel model-based framework for ILFO that can handle both discrete and continuous settings. The authors provide a regret bound of the framework and show the exponential gap between IL and ILFO and the benefit of uncertainty-based exploration. The authors further corroborate the theoretical and algorithmic results with experiments in simulation. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nAlthough the idea of optimism in the face of uncertainty is not new, its application in ILFO seems noval and inspiring. The framework that the authors develop is general enough to work under both discrete and continuous settings, and it does not require the existence of an inverse dynamics model which is required by some of the previous works. In terms of theoretical findings, the authors show an exponential gap between IL and ILFO in the regret bound with respect to the number of actions, which is interesting. In addition, the authors theoretically justify the benefits of the confidence-based exploration. Furthermore, the experimental in simulation results look encouraging. \n\nComments and questions:\n\nI appreciated that the authors are explicit about the realizability assumption on cost and transition functions. However it may be strong in practical applications. Can the current analysis be extended to lift the assumption in a straightforward way? \n\nIn Theorem 3, the rate applies to the best policy among the T polices. However, in Algorithm 1 the last policy is returned. Is it guaranteed that Algorithm 1 will always converge?\n\nHow’s the dependence on horizon H of the proposed algorithm compared to other existing ILFO algorithms including FAIL?\nIn Line 6 of Algorithm, does it mean that N expert states are required in each of the T iterations? \n\nThe paper is easy to follow in general. However, I think there’s still room for improvement. \n- I did not find where regret (mentioned in e.g. Page 2 Line 47) is formally defined or motivated in the paper before it appears implicitly late in the paper -- Theorem 3.\n- I don’t quite get the definition of Information Gain in Eq. (3). Does it assume how the dynamics model is learned? As the alg can be arbitrary, can I choose an algorithm that does not learn the dynamics so that \\sigma_t does not decrease?\n- Although Section 4.2 is one of the major contributions of the paper, it is a bit short and dense. For example, Line 254 - 256, a three-line sentence, is a bit hard to parse. Maybe to make room for it, some discussions on the different examples can be moved to the appendix. \n- In Page 3 Line 103, it seems that “query the expert” is a stronger assumption than what actually is required: access to expert state trajectories\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5 hours"}, "extra_text": {}}, {"review_id": "vWOppfCilAg", "reviewer": "Reviewer_fGHV", "summary": "This paper studies in the setting of Imitation Learning from Observations (ILFO).\nThe contribution is proposing a framework for policy learning with uncertainty estimates by dynamics model learning, which leads to better balanced exploration and imitation.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper proposes a framework for imitation learning from observations, MobLE incorporating policy learning with exploration bonus by learned dynamics model. The framework of MobILE seems reasonable for the motivation and general one, however, there seem to be some concerns on the discussion of the motivation of this study, and explanation of the method and experiments.\n\n## Pros\n* Proposing a unified framework for ILFO with a combination of policy learning, exploration bonus measured with a learned model ensemble. The combination is novel in my understanding and potentially applicable to other algorithm choices.\n* Giving a mathematical explanation for why the proposed exploration strategy is more efficient than random (or no) exploration.\n\n## Concerns\n### Discussion on the applicability of the work\nThe key idea of MobILE is dependent on exploration bonus with disagreement between learned dynamics model ensemble. I have questions about how this bonus is generalizable to other settings (e.g. input) than the ones shown in the experiments.\n\nThe experiments in the paper are performed on “state” input, and the learned policies are tested on the same robot (MDP) as the expert. For me, the assumption of no access to the actions of an expert is tricky and overly simplified in this case, since we have access to actions of the learned policy at the same time. Giving the possible situations where the assumption holds will make the paper more convincing.\n\nMoreover, in the experiments in Section 6, the bonus is measured with L2 norm of state space, however, in the practical applications, the estimate seems not straightforward. For example, learning from video experts is a promising application, since videos of performing a task are easier to obtain. But how can we measure the bonuses if we use videos as the experts without actions? There is some literature on imitation learning from video experts [1-4], and is it possible to incorporate MobiLE framework with these visual methods?\n\n### Comparison with other ILFO methods\nIn the experiments, the proposed method is compared against BC, BC-O, and GAIL. In these methods, only BC-O is ILFO method except for the proposed one. However, there are some previous methods in ILFO with different characteristics as mentioned in Section 1.1. What is the empirical difference between the proposed framework and the previous one (e.g. estimating inverse dynamics)? Adding comparison with other ILFO methods will convince the reader how exploration in the MobLE framework is promising.\n\n### Number of the models in the experiment and the formulation of uncertainty (L165)\nIn L165, the uncertainty is defined with \"sup\" of L2 norm between two different models from possible pairs in the ensemble (termed as “maximum disagreement” in L219). However, the experiments shown in the paper are performed only with two dynamics models (the uncertainty is purely measured with discrepancy of next state predictions between two models). So, there is a mismatch between the formulation and experiment. L276-277 mentions that using more than two models does not improve performance much, and I suppose these results should be included in Appendix D for justification of the formulation of taking “sup” of mismatches between pairs of models.\n\n## Additional Comments\n### Model learning for optimizing the policy\nIn the proposed algorithm, the dynamics model learning is used only for calculating the exploration bonus, which I feel is a bit costly for its computation, especially in settings with high-dimensional states. Is it possible to use them to boost policy learning as well, like [5-6]?\n\n### Minor correction\n* L298: BC-0 -> BC-O\n\n\n[1] K. Schmeckpeper, et al. Reinforcement Learning with Videos: Combining Offline Observations with Interaction. https://arxiv.org/abs/2011.06507\n\n[2] A. Edwards, et al. Imitating Latent Policies from Observation. https://arxiv.org/abs/1805.07914\n\n[3] K. Schmeckpeper, et al. Learning Predictive Models From Observation and Interaction. https://arxiv.org/abs/1912.12773\n\n[4] M. Chang, et al. Semantic Visual Navigation by Watching YouTube Videos. https://arxiv.org/abs/2006.10034\n\n[5] S. Sutton, et al. (1991). ​​Dyna, an integrated architecture for learning, planning, and reacting.\n\n[6] T. Kurutach, et al. Model-Ensemble Trust-Region Policy Optimization. \nhttps://arxiv.org/abs/1802.10592\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper tackles sample inefficiency in ILFO settings, which is a key problem in the previous studies. However, as mentioned in the main review, the paper does not discuss well on its applicability, which potentially could be the limitation of the proposed method.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "JlwP5BpvB7F", "reviewer": "Reviewer_ji99", "summary": "This paper studies a model-based approach to imitation learning from observation (ILFO). The main idea is to learn a forward model as well as ‘strategic’ exploration which is based on the concept of optimism in face of uncertainty. The proposed algorithm, MobILE, combines (i) model-based RL (forward dynamics learning), (ii) exploration based on uncertainty (maximum disagreement of model), and (iii) imitation learning. More specifically, models and the novelty-bonus function learned from online interaction with the environment can be plugged in to optimize the IPM imitation learning objective, where the bonus term incorporated into the discriminator would diminish the discriminator’s power for novel states. This algorithm features an ability to to trade off exploration and exploitation(imitation), and achieves good results on standard gym continuous control benchmarks (as good as imitation learning with action data available) despite the challenging ILFO setting.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this work is a well-written paper with solid theoretical contribution and strong empirical result. The main contribution is to present a novel combination of model-based imitation learning and exploration (uncertainty and optimism), and to show theoretical optimality bound as well as empirical validation. Given its novelty and contribution, I am slightly leaning towards recommending acceptance at this point.\n\nSignificance: The proposed approach seems to be designed with principle and makes sense. The MobILE framework has some good advantages compared to existing work such as FAIL and ILPO, such as it is compatible with continuous control and does not require much cost function tuning. Actually the practical algorithm is quite a straightforward combination of model/exploration/GAIL component, but some theoretical insights provided would be a novel and significant result within the ILFO setup. Although I was unable to examine and validate all theoretical results and proofs, the theoretical results on optimality bound seems significant and important. It is interesting to see that the difficulty of ILFO over standard IL is formalized through the exponential gap, which is also a good theoretical contribution. \n\nRegarding the exploration bonus $b$, don’t you need to have a weight (either as a hyperparameter or a learnable variable) to balance the impact of exploration? It is obvious that such exploration terms can improve the algorithm’s performance (as shown in Section 6.2), but how important are they and to what extent do they contribute to the imitation learning?\n\nRegarding experiments, one concern I have is that a direct comparison to sensible baselines (GAIfO [64]) is lacking. Although it is reported that GAIL outperfoms GAIfO in all tasks (which is very expected), it would have been much better to have a direct comparison between MobILE and GAIfO or other baseline. For example, is exploration mostly accountable for the performance improvement under the same “learning from observation” setting? Without optimism (i.e. just model-based GAIfO) would it still work better or comparably to those LfO approaches?\n\nA minor question) why does BC-O perform much better than BC in CartPole? Is it just a mistake? In L293, it is said that BC outperforms BC-O in all benchmark tasks.\n\n\nWriting/Clarity:\n- Motivation of the problem is well-written, and discussion of related work looks comprehensive and informative.\n- With its theoretical contribution in mind, the paper is overall a bit difficult to follow. As a reviewer and for readers who do not have much expertise on imitation learning and theoretical RL, I wish there were a more kind background or preliminaries section (around section 2. Setting) to review the GAIL and IPM setup for the sake of self-containedness and more clarity.\n- Naming of the problem: I think LfO is a more common, popular, established abbreviation to denote learning from observation. In that regard, why don’t authors use the term ILfO instead of ILFO? I also suggest the authors to refer GAIFO as “GAIfO” as per the original paper [64].\n- L107: It would be good to write down the optimization objective that includes $c$ (the cost function) for better clarity and how it connects to the IPM objective (L117). Given that $V^\\pi$ is the expected total cost under $c$, it’d be good to have a more formal and clear definition.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Section 7 discusses limitation of the work a little bit, but more detailed or comprehensive discussion would be also worth considering. There is concerns or issues regarding societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10+"}, "extra_text": {}}, {"review_id": "7gL03Mmmr0N", "reviewer": "Reviewer_iytC", "summary": "This paper introduces MobILE, an approach for training agents to imitate from expert states only, i.e., when actions are unavailable. In particular, the approach utilizes an exploration bonus to more efficiently explore within the environment. The approach is compared to standard imitation learning approaches within OpenAI gym tasks. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clarity: Overall, I found the paper to be clearly motivated and mostly well-written, though I found section 2.1 to be somewhat confusing.\n\nOriginality: This approach introduces using exploration with an imitation from observation approach. Adversarial methods have been used for training both Imitation Learning (IL) and Imitation from Observation (IfO). The main contribution is using an additional exploration bonus to train an IfO method to tradeoff exploration and imitation.\n\nQuality: I would have liked to see more ablations on each individual contribution of the paper. For example, the original GAIL-style training compared to the IPM method used in the paper (without bonus) and GAIL with the exploration bonus compared to MobILE. If this would not be straightforward then other exploration methods could perhaps also be considered. \n\nIn general, I am unsure if this work is a contribution to the IfO or IL community, particularly because the forward-dynamics model is learned online and hence does not seem specific to training IfO methods. It would be great if the authors could comment on how this approach is a contribution to IfO.\n\nSignificance: The reason I am leaning towards rejecting the paper is that I am unsure of the significance of the approach. It seems the main contribution is to utilize a better exploration approach to imitation from observation than random exploration. This could be useful, as exploration has not been well-studied in the imitation from observation literature. However, the work is not compared to any other exploration methods. Since the exploration bonus is learned from data captured online, other intrinsic exploration bonuses used for RL, such as curiosity [3] or RND [4] could have been evaluated. The bonus used in the paper is in fact quite similar to that used in [3] and so it would be good to at least include it in the related work section. Additionally, it is unclear how much of a contribution using ensembles for the bonus is given the paper mentions having more than two ensembles did not lead to any further gains. \n\nAnother problem is that the experiments are somewhat weak. The approach is only compared to a single IfO approach, BCO, but there have been several advances since this work was introduced. While the paper discusses other such approaches such as FAIL [5] and ILPO [6], the argument is that these approaches only work in environments with discrete actions, but MobILE also works with discrete actions and so it should be reasonable to compare in these settings.\n\nComments:\n\n- The adversarial setting (i.e. the purpose of discriminators) should be more clearly and formally explained. \n- Could the authors further discuss why a forward dynamics model is better defined than an inverse dynamics model? If the dynamics are stochastic for example then $\\Vert f(s,a) - s’ \\Vert$ $f$ will just learn to predict the expected $s’$ rather than a deterministic example. But this is also the case in $\\Vert I(s,s’) - a \\Vert$ if there is not a unique $a$. I.e. it will just compute the expected $a$ rather than a deterministic one. Practically, I do not imagine many environments where this would be much of an issue. \n- Why is BCO constant in the figures? It also has an online stage where the inverse dynamics model is being learned and getting increasingly better experiences to train the bc policy. How were the dynamics models trained? \n- I was surprised to see that the approach performs better than GAIL, since that method has ground-truth actions. Could the authors discuss this result in more detail?\n- Was it necessary to scale/bound the outputs of the exploration bonus? \n- The graphs should include tick marks on the plots (rather than only color) for better accessibility\n\n[1] Generative Adversarial Imitation Learning. Ho et al.\n\n[2] Generative Adversarial Imitation Learning from Observation. Torabi et al.\n\n[3] Curiosity-driven Exploration by Self-supervised Prediction. Pathak et al.\n\n[4] Exploration by Random Network Distillation. Burda et al.\n\n[5] Provably Efficient Imitation Learning from Observation Alone. Sun et al. \n\n[6] Imitating Latent Policies from Observation. Edwards et al.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss some potential societal impacts, particularly that it is import to ensure the quality of expert policies. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I appreciate the authors for detailed rebuttal and additional experimental results, and the reviewers for engaging in detailed feedback (especially reviewers axQZ and eB6p for active discussions). The novelty and impact were better communicated after the discussions. I like that the paper studies a generic problem of ILFO from model-based perspectives, discusses its fundamental differences from IL theoretically, and proposes better strategic exploration through models as a practical solution. One concern is some missing references/baselines for model-based GAIL/GAIFO-like algorithms, e.g. MAIL [Baram et al. 2016], but MobLIE ablation study without optimism appears to cover similar cases.
Decision: Accept (Poster)