MobILE: Model-Based Imitation Learning From Observation Alone

1 Introduction . This paper considers Imitation Learning from Observation Alone ( ILFO ) . In ILFO , the learner is presented with sequences of states encountered by the expert , without access to the actions taken by the expert , meaning approaches based on a reduction to supervised learning ( e.g. , Behavior cloning ( BC ) [ 49 ] , DAgger [ 50 ] ) are not applicable . ILFO is more general and has potential for applications where the learner and expert have different action spaces , applications like sim-to-real [ 56 , 14 ] etc . Recently , [ 59 ] reduced the ILFO problem to a sequence of one-step distribution matching problems that results in obtaining a non-stationary policy . This approach , however , is sample inefficient for longer horizon tasks since the algorithm does not effectively reuse previously collected samples when solving the current sub-problem . Another line of work considers model-based methods to infer the expert ’ s actions with either an inverse dynamics [ 63 ] or a forward dynamics [ 16 ] model ; these recovered actions are then fed into an IL approach like BC to output the final policy . These works rely on stronger assumptions that are only satisfied for Markov Decision Processes ( MDPs ) with injective transition dynamics [ 68 ] ; we return to this in the related works section . ∗Work initiated when RK was a post-doc at Cornell University ; work done outside Amazon . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . We introduce MobILE—Model-based Imitation Learning and Exploring , a model-based framework , to solve the ILFO problem . In contrast to existing model-based efforts , MobILE learns the forward transition dynamics model—a quantity that is well defined for any MDP . Importantly , MobILE combines strategic exploration with imitation by interleaving a model learning step with a bonus-based , optimistic distribution matching step – a perspective , to the best of our knowledge , that has not been considered in Imitation Learning . MobILE has the ability to automatically trade-off exploration and imitation . It simultaneously explores to collect data to refine the model and imitates the expert wherever the learned model is accurate and certain . At a high level , our theoretical results and experimental studies demonstrate that systematic exploration is beneficial for solving ILFO reliably and efficiently , and optimism is a both theoretically sound and practically effective approach for strategic exploration in ILFO ( see Figure 1 for comparisons with other ILFO algorithms ) . This paper extends the realm of partial information problems ( e.g . Reinforcement Learning and Bandits ) where optimism has been shown to be crucial in obtaining strong performance , both in theory ( e.g. , E3 [ 30 ] , UCB [ 3 ] ) and practice ( e.g. , RND [ 10 ] ) . This paper proves that incorporating optimism into the min-max IL framework [ 69 , 22 , 59 ] is beneficial for both the theoretical foundations and empirical performance of ILFO . Our Contributions : We present MobILE ( Algorithm 1 ) , a provably efficient , model-based framework for ILFO that offers competitive results in benchmark gym tasks . MobILE can be instantiated with various implementation choices owing to its modular design . This paper ’ s contributions are : 1 . The MobILE framework combines ideas of model-based learning , optimism for exploration , and adversarial imitation learning . MobILE achieves global optimality with near-optimal regret bounds for classes of MDP dynamics that satisfy certain well studied notions of complexity . The key idea of MobILE is to use optimism to trade-off imitation and exploration . 2 . We show an exponential sample complexity gap between ILFO and classic IL where one has access to expert ’ s actions . This indicates that ILFO is fundamentally harder than IL . Our lower bound on ILFO also indicates that to achieve near optimal regret , one needs to perform systematic exploration rather than random or no exploration , both of which will incur sub-optimal regret . 3 . We instantiate MobILE with a model ensemble of neural networks and a disagreement-based bonus . We present experimental results on benchmark OpenAI Gym tasks , indicating MobILE compares favorably to or outperforms existing approaches . Ablation studies indicate that optimism indeed helps in significantly improving the performance in practice . 1.1 Related Works . Imitation Learning ( IL ) is considered through the lens of two types of approaches : ( a ) behavior cloning ( BC ) [ 45 ] which casts IL as a reduction to supervised or full-information online learning [ 49 , 50 ] , or , ( b ) ( adversarial ) inverse RL [ 40 , 1 , 69 , 17 , 22 , 29 , 18 ] , which involves minimizing various distribution divergences to solve the IL problem , either with the transition dynamics known ( e.g. , [ 69 ] ) , or unknown ( e.g. , [ 22 ] ) . MobILE does not assume knowledge of the transition dynamics , is model-based , and operates without access to the expert ’ s actions . Imitation Learning from Observation Alone ( ILFO ) [ 59 ] presents a model-free approach FAIL that outputs a non-stationary policy by reducing the ILFO problem into a sequence of min-max problems , one per time-step . While being theoretically sound , this approach can not share data across different time steps and thus is not data efficient for long horizon problems . Also FAIL in theory only works for discrete actions . In contrast , our paper learns a stationary policy using model-based approaches by reusing data across all time steps and extends to continuous action space . Another line of work [ 63 , 16 , 66 ] relies on learning an estimate of expert action , often through the use of an inverse dynamics models , P e ( a|s , s′ ) . Unfortunately , an inverse dynamics model is not well defined in many benign problem instances . For instance , [ 68 , remark 1 , section 9.3 ] presents an example showing that inverse dynamics isn ’ t well defined except in the case when the MDP dynamics is injective ( i.e. , no two actions could lead to the same next state from the current state . Note that even deterministic transition dynamics doesn ’ t imply injectivity of the MDP dynamics ) . Furthermore , ILPO [ 16 ] applies to MDPs with deterministic transition dynamics and discrete actions . MobILE , on the other hand , learns the forward dynamics model which is always unique and well-defined for both deterministic and stochastic transitions and works with discrete and continuous actions . Another line of work in ILFO revolves around using hand-crafted cost functions that may rely on task-specific knowledge [ 44 , 4 , 53 ] . The performance of policy outputted by these efforts relies on the quality of the engineered cost functions . In contrast , MobILE does not require cost function engineering . Model-Based RL has seen several advances [ 61 , 36 , 13 ] including ones based on deep learning ( e.g. , [ 34 , 19 , 38 , 24 , 37 , 65 ] ) . Given MobILE ’ s modularity , these advances in model-based RL can be translated to improved algorithms for the ILFO problem . MobILE bears parallels to provably efficient model-based RL approaches including E3 [ 31 , 27 ] , R-MAX [ 7 ] , UCRL [ 23 ] , UCBVI [ 5 ] , Linear MDP [ 67 ] , LC3 [ 25 ] , Witness rank [ 58 ] which utilize optimism based approaches to trade-off exploration and exploitation . Our work utilizes optimism to trade-off exploration and imitation . 2 Setting . We consider episodic finite-horizon MDPM = { S , A , P ? , H , c , s0 } , where S , A are the state and action space , P ? : S × A 7→ S is the MDP ’ s transition kernel , H is the horizon , s0 is a fixed initial state ( note that our work generalizes when we have a distribution over initial states ) , and c is the state-dependent cost function c : S 7→ [ 0 , 1 ] . Our result can be extended to the setting where c : S × S 7→ [ 0 , 1 ] , i.e. , the ground truth cost c ( s , s′ ) depends on state and next state pairs . For analysis simplicity , we focus on c : S 7→ [ 0 , 1 ] .2 We denote dπP ∈ ∆ ( S×A ) as the average state-action distribution of policy π under the transition kernel P , i.e. , dπP ( s , a ) : = 1 H ∑H t=1 Pr ( st = s , at = a|s0 , π , P ) , where Pr ( st = s , at = a|s0 , π , P ) is the probability of reaching ( s , a ) at time step t starting from s0 by following π under transition kernel P . We abuse notation and write s ∼ dπP to denote a state s is sampled from the state-wise distribution which marginalizes action over dπP ( s , a ) , i.e. , d π P ( s ) : = 1 H ∑H t=1 Pr ( st = s|s0 , π , P ) . For a given cost function f : S 7→ [ 0 , 1 ] , V πP ; f denotes the expected total cost of π under transition P and cost function f . Similar to IL setting , in ILFO , the ground truth cost c is unknown . Instead , we can query the expert , denoted as πe : S 7→ ∆ ( A ) . Note that the expert πe could be stochastic and does not have to be the optimal policy . The expert , when queried , provides state-only demonstrations τ = { s0 , s1 . . . sH } , where st+1 ∼ P ? ( ·|st , at ) and at ∼ πe ( ·|st ) . The goal is to leverage expert ’ s state-wise demonstrations to learn a policy π that performs as well as πe in terms of optimizing the ground truth cost c , with polynomial sample complexity on problem parameters such as horizon , number of expert samples and online samples and underlying MDP ’ s complexity measures ( see section 4 for precise examples ) . We track the progress of any ( randomized ) algorithm by measuring the ( expected ) regret incurred by a policy π defined as E [ V π ] − V π∗ as a function of number of online interactions utilized by the algorithm to compute π . 2.1 Function Approximation Setup . Since the ground truth cost c is unknown , we utilize the notion of a function class ( i.e. , discriminators ) F ⊂ S 7→ [ 0 , 1 ] to define the costs that can then be utilized by a planning algorithm ( e.g . NPG [ 26 ] ) for purposes of distribution matching with expert states . If the ground truth c depends ( s , s′ ) , we use discriminators F ⊂ S × S 7→ [ 0 , 1 ] . Furthermore , we use a model class P ⊂ S × A 7→ ∆ ( S ) to capture the ground truth transition P ? . For the theoretical results in the paper , we assume realizability : Assumption 1 . Assume F and P captures ground truth cost and transition , i.e. , c ∈ F , P ? ∈ P . We will use Integral probability metric ( IPM ) with F as our divergence measure . Note that if c ∈ F and c : S 7→ [ 0 , 1 ] , then IPM defined as maxf∈F Es∼dπf ( s ) − Es∼dπe f ( s ) directly upper 2Without any additional assumptions , in ILFO , learning to optimize action-dependent cost c ( s , a ) ( or c ( s , a , s′ ) is not possible . For example , if there are two sequences of actions that generate the same sequence of states , without seeing expert ’ s preference over actions , we do not know which actions to commit to . Algorithm 1 MobILE : The framework of Model-based Imitation Learning and Exploring for ILFO 1 : Require : IPM class F , dynamics model class P , policy class Π , bonus function class B , expert dataset De ≡ { sei } Ni=1 . 2 : Initialize policy π0 ∈ Π , replay buffer D−1 = ∅ . 3 : for t = 0 , · · · , T − 1 do 4 : Execute πt in true environment P ? to get samples τt = { sk , ak } H−1k=0 ∪ sH . Append to replay buffer Dt = Dt−1 ∪ τt . 5 : Update model and bonus : P̂t+1 : S ×A → S and bt+1 : S ×A → R+ using buffer Dt . 6 : Optimistic model-based min-max IL : obtain πt+1 by solving equation ( 1 ) with P̂t+1 , bt+1 , De . 7 : end for 8 : Return πT . bounds sub-optimality gap V π − V πe , where V π is the expected total cost of π under cost function c. This justifies why minimizing IPM between two state distributions suffices [ 22 , 59 ] . Similarly , if c depends on s , s′ , we can simply minimize IPM between two state-next state distributions , i.e. , maxf Es , s′∼dπf ( s , s′ ) − Es , s′∼dπe f ( s , s′ ) where discriminators now take ( s , s′ ) as input.3 To permit generalization , we require P to have bounded complexity . For analytical simplicity , we assume F is discrete ( but exponentially large ) , and we require the sample complexity of any PAC algorithm to scale polynomially with respect to its complexity ln ( |F| ) . The ln |F| complexity can be replaced to bounded conventional complexity measures such as Rademacher complexity and covering number for continuous F ( e.g. , F being a Reproducing Kernel Hilbert Space ) .

Summary: This paper studies a model-based approach to imitation learning from observation (ILFO). The main idea is to learn a forward model as well as ‘strategic’ exploration which is based on the concept of optimism in face of uncertainty. The proposed algorithm, MobILE, combines (i) model-based RL (forward dynamics learning), (ii) exploration based on uncertainty (maximum disagreement of model), and (iii) imitation learning. More specifically, models and the novelty-bonus function learned from online interaction with the environment can be plugged in to optimize the IPM imitation learning objective, where the bonus term incorporated into the discriminator would diminish the discriminator’s power for novel states. This algorithm features an ability to to trade off exploration and exploitation(imitation), and achieves good results on standard gym continuous control benchmarks (as good as imitation learning with action data available) despite the challenging ILFO setting.