MobILE: Model-Based Imitation Learning From Observation Alone

1 Introduction . This paper considers Imitation Learning from Observation Alone ( ILFO ) . In ILFO , the learner is presented with sequences of states encountered by the expert , without access to the actions taken by the expert , meaning approaches based on a reduction to supervised learning ( e.g. , Behavior cloning ( BC ) [ 49 ] , DAgger [ 50 ] ) are not applicable . ILFO is more general and has potential for applications where the learner and expert have different action spaces , applications like sim-to-real [ 56 , 14 ] etc . Recently , [ 59 ] reduced the ILFO problem to a sequence of one-step distribution matching problems that results in obtaining a non-stationary policy . This approach , however , is sample inefficient for longer horizon tasks since the algorithm does not effectively reuse previously collected samples when solving the current sub-problem . Another line of work considers model-based methods to infer the expert ’ s actions with either an inverse dynamics [ 63 ] or a forward dynamics [ 16 ] model ; these recovered actions are then fed into an IL approach like BC to output the final policy . These works rely on stronger assumptions that are only satisfied for Markov Decision Processes ( MDPs ) with injective transition dynamics [ 68 ] ; we return to this in the related works section . ∗Work initiated when RK was a post-doc at Cornell University ; work done outside Amazon . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . We introduce MobILE—Model-based Imitation Learning and Exploring , a model-based framework , to solve the ILFO problem . In contrast to existing model-based efforts , MobILE learns the forward transition dynamics model—a quantity that is well defined for any MDP . Importantly , MobILE combines strategic exploration with imitation by interleaving a model learning step with a bonus-based , optimistic distribution matching step – a perspective , to the best of our knowledge , that has not been considered in Imitation Learning . MobILE has the ability to automatically trade-off exploration and imitation . It simultaneously explores to collect data to refine the model and imitates the expert wherever the learned model is accurate and certain . At a high level , our theoretical results and experimental studies demonstrate that systematic exploration is beneficial for solving ILFO reliably and efficiently , and optimism is a both theoretically sound and practically effective approach for strategic exploration in ILFO ( see Figure 1 for comparisons with other ILFO algorithms ) . This paper extends the realm of partial information problems ( e.g . Reinforcement Learning and Bandits ) where optimism has been shown to be crucial in obtaining strong performance , both in theory ( e.g. , E3 [ 30 ] , UCB [ 3 ] ) and practice ( e.g. , RND [ 10 ] ) . This paper proves that incorporating optimism into the min-max IL framework [ 69 , 22 , 59 ] is beneficial for both the theoretical foundations and empirical performance of ILFO . Our Contributions : We present MobILE ( Algorithm 1 ) , a provably efficient , model-based framework for ILFO that offers competitive results in benchmark gym tasks . MobILE can be instantiated with various implementation choices owing to its modular design . This paper ’ s contributions are : 1 . The MobILE framework combines ideas of model-based learning , optimism for exploration , and adversarial imitation learning . MobILE achieves global optimality with near-optimal regret bounds for classes of MDP dynamics that satisfy certain well studied notions of complexity . The key idea of MobILE is to use optimism to trade-off imitation and exploration . 2 . We show an exponential sample complexity gap between ILFO and classic IL where one has access to expert ’ s actions . This indicates that ILFO is fundamentally harder than IL . Our lower bound on ILFO also indicates that to achieve near optimal regret , one needs to perform systematic exploration rather than random or no exploration , both of which will incur sub-optimal regret . 3 . We instantiate MobILE with a model ensemble of neural networks and a disagreement-based bonus . We present experimental results on benchmark OpenAI Gym tasks , indicating MobILE compares favorably to or outperforms existing approaches . Ablation studies indicate that optimism indeed helps in significantly improving the performance in practice . 1.1 Related Works . Imitation Learning ( IL ) is considered through the lens of two types of approaches : ( a ) behavior cloning ( BC ) [ 45 ] which casts IL as a reduction to supervised or full-information online learning [ 49 , 50 ] , or , ( b ) ( adversarial ) inverse RL [ 40 , 1 , 69 , 17 , 22 , 29 , 18 ] , which involves minimizing various distribution divergences to solve the IL problem , either with the transition dynamics known ( e.g. , [ 69 ] ) , or unknown ( e.g. , [ 22 ] ) . MobILE does not assume knowledge of the transition dynamics , is model-based , and operates without access to the expert ’ s actions . Imitation Learning from Observation Alone ( ILFO ) [ 59 ] presents a model-free approach FAIL that outputs a non-stationary policy by reducing the ILFO problem into a sequence of min-max problems , one per time-step . While being theoretically sound , this approach can not share data across different time steps and thus is not data efficient for long horizon problems . Also FAIL in theory only works for discrete actions . In contrast , our paper learns a stationary policy using model-based approaches by reusing data across all time steps and extends to continuous action space . Another line of work [ 63 , 16 , 66 ] relies on learning an estimate of expert action , often through the use of an inverse dynamics models , P e ( a|s , s′ ) . Unfortunately , an inverse dynamics model is not well defined in many benign problem instances . For instance , [ 68 , remark 1 , section 9.3 ] presents an example showing that inverse dynamics isn ’ t well defined except in the case when the MDP dynamics is injective ( i.e. , no two actions could lead to the same next state from the current state . Note that even deterministic transition dynamics doesn ’ t imply injectivity of the MDP dynamics ) . Furthermore , ILPO [ 16 ] applies to MDPs with deterministic transition dynamics and discrete actions . MobILE , on the other hand , learns the forward dynamics model which is always unique and well-defined for both deterministic and stochastic transitions and works with discrete and continuous actions . Another line of work in ILFO revolves around using hand-crafted cost functions that may rely on task-specific knowledge [ 44 , 4 , 53 ] . The performance of policy outputted by these efforts relies on the quality of the engineered cost functions . In contrast , MobILE does not require cost function engineering . Model-Based RL has seen several advances [ 61 , 36 , 13 ] including ones based on deep learning ( e.g. , [ 34 , 19 , 38 , 24 , 37 , 65 ] ) . Given MobILE ’ s modularity , these advances in model-based RL can be translated to improved algorithms for the ILFO problem . MobILE bears parallels to provably efficient model-based RL approaches including E3 [ 31 , 27 ] , R-MAX [ 7 ] , UCRL [ 23 ] , UCBVI [ 5 ] , Linear MDP [ 67 ] , LC3 [ 25 ] , Witness rank [ 58 ] which utilize optimism based approaches to trade-off exploration and exploitation . Our work utilizes optimism to trade-off exploration and imitation . 2 Setting . We consider episodic finite-horizon MDPM = { S , A , P ? , H , c , s0 } , where S , A are the state and action space , P ? : S × A 7→ S is the MDP ’ s transition kernel , H is the horizon , s0 is a fixed initial state ( note that our work generalizes when we have a distribution over initial states ) , and c is the state-dependent cost function c : S 7→ [ 0 , 1 ] . Our result can be extended to the setting where c : S × S 7→ [ 0 , 1 ] , i.e. , the ground truth cost c ( s , s′ ) depends on state and next state pairs . For analysis simplicity , we focus on c : S 7→ [ 0 , 1 ] .2 We denote dπP ∈ ∆ ( S×A ) as the average state-action distribution of policy π under the transition kernel P , i.e. , dπP ( s , a ) : = 1 H ∑H t=1 Pr ( st = s , at = a|s0 , π , P ) , where Pr ( st = s , at = a|s0 , π , P ) is the probability of reaching ( s , a ) at time step t starting from s0 by following π under transition kernel P . We abuse notation and write s ∼ dπP to denote a state s is sampled from the state-wise distribution which marginalizes action over dπP ( s , a ) , i.e. , d π P ( s ) : = 1 H ∑H t=1 Pr ( st = s|s0 , π , P ) . For a given cost function f : S 7→ [ 0 , 1 ] , V πP ; f denotes the expected total cost of π under transition P and cost function f . Similar to IL setting , in ILFO , the ground truth cost c is unknown . Instead , we can query the expert , denoted as πe : S 7→ ∆ ( A ) . Note that the expert πe could be stochastic and does not have to be the optimal policy . The expert , when queried , provides state-only demonstrations τ = { s0 , s1 . . . sH } , where st+1 ∼ P ? ( ·|st , at ) and at ∼ πe ( ·|st ) . The goal is to leverage expert ’ s state-wise demonstrations to learn a policy π that performs as well as πe in terms of optimizing the ground truth cost c , with polynomial sample complexity on problem parameters such as horizon , number of expert samples and online samples and underlying MDP ’ s complexity measures ( see section 4 for precise examples ) . We track the progress of any ( randomized ) algorithm by measuring the ( expected ) regret incurred by a policy π defined as E [ V π ] − V π∗ as a function of number of online interactions utilized by the algorithm to compute π . 2.1 Function Approximation Setup . Since the ground truth cost c is unknown , we utilize the notion of a function class ( i.e. , discriminators ) F ⊂ S 7→ [ 0 , 1 ] to define the costs that can then be utilized by a planning algorithm ( e.g . NPG [ 26 ] ) for purposes of distribution matching with expert states . If the ground truth c depends ( s , s′ ) , we use discriminators F ⊂ S × S 7→ [ 0 , 1 ] . Furthermore , we use a model class P ⊂ S × A 7→ ∆ ( S ) to capture the ground truth transition P ? . For the theoretical results in the paper , we assume realizability : Assumption 1 . Assume F and P captures ground truth cost and transition , i.e. , c ∈ F , P ? ∈ P . We will use Integral probability metric ( IPM ) with F as our divergence measure . Note that if c ∈ F and c : S 7→ [ 0 , 1 ] , then IPM defined as maxf∈F Es∼dπf ( s ) − Es∼dπe f ( s ) directly upper 2Without any additional assumptions , in ILFO , learning to optimize action-dependent cost c ( s , a ) ( or c ( s , a , s′ ) is not possible . For example , if there are two sequences of actions that generate the same sequence of states , without seeing expert ’ s preference over actions , we do not know which actions to commit to . Algorithm 1 MobILE : The framework of Model-based Imitation Learning and Exploring for ILFO 1 : Require : IPM class F , dynamics model class P , policy class Π , bonus function class B , expert dataset De ≡ { sei } Ni=1 . 2 : Initialize policy π0 ∈ Π , replay buffer D−1 = ∅ . 3 : for t = 0 , · · · , T − 1 do 4 : Execute πt in true environment P ? to get samples τt = { sk , ak } H−1k=0 ∪ sH . Append to replay buffer Dt = Dt−1 ∪ τt . 5 : Update model and bonus : P̂t+1 : S ×A → S and bt+1 : S ×A → R+ using buffer Dt . 6 : Optimistic model-based min-max IL : obtain πt+1 by solving equation ( 1 ) with P̂t+1 , bt+1 , De . 7 : end for 8 : Return πT . bounds sub-optimality gap V π − V πe , where V π is the expected total cost of π under cost function c. This justifies why minimizing IPM between two state distributions suffices [ 22 , 59 ] . Similarly , if c depends on s , s′ , we can simply minimize IPM between two state-next state distributions , i.e. , maxf Es , s′∼dπf ( s , s′ ) − Es , s′∼dπe f ( s , s′ ) where discriminators now take ( s , s′ ) as input.3 To permit generalization , we require P to have bounded complexity . For analytical simplicity , we assume F is discrete ( but exponentially large ) , and we require the sample complexity of any PAC algorithm to scale polynomially with respect to its complexity ln ( |F| ) . The ln |F| complexity can be replaced to bounded conventional complexity measures such as Rademacher complexity and covering number for continuous F ( e.g. , F being a Reproducing Kernel Hilbert Space ) .

Summary: This paper investigates the problem of imitation learning from observation alone (ILFO). The authors leverage the idea of optimism in the face of uncertainty and propose a novel model-based framework for ILFO that can handle both discrete and continuous settings. The authors provide a regret bound of the framework and show the exponential gap between IL and ILFO and the benefit of uncertainty-based exploration. The authors further corroborate the theoretical and algorithmic results with experiments in simulation.