Reparameterized Variational Divergence Minimization for Stable Imitation

1 INTRODUCTION . Imitation Learning ( IL ) ( Osa et al. , 2018 ) refers to a paradigm of reinforcement learning in which the learning agent has access to an optimal , reward-maximizing expert for the underlying environment . In most work , this access is provided through a dataset of trajectories where each observed state is annotated with the action prescribed by the expert policy . This is often an extremely powerful learning paradigm in contrast to standard reinforcement learning , since not all tasks of interest admit easily-specified reward functions . Additionally , not all environments are amenable to the prolonged and potentially unsafe exploration needed for reward-maximizing agents to arrive at satisfactory policies ( Achiam et al. , 2017 ; Chow et al. , 2019 ) . While the traditional formulation of the IL problem assumes access to optimal expert action labels , the provision of such information can often be laborious ( in the case of a real , human expert ) or incur significant financial cost ( such as using elaborate instrumentation to record expert actions ) . Additionally , this restrictive assumption removes a vast number of rich , observation-only data sources from consideration ( Zhou et al. , 2018 ) . To bypass these challenges , recent work ( Liu et al. , 2018 ; Torabi et al. , 2018a ; b ; Edwards et al. , 2019 ; Sun et al. , 2019 ) has explored what is perhaps a more natural problem formulation in which an agent must recover an imitation policy from a dataset containing only expert observation sequences . While this Imitation Learning from Observations ( ILfO ) setting carries tremendous potential , such as enabling an agent to learn complex tasks from watching freely available videos on the Internet , it also is fraught with significant additional challenges . In this paper , we show how to incorporate recent advances in generative-adversarial training of deep neural networks to tackle imitation-learning problems and advance the state-of-the-art in ILfO . With these considerations in mind , the overarching goal of this work is to enable sample-efficient imitation from expert demonstrations , both with and without the provision of expert action labels . The rich literature on Generative Adversarial Networks ( Goodfellow et al. , 2014 ) has expanded in recent years to include alternative formulations of the underlying objective that yield qualitatively different solutions to the saddle-point optimization problem ( Li et al. , 2015 ; Dziugaite et al. , 2015 ; Zhao et al. , 2016 ; Nowozin et al. , 2016 ; Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) . Of notable interest are the findings of Nowozin et al . ( 2016 ) who present Variational Divergence Minimization ( VDM ) , a generalization of the generative-adversarial approach to arbitrary choices of distance measures between probability distributions drawn from the class of f -divergences ( Ali & Silvey , 1966 ; Csiszár et al. , 2004 ) . Applying VDM with varying choices of f - divergence , Nowozin et al . ( 2016 ) encounter learned synthetic distribu- tions that can exhibit differences from one another while producing equally realistic samples . Translating this idea for imitation is complicated by the fact that the optimization of the generator occurs via policy-gradient reinforcement learning ( Sutton et al. , 2000 ) . Existing work in combining adversarial IL and f -divergences ( Ke et al. , 2019 ) , despite being well-motivated , fails to account for this difference ; the end results ( shown partially in Figure 1 , where TV-VIM is the method of Ke et al . ( 2019 ) , and discussed further in later sections ) are imitation-learning algorithms that scale poorly to environments with higher-dimensional observations . In this work , we assess the effect of the VDM principle and consideration of alternative f - divergences in the contexts of IL and ILfO . We begin by reparameterizing the framework of Ke et al . ( 2019 ) for the standard IL problem . Our version transparently exposes the choices practitioners must make when designing adversarial imitation algorithms for arbitrary choices of f -divergence . We then offer a single instantiation of our framework that , in practice , allows stable training of good policies across multiple choices of f -divergence . An example is illustrated in Figure 1 where our methods ( TV-VIM-sigmoid and TV-VIMO-sigmoid ) result in significantly superior policies . We go on to extend our framework to encapsulate the ILfO setting and examine the efficacy of the resulting new algorithms across a range of continuous-control tasks in the MuJoCo ( Todorov et al. , 2012 ) domain . Our empirical results validate our framework as a viable unification of adversarial imitation methods under the VDM principle . With the assistance of recent advances in stabilizing regularization for adversarial training ( Mescheder et al. , 2018 ) , improvements in performance can be attained under an appropriate choice of f -divergence . However , there is still a significant performance gap between the recovered imitation policies and expert behavior for tasks with high dimensional observations , leaving open directions for future work in developing improved ILfO algorithms . 2 RELATED WORK . The algorithms presented in this work fall in with inverse reinforcement learning ( IRL ) ( Ng et al . ; Abbeel & Ng , 2004 ; Syed & Schapire , 2007 ; Ziebart et al. , 2008 ; Finn et al. , 2016 ; Ho & Ermon , 2016 ) approaches to IL . Early successes in this regime tend to rely on hand-engineered feature rep- resentations for success ( Abbeel & Ng , 2004 ; Ziebart et al. , 2008 ; Levine et al. , 2011 ) . Only in recent years , with the aid of deep neural networks , has there been a surge in the number of approaches that are capable of scaling to the raw , high-dimensional observations found in real-world control problems ( Finn et al. , 2016 ; Ho & Ermon , 2016 ; Duan et al. , 2017 ; Li et al. , 2017 ; Fu et al. , 2017 ; Kim & Park , 2018 ) . Our work focuses attention exclusively on adversarial methods for their widespread effectiveness across a range of imitation tasks without requiring interactive experts ( Ho & Ermon , 2016 ; Li et al. , 2017 ; Fu et al. , 2017 ; Kostrikov et al. , 2018 ) ; at the heart of these methods is the Generative Adversarial Imitation Learning ( GAIL ) ( Ho & Ermon , 2016 ) approach which produces high-fidelity imitation policies and achieves state-of-the-art results across numerous continuous-control benchmarks by leveraging the expressive power of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) for modeling complex distributions over a high-dimensional support . From an IRL perspective , GAIL can be viewed as iteratively optimizing a parameterized reward function ( discriminator ) that , when used to optimize an imitation policy ( generator ) via policy-gradient reinforcement learning ( Sutton et al. , 2000 ) , allows the agent to shift its own behavior closer to that of the expert . From the perspective of GANs , this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training a generator to fool the discriminator and induce a state-action visitation distribution similar to that of the expert . While a large body of prior work exists for IL , recent work has drawn attention to the more challenging problem of imitation learning from observation ( Sermanet et al. , 2017 ; Liu et al. , 2018 ; Goo & Niekum , 2018 ; Kimura et al. , 2018 ; Torabi et al. , 2018a ; b ; Edwards et al. , 2019 ; Sun et al. , 2019 ) . To more closely resemble observational learning in humans and leverage the wealth of publiclyavailable , observation-only data sources , the ILfO problem considers learning from expert demonstration data where no expert action labels are provided . Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies can serve as a cost signal to be minimized via reinforcement learning ( Gupta et al. , 2017 ; Sermanet et al. , 2017 ; Dwibedi et al. , 2018 ; Liu et al. , 2018 ) . In contrast , Torabi et al . ( 2018a ) introduce Behavioral Cloning from Observation ( BCO ) which leverages state-action trajectories collected under a random policy to train an inverse dynamics model for inferring the action responsible for a transition between two input states ( assuming the two represent a state and next-state pair ) . With this inverse model in hand , the observation-only demonstration data can be converted into the more traditional dataset of state-action pairs over which standard BC can be applied . Recognizing the previously discussed limitations of BC approaches , Torabi et al . ( 2018b ) introduce the natural GAIL counterpart for ILfO , Generative Adversarial Imitation from Observation ( GAIFO ) ; GAIFO is identical to GAIL except the distributions under consideration in the adversarial game are over state transitions ( state and next-state pairs ) , as opposed to stateaction pairs requiring expert action labels . While Torabi et al . ( 2018b ) offer empirical results for continuous-control tasks with low-dimensional features as well as raw image observations , GAIFO falls short of expert performance in both settings leaving an open challenge for scalable ILfO algorithms that achieve expert performance across a wide spectrum of tasks . A central question of this work is to explore how alternative formulations of the GAN objective that underlies GAIFO might yield superior ILfO algorithms . For a more in-depth survey of ILfO approaches , we refer readers to Torabi et al . ( 2019 ) . We refer readers to the Appendix for a broader overview of prior work . 3 BACKGROUND . We begin by formulating the problems of imitation learning and imitation learning from observation respectively before taking a closer look at f -divergences and connecting them to imitation learning . 3.1 IMITATION LEARNING & IMITATION FROM OBSERVATION . We operate within the Markov Decision Process ( MDP ) formalism ( Bellman , 1957 ; Puterman , 2014 ) defined as a five-tupleM = 〈S , A , R , T , γ〉 where S denotes a ( potentially infinite ) set of states , A denotes a ( potentially infinite ) set of actions , R : S × A × S → R is a reward function , T : S × A → ∆ ( S ) is a transition function , and γ ∈ [ 0 , 1 ) is a discount factor . At each timestep , the agent observes the current state of the world , st ∈ S , and randomly samples an action according to its stochastic policy π : S → ∆ ( A ) . The environment then transitions to a new state according to the transition function T and produces a reward signal according to the reward function R that is communicative of the agent ’ s progress through the overall task . Unlike , the traditional reinforcement learning paradigm , the decision-making problem presented in IL lacks a concrete reward function ; in lieu of R , a learner is provided with a dataset of expert demonstrationsD = { τ1 , τ2 , . . . τN } where each τi = ( si1 , ai1 , si2 , ai2 , . . . ) represents the sequence of states and corresponding actions taken by an expert policy , π∗ . Naturally , the goal of an IL algorithm is to synthesize a policy π using D , along with access to the MDPM , whose behavior matches that of π∗ . While the previous section outlines several possible avenues for using D to arrive at a satisfactory imitation policy , our work focuses on adversarial methods that build around GAIL ( Ho & Ermon , 2016 ) . Following from the widespread success of GANs ( Goodfellow et al. , 2014 ) , GAIL offers a highly-performant approach to IL wherein , at each iteration of the algorithm , transitions sampled from the current imitation policy are first used to update a discriminator , Dω ( s , a ) , that acts a binary classifier to distinguish between state-action pairs sampled according to the distributions induced by the expert and student . Subsequently , treating the imitation policy as a generator , policy-gradient reinforcement learning is used to shift the current policy towards expert behavior , issuing higher rewards for those generated state-action pairs that are regarded as belonging to the expert according to Dω ( s , a ) . More formally , this minimax optimization follows as min π max ω E ( s , a ) ∼ρπ∗ [ log ( Dω ( s , a ) ) ] + E ( s , a ) ∼ρπ [ log ( 1−Dω ( s , a ) ) ] ( 1 ) where ρπ ∗ ( s , a ) and ρπ ( s , a ) denote the undiscounted stationary distributions over state-action pairs for the expert and imitation policies respectively . Here Dω ( s , a ) = σ ( Vω ( s , a ) ) where Vω ( s , a ) represents the unconstrained output of a discriminator neural network with parameters ω and σ ( v ) = ( 1 + e−x ) −1 denotes the sigmoid activation function . Since the imitation policy only exerts control over the latter term in the above objective , the per-timestep reward function maximized by reinforcement learning is given as r ( s , a , s′ ) = − log ( 1 − Dω ( s , a ) ) . In practice , an entropy regularization term is often added to the objective when optimizing the imitation policy so as to avoid premature convergence to a suboptimal solution ( Mnih et al. , 2016 ; Ho & Ermon , 2016 ; Neu et al. , 2017 ) . In order to accommodate various observation-only data sources ( Zhou et al. , 2018 ) and remove the burden of requiring expert action labels , the imitation from observation setting adjusts the expert demonstration dataset D such that each trajectory τi = ( si1 , si2 , . . . ) consists only of expert observation sequences . Retaining the goal of recovering an imitation policy that closely resembles expert behavior , Torabi et al . ( 2018b ) introduce GAIFO as the natural extension of GAIL for matching the state transition distribution of the expert policy . Note that an objective for matching the stationary distribution over expert state transitions enables the provision of per-timestep feedback while simultaneously avoid the issues of temporal alignment that arise when trying to match trajectories directly . The resulting algorithm iteratively finds a solution to the following minimax optimization : min π max ω E ( s , s′ ) ∼ρπ∗ [ log ( Dω ( s , s′ ) ) ] + E ( s , s′ ) ∼ρπ [ log ( 1−Dω ( s , s′ ) ) ] ( 2 ) where ρπ ∗ ( s , s′ ) and ρπ ( s , s′ ) now denote the analogous stationary distributions over successive state pairs while Dω ( s , s′ ) = σ ( Vω ( s , s′ ) ) represents binary classifier over state pairs . Similar to GAIL , the imitation policy is optimized via policy-gradient reinforcement learning with per-timestep rewards computed according to r ( s , a , s′ ) = − log ( 1−Dω ( s , s′ ) ) and using entropy regularization as needed .

Summary: This paper proposes the application of the f-VIM framework (Ke et. al., 2019) to the problem of imitation learning from observations (no expert actions). The authors first identify a potential source of numerical instability in the application of f-VIM to imitation learning – the rewards for the policy-gradient RL are given by a combination of a convex conjugate and an activation function. To alleviate this, f-VIM is reparameterized by curating the activation using conjugate inverse (Equation 8), yielding a potentially more stable reward for deep-RL.