Reparameterized Variational Divergence Minimization for Stable Imitation

1 INTRODUCTION . Imitation Learning ( IL ) ( Osa et al. , 2018 ) refers to a paradigm of reinforcement learning in which the learning agent has access to an optimal , reward-maximizing expert for the underlying environment . In most work , this access is provided through a dataset of trajectories where each observed state is annotated with the action prescribed by the expert policy . This is often an extremely powerful learning paradigm in contrast to standard reinforcement learning , since not all tasks of interest admit easily-specified reward functions . Additionally , not all environments are amenable to the prolonged and potentially unsafe exploration needed for reward-maximizing agents to arrive at satisfactory policies ( Achiam et al. , 2017 ; Chow et al. , 2019 ) . While the traditional formulation of the IL problem assumes access to optimal expert action labels , the provision of such information can often be laborious ( in the case of a real , human expert ) or incur significant financial cost ( such as using elaborate instrumentation to record expert actions ) . Additionally , this restrictive assumption removes a vast number of rich , observation-only data sources from consideration ( Zhou et al. , 2018 ) . To bypass these challenges , recent work ( Liu et al. , 2018 ; Torabi et al. , 2018a ; b ; Edwards et al. , 2019 ; Sun et al. , 2019 ) has explored what is perhaps a more natural problem formulation in which an agent must recover an imitation policy from a dataset containing only expert observation sequences . While this Imitation Learning from Observations ( ILfO ) setting carries tremendous potential , such as enabling an agent to learn complex tasks from watching freely available videos on the Internet , it also is fraught with significant additional challenges . In this paper , we show how to incorporate recent advances in generative-adversarial training of deep neural networks to tackle imitation-learning problems and advance the state-of-the-art in ILfO . With these considerations in mind , the overarching goal of this work is to enable sample-efficient imitation from expert demonstrations , both with and without the provision of expert action labels . The rich literature on Generative Adversarial Networks ( Goodfellow et al. , 2014 ) has expanded in recent years to include alternative formulations of the underlying objective that yield qualitatively different solutions to the saddle-point optimization problem ( Li et al. , 2015 ; Dziugaite et al. , 2015 ; Zhao et al. , 2016 ; Nowozin et al. , 2016 ; Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) . Of notable interest are the findings of Nowozin et al . ( 2016 ) who present Variational Divergence Minimization ( VDM ) , a generalization of the generative-adversarial approach to arbitrary choices of distance measures between probability distributions drawn from the class of f -divergences ( Ali & Silvey , 1966 ; Csiszár et al. , 2004 ) . Applying VDM with varying choices of f - divergence , Nowozin et al . ( 2016 ) encounter learned synthetic distribu- tions that can exhibit differences from one another while producing equally realistic samples . Translating this idea for imitation is complicated by the fact that the optimization of the generator occurs via policy-gradient reinforcement learning ( Sutton et al. , 2000 ) . Existing work in combining adversarial IL and f -divergences ( Ke et al. , 2019 ) , despite being well-motivated , fails to account for this difference ; the end results ( shown partially in Figure 1 , where TV-VIM is the method of Ke et al . ( 2019 ) , and discussed further in later sections ) are imitation-learning algorithms that scale poorly to environments with higher-dimensional observations . In this work , we assess the effect of the VDM principle and consideration of alternative f - divergences in the contexts of IL and ILfO . We begin by reparameterizing the framework of Ke et al . ( 2019 ) for the standard IL problem . Our version transparently exposes the choices practitioners must make when designing adversarial imitation algorithms for arbitrary choices of f -divergence . We then offer a single instantiation of our framework that , in practice , allows stable training of good policies across multiple choices of f -divergence . An example is illustrated in Figure 1 where our methods ( TV-VIM-sigmoid and TV-VIMO-sigmoid ) result in significantly superior policies . We go on to extend our framework to encapsulate the ILfO setting and examine the efficacy of the resulting new algorithms across a range of continuous-control tasks in the MuJoCo ( Todorov et al. , 2012 ) domain . Our empirical results validate our framework as a viable unification of adversarial imitation methods under the VDM principle . With the assistance of recent advances in stabilizing regularization for adversarial training ( Mescheder et al. , 2018 ) , improvements in performance can be attained under an appropriate choice of f -divergence . However , there is still a significant performance gap between the recovered imitation policies and expert behavior for tasks with high dimensional observations , leaving open directions for future work in developing improved ILfO algorithms . 2 RELATED WORK . The algorithms presented in this work fall in with inverse reinforcement learning ( IRL ) ( Ng et al . ; Abbeel & Ng , 2004 ; Syed & Schapire , 2007 ; Ziebart et al. , 2008 ; Finn et al. , 2016 ; Ho & Ermon , 2016 ) approaches to IL . Early successes in this regime tend to rely on hand-engineered feature rep- resentations for success ( Abbeel & Ng , 2004 ; Ziebart et al. , 2008 ; Levine et al. , 2011 ) . Only in recent years , with the aid of deep neural networks , has there been a surge in the number of approaches that are capable of scaling to the raw , high-dimensional observations found in real-world control problems ( Finn et al. , 2016 ; Ho & Ermon , 2016 ; Duan et al. , 2017 ; Li et al. , 2017 ; Fu et al. , 2017 ; Kim & Park , 2018 ) . Our work focuses attention exclusively on adversarial methods for their widespread effectiveness across a range of imitation tasks without requiring interactive experts ( Ho & Ermon , 2016 ; Li et al. , 2017 ; Fu et al. , 2017 ; Kostrikov et al. , 2018 ) ; at the heart of these methods is the Generative Adversarial Imitation Learning ( GAIL ) ( Ho & Ermon , 2016 ) approach which produces high-fidelity imitation policies and achieves state-of-the-art results across numerous continuous-control benchmarks by leveraging the expressive power of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) for modeling complex distributions over a high-dimensional support . From an IRL perspective , GAIL can be viewed as iteratively optimizing a parameterized reward function ( discriminator ) that , when used to optimize an imitation policy ( generator ) via policy-gradient reinforcement learning ( Sutton et al. , 2000 ) , allows the agent to shift its own behavior closer to that of the expert . From the perspective of GANs , this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training a generator to fool the discriminator and induce a state-action visitation distribution similar to that of the expert . While a large body of prior work exists for IL , recent work has drawn attention to the more challenging problem of imitation learning from observation ( Sermanet et al. , 2017 ; Liu et al. , 2018 ; Goo & Niekum , 2018 ; Kimura et al. , 2018 ; Torabi et al. , 2018a ; b ; Edwards et al. , 2019 ; Sun et al. , 2019 ) . To more closely resemble observational learning in humans and leverage the wealth of publiclyavailable , observation-only data sources , the ILfO problem considers learning from expert demonstration data where no expert action labels are provided . Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies can serve as a cost signal to be minimized via reinforcement learning ( Gupta et al. , 2017 ; Sermanet et al. , 2017 ; Dwibedi et al. , 2018 ; Liu et al. , 2018 ) . In contrast , Torabi et al . ( 2018a ) introduce Behavioral Cloning from Observation ( BCO ) which leverages state-action trajectories collected under a random policy to train an inverse dynamics model for inferring the action responsible for a transition between two input states ( assuming the two represent a state and next-state pair ) . With this inverse model in hand , the observation-only demonstration data can be converted into the more traditional dataset of state-action pairs over which standard BC can be applied . Recognizing the previously discussed limitations of BC approaches , Torabi et al . ( 2018b ) introduce the natural GAIL counterpart for ILfO , Generative Adversarial Imitation from Observation ( GAIFO ) ; GAIFO is identical to GAIL except the distributions under consideration in the adversarial game are over state transitions ( state and next-state pairs ) , as opposed to stateaction pairs requiring expert action labels . While Torabi et al . ( 2018b ) offer empirical results for continuous-control tasks with low-dimensional features as well as raw image observations , GAIFO falls short of expert performance in both settings leaving an open challenge for scalable ILfO algorithms that achieve expert performance across a wide spectrum of tasks . A central question of this work is to explore how alternative formulations of the GAN objective that underlies GAIFO might yield superior ILfO algorithms . For a more in-depth survey of ILfO approaches , we refer readers to Torabi et al . ( 2019 ) . We refer readers to the Appendix for a broader overview of prior work . 3 BACKGROUND . We begin by formulating the problems of imitation learning and imitation learning from observation respectively before taking a closer look at f -divergences and connecting them to imitation learning . 3.1 IMITATION LEARNING & IMITATION FROM OBSERVATION . We operate within the Markov Decision Process ( MDP ) formalism ( Bellman , 1957 ; Puterman , 2014 ) defined as a five-tupleM = 〈S , A , R , T , γ〉 where S denotes a ( potentially infinite ) set of states , A denotes a ( potentially infinite ) set of actions , R : S × A × S → R is a reward function , T : S × A → ∆ ( S ) is a transition function , and γ ∈ [ 0 , 1 ) is a discount factor . At each timestep , the agent observes the current state of the world , st ∈ S , and randomly samples an action according to its stochastic policy π : S → ∆ ( A ) . The environment then transitions to a new state according to the transition function T and produces a reward signal according to the reward function R that is communicative of the agent ’ s progress through the overall task . Unlike , the traditional reinforcement learning paradigm , the decision-making problem presented in IL lacks a concrete reward function ; in lieu of R , a learner is provided with a dataset of expert demonstrationsD = { τ1 , τ2 , . . . τN } where each τi = ( si1 , ai1 , si2 , ai2 , . . . ) represents the sequence of states and corresponding actions taken by an expert policy , π∗ . Naturally , the goal of an IL algorithm is to synthesize a policy π using D , along with access to the MDPM , whose behavior matches that of π∗ . While the previous section outlines several possible avenues for using D to arrive at a satisfactory imitation policy , our work focuses on adversarial methods that build around GAIL ( Ho & Ermon , 2016 ) . Following from the widespread success of GANs ( Goodfellow et al. , 2014 ) , GAIL offers a highly-performant approach to IL wherein , at each iteration of the algorithm , transitions sampled from the current imitation policy are first used to update a discriminator , Dω ( s , a ) , that acts a binary classifier to distinguish between state-action pairs sampled according to the distributions induced by the expert and student . Subsequently , treating the imitation policy as a generator , policy-gradient reinforcement learning is used to shift the current policy towards expert behavior , issuing higher rewards for those generated state-action pairs that are regarded as belonging to the expert according to Dω ( s , a ) . More formally , this minimax optimization follows as min π max ω E ( s , a ) ∼ρπ∗ [ log ( Dω ( s , a ) ) ] + E ( s , a ) ∼ρπ [ log ( 1−Dω ( s , a ) ) ] ( 1 ) where ρπ ∗ ( s , a ) and ρπ ( s , a ) denote the undiscounted stationary distributions over state-action pairs for the expert and imitation policies respectively . Here Dω ( s , a ) = σ ( Vω ( s , a ) ) where Vω ( s , a ) represents the unconstrained output of a discriminator neural network with parameters ω and σ ( v ) = ( 1 + e−x ) −1 denotes the sigmoid activation function . Since the imitation policy only exerts control over the latter term in the above objective , the per-timestep reward function maximized by reinforcement learning is given as r ( s , a , s′ ) = − log ( 1 − Dω ( s , a ) ) . In practice , an entropy regularization term is often added to the objective when optimizing the imitation policy so as to avoid premature convergence to a suboptimal solution ( Mnih et al. , 2016 ; Ho & Ermon , 2016 ; Neu et al. , 2017 ) . In order to accommodate various observation-only data sources ( Zhou et al. , 2018 ) and remove the burden of requiring expert action labels , the imitation from observation setting adjusts the expert demonstration dataset D such that each trajectory τi = ( si1 , si2 , . . . ) consists only of expert observation sequences . Retaining the goal of recovering an imitation policy that closely resembles expert behavior , Torabi et al . ( 2018b ) introduce GAIFO as the natural extension of GAIL for matching the state transition distribution of the expert policy . Note that an objective for matching the stationary distribution over expert state transitions enables the provision of per-timestep feedback while simultaneously avoid the issues of temporal alignment that arise when trying to match trajectories directly . The resulting algorithm iteratively finds a solution to the following minimax optimization : min π max ω E ( s , s′ ) ∼ρπ∗ [ log ( Dω ( s , s′ ) ) ] + E ( s , s′ ) ∼ρπ [ log ( 1−Dω ( s , s′ ) ) ] ( 2 ) where ρπ ∗ ( s , s′ ) and ρπ ( s , s′ ) now denote the analogous stationary distributions over successive state pairs while Dω ( s , s′ ) = σ ( Vω ( s , s′ ) ) represents binary classifier over state pairs . Similar to GAIL , the imitation policy is optimized via policy-gradient reinforcement learning with per-timestep rewards computed according to r ( s , a , s′ ) = − log ( 1−Dω ( s , s′ ) ) and using entropy regularization as needed .

Summary: The paper proposes an IL method based on the f-divergence. Specifically, the paper extends f-VIM (Ke et al., 2019), which uses the f-divergence for IL, by using a sigmoid function for discriminator output’s activation function. This choice of activation function yields an alternative objective function, where the reward function for an RL agent does not directly depend on the convex conjugate of the function f; the paper claims that this independency improves stability of policy learning. This proposed method is named f-VIM-sigmoid. The paper extends f-VIM-sigmoid to the setting of IL with observation and proposes f-VIMO-sigmoid. Experiments on Mujoco locomotion tasks show that f-VIM-sigmoid and f-VIMO-sigmoid perform better than existing methods.