Unbiased Learning with State-Conditioned Rewards in Adversarial Imitation Learning

1 INTRODUCTION . Inverse reinforcement learning ( IRL ) is an algorithm of recovering the ground truth reward function from observed behavior ( Ng & Russell , 2000 ) . IRL algorithms—followed by appropriate reinforcement learning ( RL ) algorithms—can optimize policy through farsighted cumulative value measures in the given system ( Sutton & Barto , 2018 ) ; hence it can usually achieve more satisfying results than mere supervision . While a few studies have investigated recovering reward functions to continuous spaces ( Babes et al. , 2011 ; Levine & Koltun , 2012 ) , IRL algorithms often fail to find the ground-truth reward function in high-dimensional complex domains ( Finn et al. , 2016b ) . The notion of the ground-truth reward requires elaboration since IRL is an ill-posed problem ; there can be numerous solutions to the reward function inducing the same optimal policy ( Ng et al. , 1999 ; Ng & Russell , 2000 ) . Recently , adversarial imitation learning ( AIL ) as a reward acquisition method has shown promising results ( Ho & Ermon , 2016 ) . One of the distinctive strengths of AIL is the scalability through parameterized non-linear functions such as neural networks . The maximum causal entropy principles are widely regarded as the solution when the optimal control problem is modeled as probabilistic inference ( Ziebart et al. , 2010 ; Haarnoja et al. , 2017 ) . In particular , probabilistic modeling using a continuous energy function forms a representation called an energy-based model ( EBM ) . We highlight the following advantages of the energy-based IRL : • It provides a unified framework for stochastic policies to the learning ; most probabilistic models can be viewed as special types of EBMs ( LeCun et al. , 2006 ) . • It rationalizes the stochasticity of behavior ; this provides robustness in the face of uncertain dy- namics ( Ziebart et al. , 2010 ) and a natural way of modeling complex multi-modal distribution . AIL reward functions seem to be exceptions to these arguments—the AIL framework produces distinct types of rewards that are ever-changing and are intended for discriminating joint densities . We argue that these characteristics hinder proper information projection to the optimal decision . This work points out that there remain two kinds of biases in AIL . The established AIL algorithms are typically formalized by the cumulative densities called occupancy measure . We claim that the accumulated measures contain biases that are not related to modeling purposeful behavior , and the formulation is vulnerable to distributional shifts of an MDP . Empirically , they work as dominant noises in training because of the formulation ’ s innate high variance . The other bias is implicit survival or early termination bias caused by reward formulation , which lacks consideration for the terminal states in finite episodes . These unnormalized rewards often provokes sub-optimal behaviors where the agent learns to maliciously make use of temporal-aware strategies . This paper proposes an adversarial IRL method called causal adversarial inverse reinforcement learning ( CAIRL ) . We primarily associate the reward acquisition method with approaches for energy-based RL and IRL algorithms ; the CAIRL reward function can induce complex probabilistic behaviors with multiple modalities . We then show that learning with a dual discriminator architecture provides stepwise , state-conditioned rewards . For handling biases induced by finite-horizon , the model postulates the reward function satisfies a Bellman equation , including “ self-looping ” terminal states . As a result , it learns the reward function satisfying the property of EBMs . Noteworthy contributions of this work are 1 ) a model-free , energy-based IRL algorithm that is effective in high-dimensional environments , 2 ) a dual discriminator architecture for recovering a robust state-conditioned reward function , 3 ) an effective approach for handling terminal states , and 4 ) meaningful experiments and comparison studies with state-of-the-art algorithms in various topics . 2 RELATED WORKS . Imitation learning is a fundamental approach for modeling intellectual behavior from an expert at specific tasks ( Pomerleau , 1991 ; Zhang et al. , 2018 ) . For the standard framework called Behavioral Cloning , learning from demonstrations is treated as supervised learning for a trajectory dataset . On the other hand , IRL aims to study the reward function of the underlying system , which characterizes the expert . In this perspective , training a policy with an IRL reward function is a branch of imitation learning , specialized in dealing with sequential decision-making problems by recovering the concise representation of a task ( Ng & Russell , 2000 ; Abbeel & Ng , 2004 ) . For modeling stochastic expert policies , Boltzmann distributions appeared in early IRL research , such as Bayesian IRL , natural gradient IRL , and maximum likelihood IRL ( Ramachandran & Amir , 2007 ; Neu & Szepesvári , 2012 ; Babes et al. , 2011 ) . Notably , maximum entropy IRL ( Ziebart et al. , 2008 ) is explicitly formulated based on the principle of maximum entropy . The framework has also been derived from causal entropy—the derived algorithm can model the purposeful distribution of optimal policy into a reward function ( Ziebart et al. , 2010 ) . Our work draws significant inspirations from these prior works and aims to redeem the perspective of probabilistic causality . Recently , AIL methods ( Ho & Ermon , 2016 ; Fu et al. , 2017 ; Ghasemipour et al. , 2020 ) have shown great success on continuous control benchmarks . Each of the them provides a unique divergence minimization scheme by its architecture . In particular , our work shares major components with AIRL . It has been argued that the algorithm does not recover the energy of the expert policy ( Liu et al. , 2020 ) . We stress that our work introduces essential concepts to draw an energy-based representation of the expert policy correctly . The discriminator design is based on the rich energy-based interpretation of GANs ( Zhao et al. , 2016 ; Azadi et al. , 2018 ; Che et al. , 2020 ) and numerous studies with multiple discriminators ( Chongxuan et al. , 2017 ; Gan et al. , 2017 ; Choi et al. , 2018 ) . The issues of finite-horizon tasks were initially raised in RL during the discussion of time limits in MDP benchmarks ( Pardo et al. , 2017 ; Tucker et al. , 2018 ) . It turned out that the time limits , or even the existence of terminal states , would significantly affect the value learning procedure of RL compared to that generated in infinite horizon MDPs . IRL suffers from the identical problem that reward learning of finite episodes is not really stable for tasks outside of appropriate benchmarks . Kostrikov et al . ( 2018 ) suggested explicitly adding a self-repeating absorbing state ( Sutton & Barto , 2018 ) after the terminal state ; consequently , AIL discriminators can evaluate the termination frequencies . 3 BACKGROUND . Markov Decision Process ( MDP ) . We define an MDP as a tuple M = ( S , A , P , r , p0 , γ ) where S and A denote the state and action spaces , and γ is the discount factor . The transition distribution P , the deterministic state-action reward function r , and the initial state distribution p0 are unknown . Let τπ and τE be sequences of finite states and actions ( s0 , a0 , . . . , aT−1 , sT ) obtained by a policy π and the expert policy πE , respectively . The term ρπ denotes the occupancy measures derived by Table 1 : The objectives for AIL algorithms in a form as the minimization of statistical divergences . Method Optimized Objective ( Minimization ) Behavioral Cloning EπE [ DKL ( πE ( a|s ) ‖π ( a|s ) ) ] = −EπE [ log π ( a|s ) ] + const GAIL ( Ho & Ermon , 2016 ) Eπ [ DJS ( ρπ ( s , a ) , ρE ( s , a ) ) −H ( π ( ·|s ) ) ] AIRL ( Fu et al. , 2017 ) Eπ [ DKL ( ρπ ( s , a ) ∥∥ρE ( s , a ) ) ] = −Eπ [ log ρE ( s , a ) +H ( ρπ ) ] FAIRL ( Ghasemipour et al. , 2020 ) Eπ [ DKL ( ρE ( s , a ) ∥∥ρπ ( s , a ) ) ] = −EπE [ log ρπ ( s , a ) +H ( ρE ) ] CAIRL ( this work ) Eπ [ DKL ( π ( a|s ) ∥∥πE ( a|s ) ) ] = −Eπ [ r ( s , a ) +H ( π ( ·|s ) ) ] + const π , and is defined as ρπ ( s , a ) = π ( a|s ) ∑∞ t=0 γ t Pr ( st = s|π ) . With a slight abuse of notation , we refer to the occupancy measures of states as ρE ( s ) and ρπ ( s ) . The expectation of π for an arbitrary function c denotes an expected return for infinite-horizon : Eπ [ c ( s , a ) ] , E [ ∑∞ t=0 γ tc ( s , a ) |π ] . Maximum Entropy IRL ( MaxEnt IRL ) . Ziebart ( 2010 ) and Haarnoja et al . ( 2017 ) defined the optimality of stochastic policy with an entropy-regularized RL objective as follows : π ? = arg max π∈Π ∑ t E ( st , at ) ∼ρπ [ r ( st , at ) + αH ( at|st ) ] whereH denotes the causal entropy function.1 If πE is the MaxEnt RL policy , the softmax Bellman optimality equations can be defined by the following recursive logic : Q ? ( st , at ) = r ( st , at ) + γEst+1∼P ( ·|st , at ) [ V ? ( st+1 ) ] V ? ( st ) = Eat∼πE ( ·|st ) [ Q ? ( st , at ) − log πE ( at|st ) ] ( 1 ) MaxEnt IRL algorithms ( Ziebart et al. , 2008 ; 2010 ) are energy-based interpretations of IRL which aim to find behavior abiding the MaxEnt principle . Such algorithms , however , are difficult to be computed when the given spaces are continuous or dynamics are unknown ( Finn et al. , 2016a ) . Adversarial Imitation Learning . Ho & Ermon ( 2016 ) considered adversarial learning as a modelfree , sampling-based approximation to MaxEnt IRL . Instead of exhaustively solving the problem , GAIL performs imitation learning by minimizing the divergence between the state-action occupancy measures from expert and learner through the following logistic objective : min π∈Π max D EπE [ logD ( s , a ) ] + Eπ [ log ( 1−D ( s , a ) ) ] −H ( π ) ( 2 ) where D ∈ ( 0 , 1 ) |S||A| indicates a binary classifier trained to distinguish between τπ and τE . The AIRL discriminator tries to disentangle a reward function that is invariant to dynamics . It takes a particular form : Dθ ( s , a ) = exp ( fθ , ψ ( s , a ) ) / ( exp ( fθ , ψ ( s , a ) ) + πφ ( a|s ) ) . Learning with the AIRL can be considered as the reverse KL divergence between occupancy measures . Ghasemipour et al . ( 2020 ) proposed the FAIRL algorithm as an adversarial method for the forward KL divergence .

Summary: The line of reasoning and analysis followed in the paper is mostly sound. The paper claims that the use of (state-action) occupancy measure make IL and IRL methods brittle due to the high variance of these measures and their inability to transfer to other domains. These two claims are neither properly defined and grounded in the literature, nor are they isolated experimentally. It is important to show clearly that a) these are problems, b) hitherto unaddressed in the research landscape, and that the proposed methods and techniques address these problems specifically.