Auto-Encoding Inverse Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) provides a powerful framework for automating decision making . However , RL still requires significantly engineered reward functions for good practical performance . To make RL more applicable in the real-world , it is important to learn a reward function from expert demonstrations . Imitation learning offers the instruments to learn policies directly from the data , without an explicit reward function . Imitation learning enables the agents to learn to solve tasks from expert demonstrations , such as helicopter control ( Abbeel et al. , 2006 ; 2007 ; Ng et al. , 2004 ; Coates et al. , 2008 ; Abbeel et al. , 2008a ; 2010 ) , robot navigation ( Ratliff et al. , 2006 ; Abbeel et al. , 2008b ; Ziebart et al. , 2008 ; 2010 ) , and building controls ( Barrett & Linder , 2015 ) . The goal of imitation learning is to extract the expert policies from the expert demonstrations without the access to the reward signal from the environment . The algorithms in this field can be divided into two broad categories : behavioral cloning ( BC ) and Inverse Reinforcement Learning ( IRL ) . BC formulates the learning task as a supervised learning problem , which learns a mapping from the states to the actions , using the expert trajectories . However , BC methods suffer from the problem of compounding errors , i.e. , covariate shift , which only learn the actions of the expert but not reason about what the expert attempts to achieve . In the contrast , IRL directly infers the underlying reward function from the data , and then teaches the RL agent to learn a policy to achieve the highest accumulated reward . Empirical results show that IRL methods are more efficient than BC methods in multi-step decision making tasks . However , how to make IRL stable and efficient to use is still subject to research . Current IRL methods , such as adversarial imitation learning approaches , model the reward function as a discriminator to learn the mapping from the state-action pair to a scalar value , i.e. , the reward ( Ho & Ermon , 2016 ; Fu et al. , 2017 ; Ghasemipour et al. , 2020 ) . However , this formulation may be easily over-confident to focus on the minor differences between the state-action features of the expert and generated samples . To the best of our knowledge , this is the first instance of formulating the reward function as an auto-encoder , which has the ability to learn full scale differences between expert and alternative policies . The derived formulation ensures the reward signal is informative , which is efficient for optimization . Additionally , the encoding-decoding process empowers the denoising capability and makes the agent robust to the noisy expert , in more realistic settings . In the experiments , we show that the proposed method AEIRL achieves the best overall performance on both clean and noisy expert demonstrations . Our contributions are three-fold : • We propose the Auto-Encoding Inverse Reinforcement Learning ( AEIRL ) architecture , which models the reward function as a surrogate function using the reconstruction error of an auto-encoder . The reconstruction error provides more informative learning signals , compared to the binary logistic loss . • To show the contributing factors of our method , we conduct ablation studies based on different distribution divergences and alternative auto-encoders . Experiments show that they achieve comparable results which indicates the encoding-decoding process is the major contributing factor for our method . • The experimental results on the MoJoCo tasks show that our method outperforms state-ofthe-art imitation learning methods on both clean and noisy expert demonstrations . Empirical analysis show that our learned reward function can be more informative and robust . Furthermore , the learning processes of our methods are also more stable in general . 2 RELATED WORK . 2.1 INVERSE REINFORCEMENT LEARNING . Adversarial imitation learning such as GAIL , AIRL , DAC , f -GAIL , EAIRL , FAIRL ( Ho & Ermon , 2016 ; Fu et al. , 2017 ; Kostrikov et al. , 2019 ; Zhang et al. , 2020 ; Qureshi et al. , 2019 ; Ghasemipour et al. , 2020 ) formulates the learned reward function as a discriminator that learns to differentiate expert transitions from non-expert ones . Among these methods , GAIL ( Ho & Ermon , 2016 ) considers the Jensen-Shannon divergence , while AIRL ( Fu et al. , 2017 ) considers the Kullback-Leibler ( KL ) divergence . DAC ( Kostrikov et al. , 2019 ) extends GAIL to the off-policy setting and significantly improves the sample-efficiency of adversarial imitation learning . Furthermore , f -divergence is utilized in f -GAIL ( Zhang et al. , 2020 ) , which is considered more sample-efficient . Recently , FAIRL utilizes the forward KL divergence ( Ghasemipour et al. , 2020 ) and achieves better performance than AIRL ( Fu et al. , 2017 ) comprehensively , but it is still not robust enough . However , these methods rely heavily on a carefully tuned discriminator , which might easily overfit to the minor differences between the expert and the generated samples . In comparison , our auto-encoder based reward function helps to learn the full scale differences between the expert and generated samples , which provides more informative reward signals . The robustness of adversarial imitation learning is also questionable with imperfection in observations ( Stadie et al. , 2017 ; Berseth & Pal , 2020 ) , actions , transition models ( Gangwani & Peng , 2020 ; Christiano et al. , 2016 ) , expert demonstrations ( Brown et al. , 2019 ; Shiarlis et al. , 2016 ; Jing et al. , 2020 ) and their combinations ( Kim et al. , 2020 ) . Previous robust IRL methods require the demonstrations to be annotated with confidence scores ( Wu et al. , 2019 ; Brown et al. , 2019 ; Grollman & Billard , 2012 ) , when the expert data is noisy . However , these annotations are rather expensive . Compared to this , our auto-encoder based reward function helps to denoise the expert data through the encoding-decoding process . Our method AEIRL is relatively succinct and robust to noisy expert demonstrations and does not require any annotated data . Another category of IRL uses an offline similarity function to estimate the rewards ( Boularias et al. , 2011 ; Klein et al. , 2013 ; Piot et al. , 2016 ) . The idea of these methods is still inducing the expert policy by minimizing the distance between the state action distributions of the expert and sampled trajectories . To the best of our knowledge , the most powerful method in this category is Primal Wasserstein Imitation Learning ( PWIL ) ( Dadashi et al. , 2021 ) , which utilizes the upper bound of its primal form as the optimization objective . The advantage of these methods is that they are relatively more robust compared to adversarial imitation learning methods , when the expert data is noisy . However , the performance of these methods heavily depends on the similarity measurement , and therefore , it varies greatly on different tasks . Compared to PWIL , our method achieves superior performance . 2.2 AUTO-ENCODING BASED GANS . Auto-encoders have been successfully applied to improve the training stability and modes capturing in GANs . Auto-encoding based GANs can be classified into three categories : ( 1 ) utilizing an auto-encoder as the discriminator such as energy-based GANs ( Zhao et al. , 2016 ) and boundaryequilibrium GANs ( Berthelot et al. , 2017 ) ; ( 2 ) using a denoising auto-encoder to derive an auxiliary loss for the generator ( Warde-Farley & Bengio , 2017 ) ; ( 3 ) combining variational auto-encoder and GANs to generate both vivid and diverse samples by balancing the objective of reconstructing the training data and confusing the discriminator ( Larsen et al. , 2016 ) . Our method AEIRL takes inspirations from EBGAN ( Zhao et al. , 2016 ) and utilizes an auto-encoder as the reward function and derives the efficient reconstruction error based surrogate reward signal and its corresponding objective functions . 3 BACKGROUND . A Markov decision process ( MDP ) is a tuple ( S , A , T , γ , P , r ) . In this tuple , S is a state space ; A is an action space ; T is a probability matrix for state transitions ; γ ∈ ( 0 , 1 ] is a discount factor ; P is a initial-state transition distribution ; and r : S × A → R is a reward function . Additionally , we also define a stochastic policy π , which is a mapping from states to probability distributions over actions . IRL infers the reward function using the expert demonstrations , which are assumed to be the observations of optimal behaviors ( Ng & Russell , 2000 ) . In general , IRL is formulated as a bi-level optimization process meaning that iteratively training the reward function and optimizing the policy . Assume that we are given an expert policy πE , IRL ( Ziebart et al. , 2008 ; 2010 ) fits a reward function from a family of functionsR with the optimization problem : min r∈R ( max π∈Π Eπ [ r ( s , a ) ] ) −EπE [ r ( s , a ) ] ( 1 ) Moreover , the expert policy πE will only be provided by a set of expert demonstrations , so the expected reward of πE is estimated by these trajectories . IRL looks for a reward function r ( s , a ) that assigns high values to the expert policy and low values to other policies . Therefore , it allows the expert policy to be found with a certain reinforcement learning procedure : RL ( r ) = argmax π∈Π Eπ [ r ( s , a ) ] ( 2 ) The RL process will induce the expert policy via maximizing the expected cumulative rewards . Meanwhile the entropy term can be optionally added to the reinforcement learning objective to encourage the exploration in policy searching . Typically , IRL models the reward function as a discriminator which can be easily overfit to the expert data . Also its training stability is questionable once confronted with even a little noise in the expert data . Therefore , we propose auto-encoding IRL to utilize an auto-encoder based reward function which achieve strong performance both on clean and noisy expert demonstrations . 4 AUTO-ENCODING INVERSE REINFORCEMENT LEARNING . 4.1 OVERVIEW . The reward function in GAIL is a discriminator , which attempts to assign high values to the regions near the expert demonstrations and assign low values to the other regions . However , this form of reward function could be easily overfitting to the expert data . Consider the CartPole balancing task , the state of the environment is a feature consisting of position , angle , angle ’ s rate , and cart velocity , while the action is moving left or right . Here , we assume that all the expert states ’ velocity is 2 for example . When the generated states ’ velocity is 1.9 and other dimensions of state action pairs are the same as the expert ’ s , the discriminator of GAIL would still give a low reward on these generated state action pairs . However it may actually perform very well on the goal of mimicking the expert ’ s behaviors . In other words , the reward function in GAIL on this example could easily overfit to the minor differences between the expert and the sampled data , while missing the underlying goal of the expert . In our paper , we propose an auto-encoder based reward function for inverse reinforcement learning . It utilizes the reconstruction error of the state action pairs to yield an informative reward signal . The reconstruction error based reward signal significantly retains the information of state action pairs , rather than focusing on the minor differences . Such a reward signal wouldn ’ t lead to overconfidence in distinguishing the expert and the generated samples . Recall the CartPole balancing example , the mean square error between states ’ velocity 1.9 and 2 is very small . And it could still feed a good reward to the agent under this situation . Thus , the reconstruction error based reward signal focuses on the full scale differences between the expert and generated state action pairs rather than the minor parts . This yields a much more informative reward signal . Figure 5 ( See in Section 5.4 ) , shows a more informative reward signal recovered with our method . Furthermore , the training process of the policy becomes smoother in general , which is shown in Figure 11 ( See in Appendix A.6 ) . The expert demonstrations usually contains noise bacause we sample these human trajectories with sensors and other devices in the real world . Therefore , adversarial imitation learning , such as GAIL , might be easily affected to learn the noisy expert behaviors , which are not the real intentions of the expert . The reward function under this situation could overfit to the noisy features . Recall the CartPole balancing example , we assume that the expert states ’ velocity is 2 but it tends to be 2 + δ due to the noisy sampling process . When the learned policy is good enough as the sampled states ’ velocity , the discriminator in GAIL still takes it as a bad policy , and therefore , it loses its efficacy , when learning from noisy demonstrations . Figure 2 depicts the denoising process of an autoencoder . When the auto-encoder is trained to minimize the averaged squared errors , the vector points approximately towards the nearest points on the manifold , since the auto-encoder estimates the cen- ter of mass of the clean points ( Goodfellow et al. , 2016 ) . Therefore , the reconstruction error of the auto-encoder can eliminate some effects when learning from noisy expert demonstrations and make the reward signal more robust . Recall the CartPole example , the reconstruction error of the auto-encoder is δ2 which doesn ’ t play an important role in the rewards for the agent .

Summary: This paper proposes the Auto-Encoding Inverse Reinforcement Learning (AEIRL) method for better imitation learning, especially in the presence of noisy expert demonstrations. The paper’s key insight is that auto-encoders can eliminate the effects of noise from expert demonstrations and provide a more stable reward signal based on the auto-encoder error. The experiments feature extensive analyses across standard benchmarks compared to a large number of baselines.