Auto-Encoding Inverse Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) provides a powerful framework for automating decision making . However , RL still requires significantly engineered reward functions for good practical performance . To make RL more applicable in the real-world , it is important to learn a reward function from expert demonstrations . Imitation learning offers the instruments to learn policies directly from the data , without an explicit reward function . Imitation learning enables the agents to learn to solve tasks from expert demonstrations , such as helicopter control ( Abbeel et al. , 2006 ; 2007 ; Ng et al. , 2004 ; Coates et al. , 2008 ; Abbeel et al. , 2008a ; 2010 ) , robot navigation ( Ratliff et al. , 2006 ; Abbeel et al. , 2008b ; Ziebart et al. , 2008 ; 2010 ) , and building controls ( Barrett & Linder , 2015 ) . The goal of imitation learning is to extract the expert policies from the expert demonstrations without the access to the reward signal from the environment . The algorithms in this field can be divided into two broad categories : behavioral cloning ( BC ) and Inverse Reinforcement Learning ( IRL ) . BC formulates the learning task as a supervised learning problem , which learns a mapping from the states to the actions , using the expert trajectories . However , BC methods suffer from the problem of compounding errors , i.e. , covariate shift , which only learn the actions of the expert but not reason about what the expert attempts to achieve . In the contrast , IRL directly infers the underlying reward function from the data , and then teaches the RL agent to learn a policy to achieve the highest accumulated reward . Empirical results show that IRL methods are more efficient than BC methods in multi-step decision making tasks . However , how to make IRL stable and efficient to use is still subject to research . Current IRL methods , such as adversarial imitation learning approaches , model the reward function as a discriminator to learn the mapping from the state-action pair to a scalar value , i.e. , the reward ( Ho & Ermon , 2016 ; Fu et al. , 2017 ; Ghasemipour et al. , 2020 ) . However , this formulation may be easily over-confident to focus on the minor differences between the state-action features of the expert and generated samples . To the best of our knowledge , this is the first instance of formulating the reward function as an auto-encoder , which has the ability to learn full scale differences between expert and alternative policies . The derived formulation ensures the reward signal is informative , which is efficient for optimization . Additionally , the encoding-decoding process empowers the denoising capability and makes the agent robust to the noisy expert , in more realistic settings . In the experiments , we show that the proposed method AEIRL achieves the best overall performance on both clean and noisy expert demonstrations . Our contributions are three-fold : • We propose the Auto-Encoding Inverse Reinforcement Learning ( AEIRL ) architecture , which models the reward function as a surrogate function using the reconstruction error of an auto-encoder . The reconstruction error provides more informative learning signals , compared to the binary logistic loss . • To show the contributing factors of our method , we conduct ablation studies based on different distribution divergences and alternative auto-encoders . Experiments show that they achieve comparable results which indicates the encoding-decoding process is the major contributing factor for our method . • The experimental results on the MoJoCo tasks show that our method outperforms state-ofthe-art imitation learning methods on both clean and noisy expert demonstrations . Empirical analysis show that our learned reward function can be more informative and robust . Furthermore , the learning processes of our methods are also more stable in general . 2 RELATED WORK . 2.1 INVERSE REINFORCEMENT LEARNING . Adversarial imitation learning such as GAIL , AIRL , DAC , f -GAIL , EAIRL , FAIRL ( Ho & Ermon , 2016 ; Fu et al. , 2017 ; Kostrikov et al. , 2019 ; Zhang et al. , 2020 ; Qureshi et al. , 2019 ; Ghasemipour et al. , 2020 ) formulates the learned reward function as a discriminator that learns to differentiate expert transitions from non-expert ones . Among these methods , GAIL ( Ho & Ermon , 2016 ) considers the Jensen-Shannon divergence , while AIRL ( Fu et al. , 2017 ) considers the Kullback-Leibler ( KL ) divergence . DAC ( Kostrikov et al. , 2019 ) extends GAIL to the off-policy setting and significantly improves the sample-efficiency of adversarial imitation learning . Furthermore , f -divergence is utilized in f -GAIL ( Zhang et al. , 2020 ) , which is considered more sample-efficient . Recently , FAIRL utilizes the forward KL divergence ( Ghasemipour et al. , 2020 ) and achieves better performance than AIRL ( Fu et al. , 2017 ) comprehensively , but it is still not robust enough . However , these methods rely heavily on a carefully tuned discriminator , which might easily overfit to the minor differences between the expert and the generated samples . In comparison , our auto-encoder based reward function helps to learn the full scale differences between the expert and generated samples , which provides more informative reward signals . The robustness of adversarial imitation learning is also questionable with imperfection in observations ( Stadie et al. , 2017 ; Berseth & Pal , 2020 ) , actions , transition models ( Gangwani & Peng , 2020 ; Christiano et al. , 2016 ) , expert demonstrations ( Brown et al. , 2019 ; Shiarlis et al. , 2016 ; Jing et al. , 2020 ) and their combinations ( Kim et al. , 2020 ) . Previous robust IRL methods require the demonstrations to be annotated with confidence scores ( Wu et al. , 2019 ; Brown et al. , 2019 ; Grollman & Billard , 2012 ) , when the expert data is noisy . However , these annotations are rather expensive . Compared to this , our auto-encoder based reward function helps to denoise the expert data through the encoding-decoding process . Our method AEIRL is relatively succinct and robust to noisy expert demonstrations and does not require any annotated data . Another category of IRL uses an offline similarity function to estimate the rewards ( Boularias et al. , 2011 ; Klein et al. , 2013 ; Piot et al. , 2016 ) . The idea of these methods is still inducing the expert policy by minimizing the distance between the state action distributions of the expert and sampled trajectories . To the best of our knowledge , the most powerful method in this category is Primal Wasserstein Imitation Learning ( PWIL ) ( Dadashi et al. , 2021 ) , which utilizes the upper bound of its primal form as the optimization objective . The advantage of these methods is that they are relatively more robust compared to adversarial imitation learning methods , when the expert data is noisy . However , the performance of these methods heavily depends on the similarity measurement , and therefore , it varies greatly on different tasks . Compared to PWIL , our method achieves superior performance . 2.2 AUTO-ENCODING BASED GANS . Auto-encoders have been successfully applied to improve the training stability and modes capturing in GANs . Auto-encoding based GANs can be classified into three categories : ( 1 ) utilizing an auto-encoder as the discriminator such as energy-based GANs ( Zhao et al. , 2016 ) and boundaryequilibrium GANs ( Berthelot et al. , 2017 ) ; ( 2 ) using a denoising auto-encoder to derive an auxiliary loss for the generator ( Warde-Farley & Bengio , 2017 ) ; ( 3 ) combining variational auto-encoder and GANs to generate both vivid and diverse samples by balancing the objective of reconstructing the training data and confusing the discriminator ( Larsen et al. , 2016 ) . Our method AEIRL takes inspirations from EBGAN ( Zhao et al. , 2016 ) and utilizes an auto-encoder as the reward function and derives the efficient reconstruction error based surrogate reward signal and its corresponding objective functions . 3 BACKGROUND . A Markov decision process ( MDP ) is a tuple ( S , A , T , γ , P , r ) . In this tuple , S is a state space ; A is an action space ; T is a probability matrix for state transitions ; γ ∈ ( 0 , 1 ] is a discount factor ; P is a initial-state transition distribution ; and r : S × A → R is a reward function . Additionally , we also define a stochastic policy π , which is a mapping from states to probability distributions over actions . IRL infers the reward function using the expert demonstrations , which are assumed to be the observations of optimal behaviors ( Ng & Russell , 2000 ) . In general , IRL is formulated as a bi-level optimization process meaning that iteratively training the reward function and optimizing the policy . Assume that we are given an expert policy πE , IRL ( Ziebart et al. , 2008 ; 2010 ) fits a reward function from a family of functionsR with the optimization problem : min r∈R ( max π∈Π Eπ [ r ( s , a ) ] ) −EπE [ r ( s , a ) ] ( 1 ) Moreover , the expert policy πE will only be provided by a set of expert demonstrations , so the expected reward of πE is estimated by these trajectories . IRL looks for a reward function r ( s , a ) that assigns high values to the expert policy and low values to other policies . Therefore , it allows the expert policy to be found with a certain reinforcement learning procedure : RL ( r ) = argmax π∈Π Eπ [ r ( s , a ) ] ( 2 ) The RL process will induce the expert policy via maximizing the expected cumulative rewards . Meanwhile the entropy term can be optionally added to the reinforcement learning objective to encourage the exploration in policy searching . Typically , IRL models the reward function as a discriminator which can be easily overfit to the expert data . Also its training stability is questionable once confronted with even a little noise in the expert data . Therefore , we propose auto-encoding IRL to utilize an auto-encoder based reward function which achieve strong performance both on clean and noisy expert demonstrations . 4 AUTO-ENCODING INVERSE REINFORCEMENT LEARNING . 4.1 OVERVIEW . The reward function in GAIL is a discriminator , which attempts to assign high values to the regions near the expert demonstrations and assign low values to the other regions . However , this form of reward function could be easily overfitting to the expert data . Consider the CartPole balancing task , the state of the environment is a feature consisting of position , angle , angle ’ s rate , and cart velocity , while the action is moving left or right . Here , we assume that all the expert states ’ velocity is 2 for example . When the generated states ’ velocity is 1.9 and other dimensions of state action pairs are the same as the expert ’ s , the discriminator of GAIL would still give a low reward on these generated state action pairs . However it may actually perform very well on the goal of mimicking the expert ’ s behaviors . In other words , the reward function in GAIL on this example could easily overfit to the minor differences between the expert and the sampled data , while missing the underlying goal of the expert . In our paper , we propose an auto-encoder based reward function for inverse reinforcement learning . It utilizes the reconstruction error of the state action pairs to yield an informative reward signal . The reconstruction error based reward signal significantly retains the information of state action pairs , rather than focusing on the minor differences . Such a reward signal wouldn ’ t lead to overconfidence in distinguishing the expert and the generated samples . Recall the CartPole balancing example , the mean square error between states ’ velocity 1.9 and 2 is very small . And it could still feed a good reward to the agent under this situation . Thus , the reconstruction error based reward signal focuses on the full scale differences between the expert and generated state action pairs rather than the minor parts . This yields a much more informative reward signal . Figure 5 ( See in Section 5.4 ) , shows a more informative reward signal recovered with our method . Furthermore , the training process of the policy becomes smoother in general , which is shown in Figure 11 ( See in Appendix A.6 ) . The expert demonstrations usually contains noise bacause we sample these human trajectories with sensors and other devices in the real world . Therefore , adversarial imitation learning , such as GAIL , might be easily affected to learn the noisy expert behaviors , which are not the real intentions of the expert . The reward function under this situation could overfit to the noisy features . Recall the CartPole balancing example , we assume that the expert states ’ velocity is 2 but it tends to be 2 + δ due to the noisy sampling process . When the learned policy is good enough as the sampled states ’ velocity , the discriminator in GAIL still takes it as a bad policy , and therefore , it loses its efficacy , when learning from noisy demonstrations . Figure 2 depicts the denoising process of an autoencoder . When the auto-encoder is trained to minimize the averaged squared errors , the vector points approximately towards the nearest points on the manifold , since the auto-encoder estimates the cen- ter of mass of the clean points ( Goodfellow et al. , 2016 ) . Therefore , the reconstruction error of the auto-encoder can eliminate some effects when learning from noisy expert demonstrations and make the reward signal more robust . Recall the CartPole example , the reconstruction error of the auto-encoder is δ2 which doesn ’ t play an important role in the rewards for the agent .

Summary: This paper proposes to use auto-encoder for inverse reinforcement learning. The main goal of using the auto-encoder, is to use it as the reward function, which takes the auto-encoder reconstruction error to provide reward signals for the agent. The authors claim that such approach provides more informative signal than existing works, especially adversarial imitation learning approaches. Their experiments demonstrate that this method has a better performance over other baselines, and more robust to demonstration noise.