Deep Inverse Reinforcement Learning via Adversarial One-Class Classification

1 INTRODUCTION . Inverse reinforcement learning ( IRL ) ( Russell , 1998 ) refers to the problem of estimating rewards for reinforcement learning ( RL ) agents to acquire policies that can reproduce expert behavior . An RL algorithm learns a policy that maximizes the cumulative discounted reward under a given reward function . An IRL algorithm does the opposite ; it estimates the reward from the given policies or trajectories to satisfy the condition under the assumption that the expert is maximizing the reward . IRL has been applied in two main areas ( Ramachandran & Amir , 2007 ) . The first is apprenticeship learning , which enables the learning of complex policies for which it is difficult to design a reward function . Compared to behavioral cloning , IRL is robust to the covariate shift problem ( Ross et al. , 2011 ) and achieves superior performance even when the amount of data is small . The second is reward learning , where IRL is used to estimate rewards from the trajectory data of human and animal action sequences and to analyze the intention of the subject . In previous studies , IRL methods have been used to analyze human walking paths ( Kitani et al. , 2012 ) and the behavior of nematodes ( Yamaguchi et al. , 2018 ) . In traditional IRL methods , the IRL loop has an inner loop that computes the optimal policy for the reward being estimated until convergence . This inner loop presents a difficulty in applying IRL to tasks with a large state-action space because it is computation-intensive . As a solution to this , classification-based IRL methods transform the IRL problem into a problem of classifying the expert ’ s trajectory and the trajectory to be compared . Notable methods include AIRL ( Fu et al. , 2017 ) , LogReg-IRL ( Uchibe , 2018 ) , and T-REX ( Brown et al. , 2019 ) . These methods differ in the ways they are formulated , but they result in similar learning methods . Online methods , such as AIRL , collect the trajectories to be compared from the environment . Contrastingly , offline methods , such as LogReg-IRL and T-REX , collect the trajectories to be compared in advance , which enables them to further speed up and stabilize learning by not requiring access to the environment during training . However , the learning performance of current offline methods depends heavily on the properties of the trajectories to be compared or the ranking of the trajectories , which is difficult to collect . In this study , we exploited the fact that the learning process of LogReg-IRL by binary classification is equivalent to that of a discriminator in adversarial learning , such as with generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . Specifically , we developed an innovative deep IRL method , called state-only learned one-class classifier for IRL ( SOLO-IRL ) , in which binary classification is replaced with adversarial one-class classification . Figure 1 compares the traditional and proposed IRL methods . The proposed method does not require an inner loop and is an offline method ; thus , it can be trained extremely fast . In addition , it does not require that trajectories be compared . With these advantages , the proposed method greatly advances the application of IRL methods to real-world problems . 2 PRELIMINARIES . 2.1 MARKOV DECISION PROCESS ( MDP ) . RL is a learning problem based on the Markov decision process ( MDP ) . The MDP consists of a tuple M = 〈S , A , P , R , γ〉 , where S is the state space , A is the action space , P is the state-transition probability , R is the reward function , and γ is the discount factor indicating the degree of importance for future rewards . In the MDP , the state-value function for state st at time t is represented by the Bellman equation , as follows : V ( st ) = max a { R ( st , a ) + ∑ s′ p ( s′|st , a ) γV ( s′ ) } ( 1 ) where R ( st , at ) is the reward for taking action at in state st and p ( st+1|st , at ) is the probability of transitioning to the next state st+1 when taking action at in state st . 2.2 LINEARLY SOLVABLE MDP ( LMDP ) . The linearly solvable MDP ( LMDP ) is an extension of the MDP in which the agent directly determines the transition probability u ( st+1|st ) from the current state st to the next state st+1 as the control , instead of the action at in the MDP . Then , the Bellman equation is linearized under two assumptions . First , the state-transition probability p ( st+1|st , u ) is assumed to be expressed as the product of the uncontrolled transition probability p̄ ( st+1|st ) and u as follows : p ( st+1|st , u ( st+1|st ) ) = p̄ ( st+1|st ) exp { u ( st+1|st ) } ( 2 ) The uncontrolled transition probability p̄ ( st+1|st ) indicates a transitional relationship between the states in the environment . When a transition is impossible , i.e. , p̄ = 0 , then p = 0 . The second assumption is that the reward R ( st , u ) is composed of a state-dependent reward r ( st ) and penalty term DKL ( p||p̄ ) for state-transition probability p over the divergence from the uncontrolled transition probability p̄ . This assumption can be formulated as follows : R ( st , u ( st+1|st ) ) = r ( st ) −DKL ( p ( st+1|st , u ( st+1|st ) ) ||p̄ ( st+1|st ) ) ( 3 ) whereDKL ( Px||Py ) represents the Kullback–Leibler ( KL ) divergence of Px and Py . By rearranging Eq . ( 3 ) according to the definition of the KL divergence , the following equation is obtained : R ( st , u ( st+1|st ) ) = r ( st ) − ∑ s′ p ( s′|st , u ( s′|st ) ) u ( s′|st ) ( 4 ) Substituting Eq . ( 4 ) into the Bellman equation in Eq . ( 1 ) gives the following : V ( st ) = r ( st ) + max u { ∑ s′ p ( s′|st , u ( s′|st ) ) [ −u ( s′|st ) + γV ( s′ ) ] } ( 5 ) Eq . ( 2 ) is then substituted into Eq . ( 5 ) and the Lagrange multiplier applied with ∑ s′ p ( s ′|st , u ) = 1 as a constraint . Finally , the max operator is removed , resulting in the linear Bellman equation as follows : exp { V ( st ) } = exp { r ( st ) } ∑ s′ p̄ ( s′|st ) exp { γV ( s′ ) } ( 6 ) The optimal control u∗ in the LMDP is given by u∗ ( st+1|st ) = p̄ ( st+1|st ) exp { γV ( st+1 ) } ∑ s′ p̄ ( s ′|st ) exp { γV ( s′ ) } ( 7 ) 2.3 LOGISTIC REGRESSION-BASED IRL ( LOGREG-IRL ) . LogReg-IRL ( Uchibe , 2018 ) is a deep IRL method in the LMDP . The following is an overview of the IRL framework in LogReg-IRL . By rearranging the linear Bellman equation in Eq . ( 6 ) , the following is obtained : exp { V ( st ) − r ( st ) } = ∑ s′ p̄ ( s′|st ) exp { γV ( s′ ) } ( 8 ) Then , substituting Eq . ( 8 ) into Eq . ( 7 ) and rearranging the result gives u∗ ( st+1|st ) = p̄ ( st+1|st ) exp { γV ( st+1 ) } exp { V ( st ) − r ( st ) } u∗ ( st+1|st ) p̄ ( st+1|st ) = exp { r ( st ) + γV ( st+1 ) − V ( st ) } log u∗ ( st+1|st ) p̄ ( st+1|st ) = r ( st ) + γV ( st+1 ) − V ( st ) ( 9 ) Applying Bayes ’ theorem to Eq . ( 9 ) we obtain log u∗ ( st , st+1 ) p̄ ( st , st+1 ) = log u∗ ( st ) p̄ ( st ) + r ( st ) + γV ( st+1 ) − V ( st ) ( 10 ) The left-hand side and the first term on the right-hand side of Eq . ( 10 ) are the density-ratios . The density-ratio pa/pb can be estimated by assigning the label η = 1 to the samples from the probability distribution pa , assigning η = −1 to the samples from pb , and training a classifier using logistic regression ( Qin , 1998 ; Cheng et al. , 2004 ; Bickel et al. , 2007 ) . First , by Bayes ’ theorem , the following is obtained : pa ( x ) pb ( x ) = p ( η = 1|x ) p ( η = −1|x ) p ( η = −1 ) p ( η = 1 ) log pa ( x ) pb ( x ) = log p ( η = 1|x ) p ( η = −1|x ) + log p ( η = −1 ) p ( η = 1 ) ( 11 ) Next , the first discriminator D1 ( x ) is defined by the sigmoid function σ ( x ) = 1/ { 1 + exp ( −x ) } and a neural network f ( x ) : D1 ( x ) = p ( η = 1|x ) = σ ( f ( x ) ) ( 12 ) where the second term on the right-hand side of Eq . ( 11 ) can be approximated by calculating the sample number ratio Npa/Npb and taking its logarithm . For the first term , the following equation can be obtained from the definition of the discriminator in Eq . ( 12 ) : log p ( η = 1|x ) p ( η = −1|x ) = log D1 ( x ) 1−D1 ( x ) = log 1 + exp { f ( x ) } 1 + exp { −f ( x ) } = log exp { f ( x ) } = f ( x ) ( 13 ) From Eq . ( 13 ) , when Npa = Npb , the following holds : log pa ( x ) pb ( x ) = f ( x ) ( 14 ) Therefore , the density-ratio of the first term in Eq . ( 10 ) can be estimated by sampling the states s∗t ∼ τ∗ and s̄t ∼ τ̄ from the expert trajectory τ∗ according to the optimal control u∗ and the baseline trajectory τ̄ according to the uncontrolled transition probability p̄ , followed by training with the following cross-entropy loss : L1 ( D1 ) = −Es̄t∼τ̄ [ log ( 1−D1 ( s̄t ) ) ] − Es∗t∼τ∗ [ log ( D1 ( s ∗ t ) ) ] ( 15 ) The density-ratio on the left-hand side of Eq . ( 10 ) is defined as follows using the trained f ( x ) , reward-estimating neural network r̃ ( x ) , and state-value-estimating neural network Ṽ ( x ) : log u∗ ( st , st+1 ) p̄ ( st , st+1 ) = f ( st ) + r̃ ( st ) + γṼ ( st+1 ) − Ṽ ( st ) ( 16 ) The second discriminator D2 for the state-transition pair is defined as D2 ( x , y ) = σ ( f ( x ) + r̃ ( x ) + γṼ ( y ) − Ṽ ( x ) ) ( 17 ) As with D1 , the discriminator D2 is trained by cross-entropy loss L2 , given as L2 ( D2 ) = −E ( s̄t , s̄t+1 ) ∼τ̄ [ log ( 1−D2 ( s̄t , s̄t+1 ) ) ] − E ( s∗t , s∗t+1 ) ∼τ∗ [ log ( D2 ( s ∗ t , s ∗ t+1 ) ) ] ( 18 ) In the original LogReg-IRL , an L2 regularization term is added to the loss function . Following the process described above , LogReg-IRL estimates the reward and state-value by classifying the expert and baseline trajectories . Unlike traditional IRL methods , LogReg-IRL does not require RL in the reward estimation process and , thus , it can be trained very quickly .

Summary: The paper proposes an adversarial inverse reinforcement learning algorithm that learns purely from expert demonstrations, and does not require any online interaction with the environment or a dataset of unlabeled interactions with the environment. The key idea is to synthesize negative examples (i.e., examples of non-expert behavior) using a denoising autoencoder trained on the positive examples (i.e., expert demonstrations). Experiments on the BipedalWalker simulated locomotion task show that the proposed method learns a reward function such that an RL agent trained to maximize the learned rewards achieves higher true rewards than a prior method.