Deep Inverse Reinforcement Learning via Adversarial One-Class Classification

1 INTRODUCTION . Inverse reinforcement learning ( IRL ) ( Russell , 1998 ) refers to the problem of estimating rewards for reinforcement learning ( RL ) agents to acquire policies that can reproduce expert behavior . An RL algorithm learns a policy that maximizes the cumulative discounted reward under a given reward function . An IRL algorithm does the opposite ; it estimates the reward from the given policies or trajectories to satisfy the condition under the assumption that the expert is maximizing the reward . IRL has been applied in two main areas ( Ramachandran & Amir , 2007 ) . The first is apprenticeship learning , which enables the learning of complex policies for which it is difficult to design a reward function . Compared to behavioral cloning , IRL is robust to the covariate shift problem ( Ross et al. , 2011 ) and achieves superior performance even when the amount of data is small . The second is reward learning , where IRL is used to estimate rewards from the trajectory data of human and animal action sequences and to analyze the intention of the subject . In previous studies , IRL methods have been used to analyze human walking paths ( Kitani et al. , 2012 ) and the behavior of nematodes ( Yamaguchi et al. , 2018 ) . In traditional IRL methods , the IRL loop has an inner loop that computes the optimal policy for the reward being estimated until convergence . This inner loop presents a difficulty in applying IRL to tasks with a large state-action space because it is computation-intensive . As a solution to this , classification-based IRL methods transform the IRL problem into a problem of classifying the expert ’ s trajectory and the trajectory to be compared . Notable methods include AIRL ( Fu et al. , 2017 ) , LogReg-IRL ( Uchibe , 2018 ) , and T-REX ( Brown et al. , 2019 ) . These methods differ in the ways they are formulated , but they result in similar learning methods . Online methods , such as AIRL , collect the trajectories to be compared from the environment . Contrastingly , offline methods , such as LogReg-IRL and T-REX , collect the trajectories to be compared in advance , which enables them to further speed up and stabilize learning by not requiring access to the environment during training . However , the learning performance of current offline methods depends heavily on the properties of the trajectories to be compared or the ranking of the trajectories , which is difficult to collect . In this study , we exploited the fact that the learning process of LogReg-IRL by binary classification is equivalent to that of a discriminator in adversarial learning , such as with generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . Specifically , we developed an innovative deep IRL method , called state-only learned one-class classifier for IRL ( SOLO-IRL ) , in which binary classification is replaced with adversarial one-class classification . Figure 1 compares the traditional and proposed IRL methods . The proposed method does not require an inner loop and is an offline method ; thus , it can be trained extremely fast . In addition , it does not require that trajectories be compared . With these advantages , the proposed method greatly advances the application of IRL methods to real-world problems . 2 PRELIMINARIES . 2.1 MARKOV DECISION PROCESS ( MDP ) . RL is a learning problem based on the Markov decision process ( MDP ) . The MDP consists of a tuple M = 〈S , A , P , R , γ〉 , where S is the state space , A is the action space , P is the state-transition probability , R is the reward function , and γ is the discount factor indicating the degree of importance for future rewards . In the MDP , the state-value function for state st at time t is represented by the Bellman equation , as follows : V ( st ) = max a { R ( st , a ) + ∑ s′ p ( s′|st , a ) γV ( s′ ) } ( 1 ) where R ( st , at ) is the reward for taking action at in state st and p ( st+1|st , at ) is the probability of transitioning to the next state st+1 when taking action at in state st . 2.2 LINEARLY SOLVABLE MDP ( LMDP ) . The linearly solvable MDP ( LMDP ) is an extension of the MDP in which the agent directly determines the transition probability u ( st+1|st ) from the current state st to the next state st+1 as the control , instead of the action at in the MDP . Then , the Bellman equation is linearized under two assumptions . First , the state-transition probability p ( st+1|st , u ) is assumed to be expressed as the product of the uncontrolled transition probability p̄ ( st+1|st ) and u as follows : p ( st+1|st , u ( st+1|st ) ) = p̄ ( st+1|st ) exp { u ( st+1|st ) } ( 2 ) The uncontrolled transition probability p̄ ( st+1|st ) indicates a transitional relationship between the states in the environment . When a transition is impossible , i.e. , p̄ = 0 , then p = 0 . The second assumption is that the reward R ( st , u ) is composed of a state-dependent reward r ( st ) and penalty term DKL ( p||p̄ ) for state-transition probability p over the divergence from the uncontrolled transition probability p̄ . This assumption can be formulated as follows : R ( st , u ( st+1|st ) ) = r ( st ) −DKL ( p ( st+1|st , u ( st+1|st ) ) ||p̄ ( st+1|st ) ) ( 3 ) whereDKL ( Px||Py ) represents the Kullback–Leibler ( KL ) divergence of Px and Py . By rearranging Eq . ( 3 ) according to the definition of the KL divergence , the following equation is obtained : R ( st , u ( st+1|st ) ) = r ( st ) − ∑ s′ p ( s′|st , u ( s′|st ) ) u ( s′|st ) ( 4 ) Substituting Eq . ( 4 ) into the Bellman equation in Eq . ( 1 ) gives the following : V ( st ) = r ( st ) + max u { ∑ s′ p ( s′|st , u ( s′|st ) ) [ −u ( s′|st ) + γV ( s′ ) ] } ( 5 ) Eq . ( 2 ) is then substituted into Eq . ( 5 ) and the Lagrange multiplier applied with ∑ s′ p ( s ′|st , u ) = 1 as a constraint . Finally , the max operator is removed , resulting in the linear Bellman equation as follows : exp { V ( st ) } = exp { r ( st ) } ∑ s′ p̄ ( s′|st ) exp { γV ( s′ ) } ( 6 ) The optimal control u∗ in the LMDP is given by u∗ ( st+1|st ) = p̄ ( st+1|st ) exp { γV ( st+1 ) } ∑ s′ p̄ ( s ′|st ) exp { γV ( s′ ) } ( 7 ) 2.3 LOGISTIC REGRESSION-BASED IRL ( LOGREG-IRL ) . LogReg-IRL ( Uchibe , 2018 ) is a deep IRL method in the LMDP . The following is an overview of the IRL framework in LogReg-IRL . By rearranging the linear Bellman equation in Eq . ( 6 ) , the following is obtained : exp { V ( st ) − r ( st ) } = ∑ s′ p̄ ( s′|st ) exp { γV ( s′ ) } ( 8 ) Then , substituting Eq . ( 8 ) into Eq . ( 7 ) and rearranging the result gives u∗ ( st+1|st ) = p̄ ( st+1|st ) exp { γV ( st+1 ) } exp { V ( st ) − r ( st ) } u∗ ( st+1|st ) p̄ ( st+1|st ) = exp { r ( st ) + γV ( st+1 ) − V ( st ) } log u∗ ( st+1|st ) p̄ ( st+1|st ) = r ( st ) + γV ( st+1 ) − V ( st ) ( 9 ) Applying Bayes ’ theorem to Eq . ( 9 ) we obtain log u∗ ( st , st+1 ) p̄ ( st , st+1 ) = log u∗ ( st ) p̄ ( st ) + r ( st ) + γV ( st+1 ) − V ( st ) ( 10 ) The left-hand side and the first term on the right-hand side of Eq . ( 10 ) are the density-ratios . The density-ratio pa/pb can be estimated by assigning the label η = 1 to the samples from the probability distribution pa , assigning η = −1 to the samples from pb , and training a classifier using logistic regression ( Qin , 1998 ; Cheng et al. , 2004 ; Bickel et al. , 2007 ) . First , by Bayes ’ theorem , the following is obtained : pa ( x ) pb ( x ) = p ( η = 1|x ) p ( η = −1|x ) p ( η = −1 ) p ( η = 1 ) log pa ( x ) pb ( x ) = log p ( η = 1|x ) p ( η = −1|x ) + log p ( η = −1 ) p ( η = 1 ) ( 11 ) Next , the first discriminator D1 ( x ) is defined by the sigmoid function σ ( x ) = 1/ { 1 + exp ( −x ) } and a neural network f ( x ) : D1 ( x ) = p ( η = 1|x ) = σ ( f ( x ) ) ( 12 ) where the second term on the right-hand side of Eq . ( 11 ) can be approximated by calculating the sample number ratio Npa/Npb and taking its logarithm . For the first term , the following equation can be obtained from the definition of the discriminator in Eq . ( 12 ) : log p ( η = 1|x ) p ( η = −1|x ) = log D1 ( x ) 1−D1 ( x ) = log 1 + exp { f ( x ) } 1 + exp { −f ( x ) } = log exp { f ( x ) } = f ( x ) ( 13 ) From Eq . ( 13 ) , when Npa = Npb , the following holds : log pa ( x ) pb ( x ) = f ( x ) ( 14 ) Therefore , the density-ratio of the first term in Eq . ( 10 ) can be estimated by sampling the states s∗t ∼ τ∗ and s̄t ∼ τ̄ from the expert trajectory τ∗ according to the optimal control u∗ and the baseline trajectory τ̄ according to the uncontrolled transition probability p̄ , followed by training with the following cross-entropy loss : L1 ( D1 ) = −Es̄t∼τ̄ [ log ( 1−D1 ( s̄t ) ) ] − Es∗t∼τ∗ [ log ( D1 ( s ∗ t ) ) ] ( 15 ) The density-ratio on the left-hand side of Eq . ( 10 ) is defined as follows using the trained f ( x ) , reward-estimating neural network r̃ ( x ) , and state-value-estimating neural network Ṽ ( x ) : log u∗ ( st , st+1 ) p̄ ( st , st+1 ) = f ( st ) + r̃ ( st ) + γṼ ( st+1 ) − Ṽ ( st ) ( 16 ) The second discriminator D2 for the state-transition pair is defined as D2 ( x , y ) = σ ( f ( x ) + r̃ ( x ) + γṼ ( y ) − Ṽ ( x ) ) ( 17 ) As with D1 , the discriminator D2 is trained by cross-entropy loss L2 , given as L2 ( D2 ) = −E ( s̄t , s̄t+1 ) ∼τ̄ [ log ( 1−D2 ( s̄t , s̄t+1 ) ) ] − E ( s∗t , s∗t+1 ) ∼τ∗ [ log ( D2 ( s ∗ t , s ∗ t+1 ) ) ] ( 18 ) In the original LogReg-IRL , an L2 regularization term is added to the loss function . Following the process described above , LogReg-IRL estimates the reward and state-value by classifying the expert and baseline trajectories . Unlike traditional IRL methods , LogReg-IRL does not require RL in the reward estimation process and , thus , it can be trained very quickly .

Summary: The paper proposes a state-only offline ILR algorithm (learning reward function),  SOLO-IRL, by reducing IRL to adversarial one-class classification. Compared to most existing ILR algorithms, the proposed algorithm is more efficient and requires fewer assumptions. It does not require solving RL problems in the inner loop and does not require ranked expert trajectories or assumptions on trajectories generated by uncontrolled transitions probabilities. The authors show the algorithm learns reasonable reward on two simulated control tasks and significantly outperforms the LogReg-IRL algorithm that it extends.