Year: 2021

Abstract:
Traditional inverse reinforcement learning (IRL) methods require a loop to find the optimal policy for each reward update (called an inner loop), resulting in very time-consuming reward estimation. In contrast, classification-based IRL methods, which have been studied recently, do not require an inner loop and estimate rewards quickly, although it is difficult to prepare an appropriate baseline corresponding to the expert trajectory. In this study, we introduced adversarial oneclass classification into the classification-based IRL framework, and consequently developed a novel IRL method that requires only expert trajectories. We experimentally verified that the developed method can achieve the same performance as existing methods.

heading: 1 INTRODUCTION
text: Inverse reinforcement learning (IRL) (Russell, 1998) refers to the problem of estimating rewards for reinforcement learning (RL) agents to acquire policies that can reproduce expert behavior. An RL algorithm learns a policy that maximizes the cumulative discounted reward under a given reward function. An IRL algorithm does the opposite; it estimates the reward from the given policies or trajectories to satisfy the condition under the assumption that the expert is maximizing the reward.
IRL has been applied in two main areas (Ramachandran & Amir, 2007). The first is apprenticeship learning, which enables the learning of complex policies for which it is difficult to design a reward function. Compared to behavioral cloning, IRL is robust to the covariate shift problem (Ross et al., 2011) and achieves superior performance even when the amount of data is small. The second is reward learning, where IRL is used to estimate rewards from the trajectory data of human and animal action sequences and to analyze the intention of the subject. In previous studies, IRL methods have been used to analyze human walking paths (Kitani et al., 2012) and the behavior of nematodes (Yamaguchi et al., 2018).
In traditional IRL methods, the IRL loop has an inner loop that computes the optimal policy for the reward being estimated until convergence. This inner loop presents a difficulty in applying IRL to tasks with a large state-action space because it is computation-intensive. As a solution to this, classification-based IRL methods transform the IRL problem into a problem of classifying the expert’s trajectory and the trajectory to be compared. Notable methods include AIRL (Fu et al., 2017), LogReg-IRL (Uchibe, 2018), and T-REX (Brown et al., 2019).
These methods differ in the ways they are formulated, but they result in similar learning methods. Online methods, such as AIRL, collect the trajectories to be compared from the environment. Contrastingly, offline methods, such as LogReg-IRL and T-REX, collect the trajectories to be compared in advance, which enables them to further speed up and stabilize learning by not requiring access to the environment during training. However, the learning performance of current offline methods depends heavily on the properties of the trajectories to be compared or the ranking of the trajectories, which is difficult to collect.
In this study, we exploited the fact that the learning process of LogReg-IRL by binary classification is equivalent to that of a discriminator in adversarial learning, such as with generative adversarial networks (GANs) (Goodfellow et al., 2014). Specifically, we developed an innovative deep IRL method, called state-only learned one-class classifier for IRL (SOLO-IRL), in which binary classification is replaced with adversarial one-class classification. Figure 1 compares the traditional
and proposed IRL methods. The proposed method does not require an inner loop and is an offline method; thus, it can be trained extremely fast. In addition, it does not require that trajectories be compared. With these advantages, the proposed method greatly advances the application of IRL methods to real-world problems.

heading: 2 PRELIMINARIES

heading: 2.1 MARKOV DECISION PROCESS (MDP)
text: RL is a learning problem based on the Markov decision process (MDP). The MDP consists of a tuple M = 〈S,A, P,R, γ〉, where S is the state space, A is the action space, P is the state-transition probability,R is the reward function, and γ is the discount factor indicating the degree of importance for future rewards. In the MDP, the state-value function for state st at time t is represented by the Bellman equation, as follows:
V (st) = max a
{ R(st, a) +
∑ s′ p(s′|st, a)γV (s′)
} (1)
where R(st, at) is the reward for taking action at in state st and p(st+1|st, at) is the probability of transitioning to the next state st+1 when taking action at in state st.

heading: 2.2 LINEARLY SOLVABLE MDP (LMDP)
text: The linearly solvable MDP (LMDP) is an extension of the MDP in which the agent directly determines the transition probability u(st+1|st) from the current state st to the next state st+1 as the control, instead of the action at in the MDP. Then, the Bellman equation is linearized under two assumptions. First, the state-transition probability p(st+1|st, u) is assumed to be expressed as the product of the uncontrolled transition probability p̄(st+1|st) and u as follows:
p (st+1|st, u(st+1|st)) = p̄(st+1|st) exp {u(st+1|st)} (2)
The uncontrolled transition probability p̄(st+1|st) indicates a transitional relationship between the states in the environment. When a transition is impossible, i.e., p̄ = 0, then p = 0.
The second assumption is that the reward R(st, u) is composed of a state-dependent reward r(st) and penalty term DKL (p||p̄) for state-transition probability p over the divergence from the uncontrolled transition probability p̄. This assumption can be formulated as follows:
R (st, u(st+1|st)) = r(st)−DKL (p(st+1|st, u(st+1|st))||p̄(st+1|st)) (3)
whereDKL(Px||Py) represents the Kullback–Leibler (KL) divergence of Px and Py . By rearranging Eq. (3) according to the definition of the KL divergence, the following equation is obtained:
R (st, u(st+1|st)) = r(st)− ∑ s′ p(s′|st, u(s′|st))u(s′|st) (4)
Substituting Eq. (4) into the Bellman equation in Eq. (1) gives the following:
V (st) = r(st) + max u {∑ s′ p(s′|st, u(s′|st)) [ −u(s′|st) + γV (s′) ]} (5)
Eq. (2) is then substituted into Eq. (5) and the Lagrange multiplier applied with ∑ s′ p(s
′|st, u) = 1 as a constraint. Finally, the max operator is removed, resulting in the linear Bellman equation as follows:
exp{V (st)} = exp{r(st)} ∑ s′ p̄(s′|st) exp{γV (s′)} (6)
The optimal control u∗ in the LMDP is given by
u∗(st+1|st) = p̄(st+1|st) exp{γV (st+1)}∑
s′ p̄(s ′|st) exp{γV (s′)}
(7)

heading: 2.3 LOGISTIC REGRESSION-BASED IRL (LOGREG-IRL)
text: LogReg-IRL (Uchibe, 2018) is a deep IRL method in the LMDP. The following is an overview of the IRL framework in LogReg-IRL. By rearranging the linear Bellman equation in Eq. (6), the following is obtained:
exp{V (st)− r(st)} = ∑ s′ p̄(s′|st) exp{γV (s′)} (8)
Then, substituting Eq. (8) into Eq. (7) and rearranging the result gives
u∗(st+1|st) = p̄(st+1|st) exp{γV (st+1)} exp{V (st)− r(st)} u∗(st+1|st) p̄(st+1|st) = exp{r(st) + γV (st+1)− V (st)}
log u∗(st+1|st) p̄(st+1|st) = r(st) + γV (st+1)− V (st) (9)
Applying Bayes’ theorem to Eq. (9) we obtain
log u∗(st, st+1)
p̄(st, st+1) = log
u∗(st)
p̄(st) + r(st) + γV (st+1)− V (st) (10)
The left-hand side and the first term on the right-hand side of Eq. (10) are the density-ratios. The density-ratio pa/pb can be estimated by assigning the label η = 1 to the samples from the probability distribution pa, assigning η = −1 to the samples from pb, and training a classifier using logistic regression (Qin, 1998; Cheng et al., 2004; Bickel et al., 2007). First, by Bayes’ theorem, the following is obtained:
pa(x) pb(x) = p(η = 1|x) p(η = −1|x) p(η = −1) p(η = 1)
log pa(x)
pb(x) = log p(η = 1|x) p(η = −1|x) + log p(η = −1) p(η = 1)
(11)
Next, the first discriminator D1(x) is defined by the sigmoid function σ(x) = 1/{1 + exp(−x)} and a neural network f(x):
D1(x) = p(η = 1|x) = σ(f(x)) (12)
where the second term on the right-hand side of Eq. (11) can be approximated by calculating the sample number ratio Npa/Npb and taking its logarithm. For the first term, the following equation can be obtained from the definition of the discriminator in Eq. (12):
log p(η = 1|x) p(η = −1|x) = log D1(x) 1−D1(x)
= log 1 + exp{f(x)} 1 + exp{−f(x)} = log exp{f(x)} = f(x) (13)
From Eq. (13), when Npa = Npb , the following holds:
log pa(x)
pb(x) = f(x) (14)
Therefore, the density-ratio of the first term in Eq. (10) can be estimated by sampling the states s∗t ∼ τ∗ and s̄t ∼ τ̄ from the expert trajectory τ∗ according to the optimal control u∗ and the baseline trajectory τ̄ according to the uncontrolled transition probability p̄, followed by training with the following cross-entropy loss:
L1(D1) = −Es̄t∼τ̄ [log(1−D1(s̄t))]− Es∗t∼τ∗ [log(D1(s ∗ t ))] (15)
The density-ratio on the left-hand side of Eq. (10) is defined as follows using the trained f(x), reward-estimating neural network r̃(x), and state-value-estimating neural network Ṽ (x):
log u∗(st, st+1)
p̄(st, st+1) = f(st) + r̃(st) + γṼ (st+1)− Ṽ (st) (16)
The second discriminator D2 for the state-transition pair is defined as
D2(x, y) = σ(f(x) + r̃(x) + γṼ (y)− Ṽ (x)) (17) As with D1, the discriminator D2 is trained by cross-entropy loss L2, given as
L2(D2) = −E(s̄t,s̄t+1)∼τ̄ [log(1−D2(s̄t, s̄t+1))]− E(s∗t ,s∗t+1)∼τ∗ [log(D2(s ∗ t , s ∗ t+1))] (18)
In the original LogReg-IRL, an L2 regularization term is added to the loss function. Following the process described above, LogReg-IRL estimates the reward and state-value by classifying the expert and baseline trajectories. Unlike traditional IRL methods, LogReg-IRL does not require RL in the reward estimation process and, thus, it can be trained very quickly.

heading: 2.4 DIFFICULTY COLLECTING BASELINE TRAJECTORIES
text: LogReg-IRL showed that IRL in LMDP can be formulated by learning two discriminators. However, LogReg-IRL has a problem in that its learning performance is greatly affected by the baseline trajectory. For the baseline trajectory, it is desirable to collect data that follow uncontrolled transition probability p̄, such as trajectories obtained under a random policy, for a wide range of states.
However, for some tasks, the number of states that can be transitioned by a random policy may be limited. For example, in a game task, such as an Atari game, the game does not progress, and in a driving simulator task such as TORCS (Wymann et al., 2000), the car crashes into a wall immediately. In such cases, data according to a random policy cannot cover a wide range of states.
Several methods have been proposed to collect the baseline trajectory in LogReg-IRL. For Atari games, a method using state-transition pairs from random policy in any state in the expert trajectory was proposed (Uchibe, 2018). For TORCS, a method using the trajectory recorded by driving with noise added to the action output of the expert agent was proposed (Kishikawa & Arai, 2021). However, those proposed methods are task-specific, and there is no well-established generalized method for collecting baseline trajectories.
An inappropriate baseline trajectory leads to inappropriate reward estimation. The density-ratios diverge with respect to the states that the agent can reach and those for which there is no baseline trajectory, and high rewards are estimated where there are no experts. Therefore, an agent that learns according to the estimated reward may acquire a different action from the expert as the optimal policy.

heading: 3 STATE-ONLY LEARNED ONE-CLASS CLASSIFIER FOR IRL (SOLO-IRL)
text: We propose the novel IRL method SOLO-IRL, which estimates the reward given only the expert. SOLO-IRL is a combination of an IRL framework based on LMDP, a transition generator based on adversarial one-class classification, and least-squares loss. Each of these is explained below.

heading: 3.1 SOLUTION BY ADVERSARIAL ONE-CLASS CLASSIFICATION
text: Classification-based IRL methods are equivalent to learning a discriminator in adversarial learning frameworks, such as GANs (Goodfellow et al., 2014), which are binary classification problems. This means that the IRL problem can be solved by learning a discriminator that classifies the trajectory as expert or not.
Broadening our perspective to other fields, we find that the anomaly detection problem also requires a binary classifier to distinguish between normal and abnormal samples. However, in real problems, we often encounter situations in which we can obtain many normal samples but few abnormal samples. Therefore, one-class classification is a method for obtaining a binary classifier using only normal samples.
Recently, the adversarially learned one-class classifier (ALOCC) (Sabokrou et al., 2018) was proposed as a one-class classification method. In the ALOCC, the discriminator is adversarially trained with a denoising autoencoder that generates fake normal samples. Consequently, the discriminator is trained as a binary classifier that identifies normal and abnormal samples using only normal samples.
SOLO-IRL combines a classification-based IRL method with the ALOCC. The structure of SOLOIRL is illustrated in Figure 2, and its algorithm is given as Algorithm 1. In the following, we describe the learning process and features of SOLO-IRL as well as the objective function used for learning SOLO-IRL.

heading: 3.2 LEARNING PROCESS OF SOLO-IRL
text: The training of SOLO-IRL consists of two stages. In the first stage, we learn a discriminator D1 that classifies whether a state is sampled from expert data or not. Generator G1 is composed of an encoder–decoder networkR1 and generates a fake current state s̄t from the true current state s∗t plus noise ν1, as shown in Eq. (19). Then, the generator learns such that the generated s̄t and s∗t are close, and D1 judges s̄t to be the true current state.
s̄t ← R1(s∗t + ν1) (19)
Meanwhile, discriminator D1 is defined by Eq. (12) and outputs the probability that a given state is sampled from the expert. D1 learns to distinguish between states sampled from a true expert and a fake expert from a generator. Here, the density-ratio representing the expertness of each state is obtained in D1.
In the second stage, generatorG2 learns to generate a state-transition pair that is close to the sampled expert data and that discriminator D2 judges as expert. Generator G2 uses two encoder–decoder networksR2 andR3 to generate a fake state-transition pair (s̄t, s̄t+1) from the expert’s current state s∗t plus noise ν2, as given by Eqs. (20) and (21). Then, the generator learns such that the generated (s̄t, s̄t+1) and (s∗t , s ∗ t+1) are close to each other, and discriminator D2 judges the generated data to be a true state-transition pair.
s̄t ← R2(s∗t + ν2) (20) s̄t+1 ← R3(s∗t + ν2) (21)
The training of D2 is the same as that of D1 in the first stage. Here, discriminator D2 contains a reward network r̃ and state-value network Ṽ , as shown in Eq. (17). Finally, it works as an IRL algorithm to estimate the reward and state-value.
By introducing a generator and training the discriminator in an adversarial manner, the decision boundary around the expert is refined, and the appropriate reward is estimated. This makes preparation of the baseline trajectory by trial and error unnecessary. In addition, because SOLO-IRL is an offline method, it learns quickly without executing RL or interacting with the environment. To the best of our knowledge, SOLO-IRL is the only method that estimates the reward and state-value exclusively from expert trajectories.

heading: 3.3 LEAST-SQUARES LOSS AS ADVERSARIAL OBJECTIVE
text: To train the generator and discriminator, we propose using least-squares loss as an adversarial objective. The least-squares loss is represented by the following equations:
Ladv(D) = 1
2 Ex∼d∗ [(D(x)− 1)2] +
1 2 Ex̃∼d̄[(D(x̃))2] (22)
Ladv(G) = 1
2 Ex̃∼d̄[(D(x̃)− 1)2] (23)
where d∗ and d̄ denote the true and fake states or state-transition pairs, respectively. The leastsquares loss was proposed alongside LSGAN (Mao et al., 2017), which solved the learning instability and mode collapse problems of previous GANs. In SOLO-IRL, this least-squares loss is used instead of the cross-entropy loss to address the problems of learning stability and mode collapse.

heading: 3.4 RECONSTRUCTION OBJECTIVE
text: Meanwhile, the generators G1 and G2 must be trained to be close to the true expert data in addition to training by the adversarial objective. The reconstruction objective is trained by the least-squares loss given by the following equations, as in the original ALOCC:
Lrec(G1) = ||s∗t −R1(s∗t + ν1)||2 (24) Lrec(G2) = ||s∗t −R2(s∗t + ν2)||2 + ||s∗t+1 −R3(s∗t + ν2)||2 (25)
Finally, the objective of discriminators D1 and D2 becomes Ladv(D), and the objective of generators G1 and G2 becomes Ladv(G) + Lrec(G). Using these objectives, SOLO-IRL estimates the reward and state-value by training each neural network using the gradient descent method.

heading: 4 EXPERIMENTAL RESULTS AND DISCUSSION
text: We validated the performance of the proposed method in the OpenAI Gym environment (Brockman et al., 2016). For the expert trajectory, we used a trajectory generated by an agent that had learned the optimal action for the true reward as an expert. As RL algorithms, we used PPO (Schulman et al., 2017) for the tasks with a discrete action space and TD3 (Fujimoto et al., 2018) for the tasks with a continuous action space.
In SOLO-IRL, a three-layer multilayer perceptron was used as both the encoder and decoder in the generator, and a three-layer multilayer perceptron was used as the discriminator. Adam (Kingma & Ba, 2014) was used to optimize the neural network.
Algorithm 1 SOLO-IRL: State-Only Learned One-class Classifier for IRL Require: Expert trajectories τ∗, discount factor γ, noise ν1 and ν2, numbers of iterations n1 and n2 Ensure: Reward network r̃(x), state-value network Ṽ (x)
1: Initialize neural network f(x),R1(x) 2: Define discriminator D1(st) = σ(f(st)) 3: for i = 0, · · · , n1 do 4: Sample expert state s∗t from τ ∗ 5: s̄t ← R1(s∗t + ν1) 6: Train D1 according to the loss Ladv(D) in Eq. (22) 7: TrainR1 with loss Ladv(G) + Lrec(G1) in Eqs. (23) and (24) 8: end for 9: Initialize neural network r̃(x), Ṽ (x),R2(x),R3(x)
10: Define discriminator D2(st, st+1) = σ(f(st) + r̃(st) + γṼ (st+1)− Ṽ (st)) 11: Disable f updates 12: for i = 0, · · · , n2 do 13: Sample expert current state s∗t and next state s ∗ t+1 from τ ∗

heading: 14: s̄t ← R2(s∗t + ν2)

heading: 15: s̄t+1 ← R3(s∗t + ν2)
text: 16: Train D2 according to the loss Ladv(D) in Eq. (22) 17: TrainR2 andR3 according to the loss Ladv(G) + Lrec(G2) in Eqs. (23) and (25) 18: end for
We compared SOLO-IRL to LogReg-IRL (Uchibe, 2018). The baseline trajectory in LogReg-IRL was collected by running a random policy based on a uniform distribution in the environment. The LogReg-IRL implementation was created based on the SOLO-IRL implementation with the following changes: removal of the generator, changing of the cross-entropy loss, and addition of a weight decay term (coefficient: 1e-3) to the loss.
We used the following four tasks for validation.
CartPole. CartPole (Gym, b) is a basic RL task in which the agent must keep a pendulum connected to a cart upright by controlling the cart. A survival reward is given to the pendulum as long as it remains upright.
BipedalWalker. BipedalWalker (Gym, a) is a task in which a robot with four joints learns a bipedal walking task. The agent is required to output the velocities of its two hips and knees as actions, taking the state given by the virtual sensor as the input. The true reward comprises a bonus based on the speed and a penalty based on the magnitude of the action or fall.
MuJoCo. MuJoCo (Todorov et al., 2012) is an environment that collects tasks for the development of robotics. We selected Hopper and Walker2d as examples. Hopper is a task in which a one-legged robot moves forward, and Walker2d is one in which a two-legged robot walks. These two tasks require more complex continuous control than BipedalWalker.
Analysis of results. First, in each environment, we evaluated the policy of the expert agent, the random policy, the policy of the agent trained according to the reward estimated by LogReg-IRL and SOLO-IRL, and the policy obtained via behavioral cloning. We also tested the random expert distillation (RED) (Wang et al., 2019), which is a recent method of imitation learning in the MuJoCo environment1.
There are two metrics for evaluating the performance of IRL: expected value difference (EVD) (Levine et al., 2011) and expected cumulative reward. To calculate the EVD, we used the discount rate. However, since the appropriate discount rate varies depending on the environment, the value we use will also vary accordingly. Therefore, we decided to evaluate the performance using the expected cumulative reward, which is a more common of the two.
The results are presented in Table 1. The results for CartPole show that SOLO-IRL was able to achieve comparable with that of LogReg-IRL. Furthermore, with regard to the results for BipedalWalker, we saw that LogReg-IRL failed to learn, whereas SOLO-IRL obtained a good score.
The aforementioned results can be attributed to the fact that BipedalWalker has a vast state space, whereas CartPole has a relatively small state space. Consequently, the random policy could only collect data near the starting point and could not provide an appropriate baseline trajectory for the expert trajectory that progressed to the goal.
Behavioral cloning and RED generally perform better than the other methods that were used in the comparison. However, in BipedalWalker, the proposed method showed better performance than behavioral cloning due to the changing environment. In addition, RED requires more information than the proposed method because of the availability of RL. Therefore, our proposed method can be said to be superior than the traditional methods in that it can learn purely using expert trajectories.
Relationship between trajectory distribution and estimated reward in the BipedalWalker task. Among the 24 dimensions in the BipedalWalker state, Figure 3 visualizes the distribution of the expert and baseline trajectories for “velocity in the X-axis direction” and “angle of hip 1,” and Figures 4 and 5 visualize the estimated reward for these two dimensions at the starting point. As can be seen, the baseline trajectory does not cover a part of the the expert trajectory. Owing to the lack of a baseline trajectory, LogReg-IRL’s estimation failed, yielding a meaningless reward. In contrast, SOLO-IRL’s estimation, which was trained via generating samples near the expert, resulted in an appropriate reward that drove the agent forward.
Behavior of adversarial and reconstruction objectives. Figures 6 and 7 illustrate the behavior of adversarial loss and reconstruction loss during training in SOLO-IRL. With adversarial learning, the probabilities of truthfulness of the true and fake samples converge to an equilibrium near 0.5. The discriminator narrows the decision boundary toward the expert neighborhood through learning, and the generator eventually succeeds in producing samples that are close to the true. The learning success of the generator is also evident from the convergence of the reconstruction loss.
Summary of the experimental results. There is room for improving the performance of LogRegIRL using complex rule-based policies or by combining multiple policies to collect baseline tra-
1Since the author’s implementation of RED only supported MuJoCo, the experiments were also conducted only on the MuJoCo task.
jectories; however, this is expected to be significantly more difficult than adjusting the noise in SOLO-IRL. Therefore, it can be said that similar or better performance than LogReg-IRL can be achieved more easily using SOLO-IRL.

heading: 5 RELATED WORKS
text: IRL. A method similar to the proposed method is the AIRL, which is an extension of maximum entropy IRL (Ziebart et al., 2008), guided cost learning (Finn et al., 2016), and generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016). The theoretical background of the aforementioned methods is related to the proposed method in terms of entropy regularization (Chow et al., 2018). AIRL requires access to the environment, while the proposed method does not. In addition, DREX (Brown et al., 2020) is similar to the proposed method in that it creates the trajectory of the comparison target by adding noise. Since the proposed method uses adversarial learning, it can generate more appropriate samples for comparison and does not need to learn behavioral cloning.
Imitation learning. There are several methods that have been recently proposed for imitating expert demonstrations, such as RED, disagreement-regularized imitation learning (Brantley et al., 2019), and O-NAIL (Arenz & Neumann, 2020). All these methods are extensions of GAIL and require information regarding the expert’s state-action pairs. The proposed method can be used to estimate the reward from the trajectory comprising only states.
Behavioral cloning. The simplest offline learning method to imitate expert trajectory data is behavioral cloning, and (Torabi et al., 2018) is such a method that can be applied to state-only trajectories. However, compared to IRL methods, it is difficult to deal with the issues of policy transferability and covariate shift using such a method.

heading: 6 CONCLUSION
text: In this study, we exploited the fact that the classification-based IRL framework is equivalent to training a discriminator in adversarial learning and developed SOLO-IRL, in which a generator is incorporated to generate fake expert data. SOLO-IRL can be trained quickly without the need for an inner loop and easily estimates rewards with high performance exclusively from expert trajectories, with no need for baseline trajectories.
However, although it is simpler than collecting baseline trajectories, the noise used for reconstruction by the generator still requires adjustment. In future work, we will develop a method that does not require noise adjustment. We will also consider application to more advanced image-based control tasks and the incorporation of recent advances in GANs.
REPRODUCIBILITY STATEMENT
In Appendices, we have described the details of the hyperparameters and noise needed to reproduce our experiments. We also attached the source code used in our experiments.

heading: A HYPERPARAMETERS OF RL
text: This section describes the main hyperparameters used in the experiments. For RL, we used the same settings for both the training of the expert agent and the training based on IRL results.
A.1 HYPERPARAMETERS OF TD3 AGENT
The implementation of TD3 is based on https://github.com/chainer/chainerrl/ blob/master/examples/mujoco/reproduction/td3/train_td3.py, which is identical except for the environment name and the values listed in Table 2.
A.2 HYPERPARAMETERS OF PPO AGENT
The implementation of PPO is based on https://github.com/chainer/chainerrl/ blob/master/examples/mujoco/train_ppo_gym.py, which is identical except for the environment name and the values listed in Table 3.
B IRL HYPERPARAMETERS
The hyperparameters employed in training SOLO-IRL and LogReg-IRL are shown in Table 4.
B.1 ADDITIVE NOISE
For CartPole, we added noise that follows a normal distribution N(0, 0.001) to the true state. For the MuJoCo task, we added a normal distribution N(0, 0.00001) noise to Hopper and a normal distribution N(0, 0.1) noise to Walker2d.
For each dimension of BipedalWalker, we added either noise based on the normal distribution or noise for the labels. The tuning results for the experiment are shown in Table 5. The “Types of noise” column in the table lists the noise addition operations described below, and the “Parameters” column lists the noise parameters.
• “Normal” · · · sadd ← s∗ + ν, ν ∼ N(µ, σ2) • “Label” · · · sadd ← min(max(s∗ + ν, 0), 1), ν is randomly sampled from [−1, 0, 1] with
probability [pa, pb, pc]
B.2 SENSITIVITY TO NOISE SCALE
To analyze the sensitivity of the learning results to the noise scale, we conducted experiments in which we changed the standard deviation σ of the normal distribution N(0, σ2) used to generate the noise. Walker2d in the MuJoCo environment was used for the experiments.
The results are shown in Figure 8. It can be seen that the noise should have a certain level of magnitude and should not be too small or too large. The appropriate noise level is considered to be task-dependent.
B.3 DETAILS OF BEHAVIORAL CLONING
There are various implementation methods for behavioral cloning. We used supervised learning as the simplest behavioral cloning method; specifically, recording the expert’s state and action sequences and using the state as input and the action as output.
More specifically, we trained a multilayer perceptron f(x; θ) for tasks with a discrete action space using the softmax cross-entropy function shown in Eq. (26) as the loss. The number of inputs in the network was |S|, the number of outputs was equal to that of action options. The other settings of the network were the same as those of IRL.
L(θ) = −E(st,at)∼D [at log softmax (f(st; θ))] (26) In the evaluation, the argmax policy in Eq. (27) was used:
at = argmaxaf(st; θ) (27)
For tasks with a continuous action space, the multilayer perceptron was trained by using the squared loss shown in Eq. (28). The number of inputs in the network was |S|, and the number of outputs was |A|.
L(θ) = E(st,at)∼D [ (at − f(st; θ))2 ] (28)
In the evaluation, the output of the network f(st; θ) was used. Note that the output values were clipped in the defined range of the action in the environment.
B.4 DETAILS OF RED
To measure the RED scores, we used the author’s implementation https://github.com/ RuohanW/RED. We used the same hyperparameters but changed the number of trajectories to 1000.

Review summaries:
The paper proposes a state-only offline ILR algorithm (learning reward function),  SOLO-IRL, by reducing IRL to adversarial one-class classification. Compared to most existing ILR algorithms, the proposed algorithm is more efficient and requires fewer assumptions. It does not require solving RL problems in the inner loop and does not require ranked expert trajectories or assumptions on trajectories generated by uncontrolled transitions probabilities. The authors show the algorithm learns reasonable reward on two simulated control tasks and significantly outperforms the LogReg-IRL algorithm that it extends.  

The paper proposes an adversarial inverse reinforcement learning algorithm that learns purely from expert demonstrations, and does not require any online interaction with the environment or a dataset of unlabeled interactions with the environment. The key idea is to synthesize negative examples (i.e., examples of non-expert behavior) using a denoising autoencoder trained on the positive examples (i.e., expert demonstrations). Experiments on the BipedalWalker simulated locomotion task show that the proposed method learns a reward function such that an RL agent trained to maximize the learned rewards achieves higher true rewards than a prior method.

This work introduces a new IRL framework, SOLO-IRL, that learns a reward function using only expert trajectories. This has the benefit of being trained in an offline manner, which speeds up the training process. SOLO-IRL builds on top of the work of Uchibe, 2018 and exploits the fact that a discriminator can replace the binary classification used in LogReg-IRL. They improve on LogReg-IRL by overcoming the difficulty of selecting an appropriate baseline trajectories by proposing to use adversarial one-class classification (Sabokrou et al., 2018). The authors empirically demonstrate their results on the CartPole and BipedalWalker tasks, and show superior performance over LogReg-IRL.