Deep Inverse Reinforcement Learning via Adversarial One-Class Classification
Authors: Daiko Kishikawa
Sachiyo Arai
Venue: ICLR 2022 Submitted (2022)
Keywords: inverse reinforcement learning
one-class classification

Abstract: Traditional inverse reinforcement learning (IRL) methods require a loop to find the optimal policy for each reward update (called an inner loop), resulting in very time-consuming reward estimation. In contrast, classification-based IRL methods, which have been studied recently, do not require an inner loop and estimate rewards quickly, although it is difficult to prepare an appropriate baseline corresponding to the expert trajectory. In this study, we introduced adversarial one-class classification into the classification-based IRL framework, and consequently developed a novel IRL method that requires only expert trajectories. We experimentally verified that the developed method can achieve the same performance as existing methods.

Under review as a conference paper at ICLR 2022
DEEP INVERSE REINFORCEMENT LEARNING VIA AD- VERSARIAL ONE-CLASS CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Traditional inverse reinforcement learning (IRL) methods require a loop to find the optimal policy for each reward update (called an inner loop), resulting in very üime-consuming reward estimation. In contrast, ods, which have been studied recently, do not require an inner loop and estimate rewards quickly, although it is difficult to prepare an appropriate baseline corre- sponding to the expert trajectory. In this study, we introduced adversarial one- class classification into the classification-base developed a novel IRL method that requires only expert trajectories. We experi- mentally verified that the developed method can achieve the same performance existing methods.
cussification-basee IRL meth-
IRL framework, and consequently
INTRODUCTION
Inverse reinforcement learning (IRL) (Russell 1998) refers the problem of estimating rewards for reinforcement learning (RL) agents to acquire policies that can reproduce expert behavior. An RL algorithm learns policy that maximizes the cumulative discounted reward under given reward function. An IRI. algorithm does the opposite; it estimates the reward from the given policies or trajectories to satisfy the condition under the assumption that the expert maximizing the reward.
IRL has been applied in two main areas (Ramachandran & Amir 2007). The first is apprenticeship learning, which enables the learning of complex policies for which it difficult to design a reward function. Compared to behavioral cloning, IRL is robust to the covariate shift problem (Ross et al.) 201T] and achieves superior performance even when the amount of data is small. The second is reward learning. where IRL is used to estimate rewards from the trajectory data of human and animal action sequences and to analyze the intention of the subject. In previous studies, IRL methods have been used to analyze human walking paths (Kitani et al. 2012) and the behavior of nematodes Yamaguchi et al. 2018).
In traditional IRL methods, the IRL loop has an inner loop that computes the optimal policy for the reward being estimated until convergence. This inner loop presents a difficulty in applying IRL. to tasks with large state-action space because it is computation-inntesiive As solution to this, classification- based IRL methods transform the IRL problem into a problem of classifying the expert's trajectory and the trajectory to be compared. Notable methods include AIRL (Fu et al., 2017). LogReg-IRL. (Uchibe 2018), and T-REX (Brown et al. 2019).
These methods differ in the ways they are formulated, but they result in similar learning methods. Online methods, such as AIRL, collect the trajectories to be compared from the environment. Con- trastingly, offline methods, such as LogReg-IRe and T-REX, collect the trajectories to be compared in advance, which enables them to further speed up and stabilize learning by not requiring access to the environment during training. However, the learning performance of current offline methods depends heavily the properties of the trajectories to be compared or the ranking of the trajectories, which is difficult to collect.
In this study, we exploited the fact that the learning process of LogReg-IRL by binary classification is equivalent to that of a discriminator in adversarial learning, such as with generative adversarial networks (GANs) (Goodfellow al.] 2014). Specifically, method, called state-only learned one-class classifier for IRL (SOLO-IRL), in which binary clas- sification is replaced with adversarial one-class classification. Figure compares the traditional
developed an innovative deep IRL

---

Under review as a conference paper at ICLR 2022
Expert trajectory
Expert trajectory
Divergence measurement
Divergence measurement
Generated policy/ trajectory
Generated sample
Adv. loss of discriminator
Loss
Reconst. loss
Inner loop
Discriminator (reward) update
Optimal policy learning
Reward update
Generator update
Estimated reward
Adv. loss of generator
(a) Traditional IRL
(b) SOLO-IRL
Figure 1: Comparison of traditional IRL and the proposed SOLO-IRL.
and proposed IRL methods. The proposed method does not require an inner loop and is an offline method; thus, can be trained extremely fast. In addition, it does not require that trajectories be compared. With these advantages, the proposed method greatly advances the application of IRL methods real-world problems.
PRELIMINARIES
2.1 MARKOV DECISION PROCESS (MDP)
RL is a learning problem based on the Markov decision process (MDP). The MDP consists tuple M (5, A, P, R, 7), where S is the state space, A is the action space, P is the nate-transition probability, R is the reward function, and y is the discount factor indicating the degree of importance for future rewards. In the MDP, the state-value function for state s, time t is represented by the Bellman equation, as follows:
V(s:) = max R(s).a) + Episloe
(1)
where R(st, ax) is the reward for taking action a, in state 8, and p(s++1 18e, ax) is the probability of transitioning to the next state 8441 when taking action a in state 8t-
2.2 LINEARLY SOLVABLE MDP (LMDP)
The linearly solvable MDP (LMDP) is an extension of the MDP in which the agent directly deter- mines the transition probability u($++1 (s;) from the current state 8, the next state 8t+1 as the control, instead of the action in the MDP. Then, the Bellman equation is linearized under two assumptions. First, the state-transition probability p(s++1|8e: u) assumed to be expressed as the product of the uncontrolled transition probability p(s) and u as follows:
p (st+1 u(sr+1lse)) p( (s:) exp {u(sr41|st)
The uncontrolled transition probability p($ (st) indicates transitional relationship between the states in the environment. When transition is impossible, i.c., ğ = 0, then = 0.
The second assumption is that the reward R(st, u) is composed of state-dependent reward r(s() and penalty term DEL (pllp) for state-transition probability over the divergence from the uncon- trolled transition probability This assumption can be formulated follows:
R (St, u(st+1 (s)) r(se) DEL (p(st+1 118, u(8e+ (1))=1))))[((((+1111
2

---

Under review as a conference paper at ICLR 2022
where (KL(Pa represents the Leibbler (KL) divergence of Pa and Py. By rearranging Eq. (3 according to the definition the KL divergence, the following equation is obtained:
R (st, u(sc+1ls;)) = r(st) - Σ p(s (sc. u(s' ( (((()s)))
(4)
Substituting Eq. 4 into the Bellman equation in Eq. gives the following:
V(s)) = r(s)) + max Epls p( ls u(s')
(ca)))((((]]) -u(s'(s)) + +
(5)
Eq. 22) is then substituted into Eq. (5) and the Lagrange multiplier applied with Σ., p(1 u)
constraint. Finally, the max operator is removed, resulting in the linear Bellman equation as follows:
exp(V(ss)) exp(r(ss)) Σ p(s'(s,) exp(>
(6)
The optimal control u² in the LMDP is given by
1(sc+1ls)) plsc:nls))exp(
u² (8(++1)88)
E. exp(>
2.3 LOGISTIC REGRESSION-BASEE IRL (LOGREG-IRL)
LogReg-IRL (Cchibe 2018) deep IRL method in the LMDP. The following is an overview of
the IRL framework in LogReg IRL. By rearranging the linear Bellman equation in Eq. (6) the following is obtained:
- -xs()()))r(s)) = Σ exx((()))
(8)
Then, substituting Eq.
into Eq.
7 and rearranging the result gives
u" ("(ss+1
exp(V(ss)) r(.))
u² (82+1188))
exp(r(s:) + +V +VV(8++1) V(+))
p (5t)
4n (8)+1181)
log
+(s:))>((s++)) - V(st)
(9)
p(
Applying Bayes' theorem to Eq.
9 we obtain
2d (1(,), log
u (s)))
log p(s S(+1)
+ r(()) (V(((++) V(st)
(10)
p(s.)
The left-hand side and the first term on the right-hand side of Eq. (10) are the density-ratien The density-ratio Pa/Ps Ps can be estimated by assigning the label = 1 to the samples from the prob- ability distribution Pa. assigning ア) -1 to the samples from ps, and training a classifier using
logistic regression (Qin) 1998 Cheng et al. 2004 Bickel et al.) 2007). First, by Bayes' theorem, the
following is obtained:
Pa (x
p(1 1| p(n -1)
(5(x)
p(n - 1|x) p(n 1)
Pa (x)
(x(x)) log p(n 1|x))
p(n -1)
log
p(n = - 1|x) log
(11)
p(n = 1)
Next, the first discriminator Di(x)) is defined by the sigmoid function (xx) = 1/f1 + exp(-xi)
and neural network f(x):
(x) p(u a((x))
(12)
3

---

Under review as a conference paper at ICLR 2022
where the second term the hight-hand side of Eq. (1I) can be approximated by calculating the sample number ratio /Pp./P and taking its logarithm. For the first term, the following equation can be obtained from the definition of the discriminator in Eq. (12
p(1 = 1(x) p(n 11x)
D1(2)
log
log 1- (1
+ exp[
log 1 exp(-((x)}
log cxgexp(f(x))
f(x)
(13)
From Eq. 13 when N., = Nyp, the following holds:
(a
log
f(x)
(14)
(b(x))
Therefore, the density-ratio of the first term in Eq. (0) can be estimated by sampling the states st and & from the expert trajectory x" according the optimal control and the
baseline trajectory i according to the uncontrolled transition probability p. followed by training with the following cross-entropy loss:
L1(Di) -Es,~1log(11 D) Ex;~r
[log(D(
(15)
The density-ratio on the left-hand side of Eq. (10) defined as follows using the trained fxx) reward-estimating neural network F(xr), and -ltte--anue-s neural network Vxx):
log (st, p(s,, Se41)
f(s)) f(s,) +
(16)
The second discriminator D2 for the state-transition pair is defined
D2(2, v) = (f(x) + +(x) + + - (xx)
(17)
As with Dj. the discriminator D2 is trained by cross-entropy loss L2, given as
L D2) ( [log(1 D2(5,, 5r-1))] ( (ogg (lo((22(5)) s141
(18)
In the original LogReg-IRLE an L2 regularization term is added to the loss function. Following the process described above, LogReg-IRL estimates the reward and state-value by classifying the expert and baseline trajectories. Unlike traditional IRL methods, LogReg-IRL does not require RL in the reward estimation process and, thus, it can be trained very quickly.
2.4 DIFFICULTY COLLECTING BASELINE TRAJECTORIES
LogReg-IRL showed that IRL in LMDP can be formulated by learning two discriminators. How- ever, LogReg-IRL has problem in that its learning performance is greatly affected by the baseline trajectory. For the baseline trajectory, it is desirable to collect data that follow uncontrolled transition probability p. such trajectories obtained under random policy, for wide range of states.
However, for some tasks, the number of states that can be transitioned by a random policy may be limited. For example, in game task, such an Atari game, the game does not progress, and in a driving simulator task such as TORCS (Wymann et al. (000). the car crashes into a wall immediately. such cases, data according
random policy cannot cover wide range of states.
Several methods have been proposed to collect the baseline trajectory in LogReg- IRL, For Atari games, a method using state transitoanntionn pairs from random policy in any state the expert trajectory was proposed (Uchibe. 2018). For TORCS, method using the trajectory recorded by driving with noise added to the action output of the expert agent was proposed (Kishikawa Arai 202T). How- those proposed methods are task-specific, and there no well-established generalized method for collecting baseline trajectories.
An inappropriate baseline trajectory leads to inappropriate reward estimation. The density-ratios diverge with respect to the states that the agent can reach and those for which there no baseline trajectory, and high rewards estimated where there experts. Therefore. agent that leams according to the estimated reward may acquire a different action from the expert as the optimal policy.
4

---

Under review as a conference paper at ICLR 2022
Generator
Fake expert data
Discriminator
Noise
Encoder
Decoder
X
Density-rtio
True expert
True
X
Reward
data
Fake
State-ralue
Figure 2: Proposed SOLO-IRL.
STATE-ONLY LEARNED ONE-CLASS CLASSIFIER FOR IRL (SOLO-IRL)
We propose the novel IRL method SOLO-IRL, which estimates the reward given only the expert. SOLO-IRI is combination an IRI. framework based on LMDP, transition generator based on adversarial one-class classification, and least-squares loss. Each of these explained below.
3.1 SOLUTION ADVERSARIAL ONE- CLASS CLASSIFICATION
Classification- IRL methods are equivalent to learning discriminator in adversarial learning frameworks, such as GANs (Goodfellow et al. (0144,, which are binary classification problems. This means that the IRL problem can be solved by learning discriminator that classifies the trajectory as expert or not.
Broadening perspective to other fields, find that the anomaly detection problem also requires a binary classifier to distinguish between normal and abnormal samples. However, in real prob- lems, we often encounter situations in which samples. Therefore, one-class classification is method for obtaining a binary classifier using only normal samples.
obtain many normal samples but few abnormal
Recently, the adversarially learned one-class classifier (ALOCC) (Sabokrou et al. 2018 was pro- posed as one-class classification method. In the ALOCC, the discriminator is adversarially trained with a denoising autoencoder that generates fake normal samples. Consequently, the discrimina- is trained as binary classifier that identifies normal and abnormal samples using only normal samples.
SOLO-IRL combines a classification-bass IRL method with the ALOCC. The structure of SOLO- IRI. is illustrated in Figure and its algorithm is given as Algorithm [ In the following, we describe the learning process and features of SOLO-IRL as well as the objective function used for learning SOLO-IRL.
3.2 LEARNING PROCESS OF SOLO-IRL
The training of SOLO-IRI consists two stages. the first stage, we learn a discriminator Dj that classifies whether state is sampled from expert data not. Generator G is composed of an encoder-decoder network R, and generates a fake current state 3, from the true current state st plus noise V1, shown in Eq. (19 Then, the generator learns such that the generated 5, and s are close, and D2 judges & to be the true current state.
8; f- (st + 14)
(19)
Meanwhile, discriminator D; is defined by Eq. sampled from the expert. Dj learns distinguish between states sampled from true expert and a fake expert from a generator. Here, the density- rrato representing the expertness of each state is obtained in Di.
12 and outputs the probability that a given state
5

---

Under review as a conference paper at ICLR 2022
In the second stage, generator G2 learns to generate state-transition pair that is close to the sampled expert data and that discriminator D2 judges as expert. Generator G2 uses two mncoder-decodee networks R2 and Rg generate fake state-tramsition pair (š,, 8(+1) from the expert's 's current state ot plus noise V2, as given by Eqs. (20) and (2T). Then, the generator learns such that the generated (&t, St-1) and (s/ 8(+1) are close each other, and discriminator D2 judges the generated data to be a true state- calt-transition pair.
8t. f R2(0ī + v2)
(20)
8 f R3(s; v2)
(21)
The training of D2 is the same as that of Dj in the first stage. Here, discriminator D2 contains a reward network i and state-value network V, as shown in Eq. 17p. Finally, works as an IRI.
algorithm to estimate the reward and state-value.
By introducing a generator and training the discriminator in an adversarial manner, the decision boundary around the expert is refined, and the appropriate reward is estimated. This makes prepa- ration of the baseline trajectory by trial and error unnecessary. In addition, because SOLO-IRL is an offline method, it learns quickly without executing RL or interacting with the environment. To the best of our knowledge, SOLO-IRL is the only method that estimates the reward and state-value exclusively from expert trajectories.
3.3 LEAST-SQUARES LOSS AS ADVERSARIAL OBIECTIVE
To train the generator and discriminator, propose using least-squares loss adversarial ob- jective. The least-squares loss is represented by the following equations:
1
Lado (D)
E [((x)) - 1) + (=>a(((())))
(22)
2
Lado(G) ( - 1
(23)
where d' and d denote the true and fake states state-transition pairs, respectively. The least- squares loss was proposed alongside LSGAN (Mao et al., 2017), which solved the learning insta- bility and mode collapse problems of previous GANs. In SOLO-IRL, this least-squares loss used instead of the cross-entropy loss to address the problems of learning stability and mode collapse.
3.4 RECONSTRUCTION OBJECTIVE
Meanwhile, generators G1 and G2 must be trained to be close to the true expert data in addition to training by the adversarial objective. The reconstruction objective is trained by the least-squares loss given by the following equations, as in the original ALOCC:
Live(Gi)) (G2) = || - R2(5 + 12)lla + ||s - Ra(s; + #2)11
|| (s #1)l22
(24)
(25)
Finally, the objective of discriminators D, and D2 becomes Ladv (D), and the objective of gener- ators G1 and G2 becomes Lado((G) + LG)e Using these objectives, SOLO-IRL estimates the reward and state-value by training each neural network using the gradient descent method.
EXPERIMENTAL RESULTS AND DISCUSSION
We validated the performance of the proposed method in the OpenAl Gym environment (Brockman) al. 2016). For the expert trajectory, we used trajectory generated by agent that had learned the optimal action for the true reward as an expert. As RL algorithms, we used PPO (Schulman et al. (017) for the tasks with discrete action space and TD3 (Fujimoto al. 2018) for the tasks with a continuous action space.
In SOLO-IRL, three-layer multilayer perceptron used as both the encoder and decoder in the generator, and a three-layer multilayer perceptron was used as the discriminator. Adam (Kingma & Ba 2014) used to optimize the neural network.

---

Under review as a conference paper at ICLR 2022
Algorithm SOLO-IRL: State-Only Learned One-class Classifier for IRL
Require: Expert trajectories r", discount factor noise and V2, numbers of iterations 71 and
Ensure: Reward network f(a:), state-value network V(x)
Initialize neural network / (zr). R1 (2)
2: Define discriminator Dj (s,) o (s:))
3: 00. 71 do
4: Sample expert state it from T*
5: &t < 2, [1 + 11)
6: Train Di according the loss Lado in Eq. 22
7: Train R, with loss Lado (G) + Lree (1) in Eqs. (23) and 24
8: end for
9: Initialize neural network f(æ). V (2), (2),, R3(2)
10: Define discriminator (2(s), S(+1) = (f((s) + f(s:) + ((sH)))((ss))
Disable updates
12: for 0,... .2 do
Sample expert current state ot and next state 8441 from
14: < Rs + 12)
15:
$1+1 f Rs + v2)
16: Train D2 according the loss (D) in Eq. 22
17:
Train R2 and Rg according to the loss Lado (G) + Lree (Gze in Eqs. 23 and 25
18:
end for
Table I: Validation results for the OpenAl Gym tasks. These scores cumulative true rewards (averaged over 1000 trials); the higher the better.
CartPole
Walker
Hopper
Walker2d
Expert
499.96
321.69
3682.28
5272.42
Random
22.50
-99.61
17.87
1.87
LogReg-IRL (Chhiee
2018)
498.42
-118.85
5.30
376.44
SOLO-IRL (ours)
449.11
226.16
1084.46
774.87
Behavioral cloning
500.00
135.13
3677.99
4920.64
RED
Wang et al. 2019)
3633.72
3868.98
We compared SOLO-IRL to LogReg-IRL (Uchibe 2018) The baseline trajectory in LogReg-IRL was collected by running random policy based on uniform distribution in the environment. The LogReg-IRL. implementation was created based on the SOLO-IRL implementation with the follow- ing changes: removal the generator, changing of the cross-entropy loss, and addition of a weight decay term (coefficient: 1e-3) to the loss.
We used the following four tasks for validation.
CartPole. CartPole (Gym b) is basic RL task in which the agent must keep pendulum connected to a cart upright by controlling the cart. survival reward is given to pendulum as long as remains upright.
Walker Bipedal (Gym a) is task in which robot with four joints learns bipedal walking task. The agent is required to output the velocities of its two hips and knees as actions, taking the state given by the virtual sensor as the input. The true reward comprises a bonus based on the speed and penalty based on the magnitude of the action fall.
MuJoCo. MuJoCo Todorov et al., 2012 is an environment that collects tasks for the development of robotics. We selected Hopper and Walker2d examples. Hopper is task in which lng-le robot moves forward, and Walker2d is one which a two-legged robot walks. These two tasks require more complex continuous control than Bipedal

---

Under review as a conference paper at ICLR 2022
0.50 201
222
220
3.0
820
6:44
0.0
54 40 -00
026
12P
Figure 3: Visualization of the distribution of expert trajecto- ries (blue) and baseline trajec- tories (black).
Figure 4: Visualization of the reward estimated by LogReg- IRL.
Figure 5: Visualization of the reward estimated by SOLO- IRL.
Analysis of results. First, in each environment, we evaluated the policy of the expert agent, the random policy, the policy of the agent trained according to the reward estimated by LogReg-IRL. and SOLO-IRL, and the policy obtained via behavioral cloning. We also tested the random expert distillation (RED) (Wang al., 2019), which is recent method of imitation learning in the MuJoCo environment
There are two metries for evaluating the performance of IRL: expected value difference (EVD) (Levine et al. 2011) and expected cumulative reward. To calculate the EVD, we used the discount rate. However, since the appropriate discount rate varies depending on the environment, the value we use will also vary accordingly. Therefore, we decided to evaluate the performance using the expected cumulative reward, which is more common of the two.
The results are presented in Table [ The results for CartPole show that SOLO-IRI was able to achieve comparable with that of LogReg-IRL. Furthermore, with regard to the results for Bipedal- Walker, we saw that RogReg-IRL failed to learn, whereas SOLO-IRL obtained good score.
The aforementioned results can be attributed to the fact that Bipedal has a vast state space, whereas CartPole has relatively small state space. Consequently, the random policy could only collect data near the starting point and could not provide an appropriate baseline trajectory for the expert trajectory that progressed to the goal.
Behavioral cloning and RED generally perform better than the other methods that were used in the comparison. However, in Bipkda/Walker, the proposed method showed better performance than behavioral cloning due to the changing environment. In addition, RED requires more information than the proposed method because of the availability of RL. Therefore, our proposed method can be said to be superior than the traditional methods in that it can learn purely using expert trajectories.
Relationship between trajectory distribution and estimated reward in the Bipedal task. Among the 24 dimensions in the BipedalWalker state, Figure 3 visualizes the distribution of the expert and baseline trajectories for "velocity in the X-axis direction" and "angle of hip 1," and visualize Figures[aand] visualizee the estimated reward for these two dimensions at the starting point. As can be seen, the baseline trajectory does not cover part the the expert trajectory. Owing to the lack of baseline trajectory, LogReg-IRLe SOLO-IRL's estimation, which was trained via generating samples near the expert, resulted in an appropriate reward that drove the agent forward.
estimation failed, yielding meaningless reward. In contrast,
Behavior adversarial and reconstruction objectives. Figures and illustrate the behavior of adversarial loss and reconstruction loss during training in SOLO-IRL. With adversarial learning, the probabilities of truthfulness of the true and fake samples converge to an equilibrium near 0.5. The discriminator narrows the decision boundary toward the expert neighborhood through learning, and the generator eventually succeeds in producing samples that are close to the true. The learning success of the generator is also evident from the convergence of the reconstruction loss.
Summary of the experimental results. There is room for improving the performance of LogReg- IRI. using complex rule-hased policies or by combining multiple policies to collect baseline tra-
Since the author's 's implementation of RED only supported MuloCo, the experiments were also conducted only the MuloCo task.
8

---

Under review as a conference paper at ICLR 2022
Generate
0.2
0.1
0.50-
0.40
202
6:0
902
1000
200
0.00
800
3006
Figure 6: Changes in the probability that the true sample (Discriminator) and fake sample (Generator) are true, re- spectively, during training in the first
Figure 7: Reconstruction loss during training the first stage.
stage.
jectories;
however, this is expected to be significantly more difficult than adjusting the noise in SOLO-IRL. Therefore, can be said that similar better performance than LogReg-IRL. can be achieved more easily using SOLO-IRL.
RELATED WORKS
IRI. A method similar the proposed method the AIRL, which is extension of maximum
entropy IRL (Ziebart et al. 2008), guided cost learning (Finn et al. 2016), and generative adversarial imitation learning (GAIL) (Ho & Ermon 2016). The theoretical background of the aforementioned methods is related to the proposed method in terms of entropy regularization (Chow et al.) 2018). AIRL requires access the environment, while the proposed method does not. In addition, D- REX (Brown et al., 2020) is similar to the proposed method in that it creates the trajectory of the comparison target by adding noise. Since the proposed method uses adversarial learning, it can generate more appropriate samples for comparison and does not need to learn behavioral cloning.
Imitation learning. There are several methods that have been recently proposed for imitating expert demonstrations, such as RED, disagreement-reguluen and O-NAIL (Arenz& Neumann 2020). All these methods are extensions of GAIL and require information regarding the expert's is state-action pairs. The proposed method can be used estimate the reward from the trajectory comprising only states.
imitation learning (Brantley et al., (2019),
Behavioral cloning, The simplest offline learning method to imitate expert trajectory data is behav- ioral cloning, and (Torai et af. (018)) is such a method that can be applied to state-only trajectories. However, compared to IRL methods, it is difficult deal with the issues of policy transferabilityy and covariate shift using such a method.
CONCLUSION
In this study, we exploited the fact that the classification-based IRL. framework is equivalent to training a discriminator in adversarial learning and developed SOLO-IRI,, which generator incorporated to generate fake expert data. SOLO-IRL can be trained quickly without the need for an inner loop and easily estimates rewards with high performance exclusively from expert trajectories, with need for baseline trajectories.
However, although it is simpler than collecting baseline trajectories, the noise used for reconstruction by the generator still requires adjustment. In future work, will develop method that does not require noise adjustment. We will also consider application to more advanced image-based control tasks and the incorporation of recent advances in GANs.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
In Appendices, we have described the details of the hyperparameters and noise needed to reproduce our experiments. We also attached the source code used in our experiments.
REFERENCES
Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to ad- versarial methods. arXiv preprint arXiv:2008.03352 00352000...,,, 2020.
Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning for differing training and test distributions. In Proceedings of the 24th international conference on Machine learning, pp. 81-88, 2007.
Kianté Brantley, Wen Sun, and Mikael Henaff. Disagreement-regulaizzee imitation learning. In International Conference on Learning Representations, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.0115: 2016.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International Conference on Machine Learning, pp. 783-792. PMLR, 2019.
Daniel S Brown, Wonjoon Goo, and Scott Niekum. via automatically-rankes demonstrations. Conference on robot learning, pp. 330-359, PMLR, 2020.
imitation learning
Kuang Fu Cheng, Chih-Kang Chu, et al. Semiparametric density estimation under a two-sample density ratio model. Bernoulli, 10(4):583--0) 604, 2004.
Yinlam Chow, Ofir Nachum, and Mohammad Ghavamzadeh, Path consistency learning in tsallis en- tropy regularized mdps. In International Conference Machine Learning, pp. 979-988, PMLR, 2018.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. International conference on machine learning, pp. 49-58, PMLR, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv: 110::/110.1124 248, 2017.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661 2014.
OpenAl Gym. BipedalWalker--v
BipedalWalker-ae
https://eym.oppnaa..com www.:pppaaa...c/.....eoeee
OpenAl CartPole-vl, httt::///ggg..geeaaa..ccaa//eess
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29:4565 4573, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412664412.690. 2014.
Daiko Kishikawa and Sachiyo Arai. Estimation of personal driving style via deep inverse reinforce- ment learning. Artificial Life and Roboties, pp. 1-9, 2021.
10

---

Under review as a conference paper at ICLR 2022
Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In European Conference on Computer Vision, pp. 201-214. Springer, 2012.
Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with gaussian processes. Advances in neural information processing systems, 24:19-27, 2011.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international confer- ence on computer vision, pp. 2794-28280 2802, 2017.
Jing Qin. Inferences for case-control and semiparametric two-sample density ratio models. Biometrika, 85(3):619-633) 1998.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learnin. In IJCAI, vol- ume 7, pp. 2586-2591,
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Stuart Russell.
Learning agents for uncertain environments. In Proceedings of the eleventh anmual conference on Computational learning theory, pp. 101-103, 1998.
Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. Adversarially learned one-class classifier for novelty detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3379-3388, 2018
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv:180..01954. 2018.
Eiji Uchibe. Model-free deep inverse reinforcement learning by logistic regression. Neural Process- ing Letters, 47(3)-9919905, 2018
Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, and Yiannis Demiris. Random expert dis- tillation: Imitation learning via expert policy support estimation. In International Conference Machine Learning, pp. 6536-6544. PMLR, 2019.
Bernhard Wymann, Eric Espié, Christophe Guionneau, Christos Dimitrakakis, Rémi Coulom, and Andrew Sumner. Tores, the open racing car simulator. Software available at http://torcs. source- forge. net, 4(6):2, 2000.
Shoichiro Yamaguchi, Honda Naoki, Muneki Ikeda, Yuki Tsukada, Shunji Nakano, Ikue Mori, and Shin Ishii Identification animal behavioral strategies by inverse reinforcement learning. PLoS computational biology, 1406)2e00 2018.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL. USA, 2008.
11

---

Under review as a conference paper at ICLR 2022
A HYPERPARAMETERS OF RL
This section describes the main hyperparameters used in the experiments. For RL, we used the same settings for both the training of the expert agent and the training based on IRL results.
A.1 HYPERPARAMETERS TD3 AGENT
The implementation of TD3 is based on
blob/mas mmaster/exampies/mme/ /wwo/////oo//cc//////o httpap//s///mooo///eeooooooooooo http://hwwb...cccccacaaaecaaaaaa
td3.py which is
identical except for the environment name and the values listed in Table
Table 2: Some of the hyperparameters employed in training the TD3 agent.
Hyperparameter
Value
Training steps
106
Initial exploration sample size
10a
Replay buffer size
106
Batch size
103
Discount rate
0.99
A.2 HYPERPARAMETERS OF PPO AGENT
The implementation of PPO is based on blob plob/master/exampp
mu joco/ /t trrin app__yym_pp. which is identical except for the
environment name and the values listed in Table 3
Table 3: Some of the hyperparameters employed in training the PPO agent.
Hyperparameter
Value
Training steps
106
Update interval
2048
Batch size
64
Entropy coefficient
0.001
Discount rate
0.99
12

---

Under review as a conference paper at ICLR 2022
B IRL HYPERPARAMETERS
The hyperparameters employed in training SOLO-IRL and LogReg-IRL are shown Table
Table 4: Hyperparameters employed
emp in training IRL.
Hyperparameter
Value
Learning for network f Learning rate for network Learning rate for network / Learning rate for network R1 Learning for network R2 Learning rate for network R3 Number of inputs and outputs for each network layer Probability of dropout in each layer of discriminator Probability of dropout in each layer of generator
0.00004
0.00004
0.00004
0.0001
0.0001
0.0001
(IS). 1024), (1024, 1024), (1024, 1024), (1024,
0.0, 0.7, 0.7, 0.0
0.0, 0.0, 0.0, 0.0
leaky ReLU, leaky ReLU, leaky ReLU, None
Activation function in each layer
Number of training epochs in the first stage
1000
Number of training epochs in the second stage Number steps in one epoch in the first stage Number of steps in one epoch in the second stage Batch size Discount rate
1000
100
100
1024
0.99
13

---

Under review as a conference paper at ICLR 2022
B.1 ADDITIVE NOISE
For CartPole, we added noise that follows a normal distribution N (0, 0.001) to the true state. For the MuJoCo task, we added normal distribution [(0, 0.00001) noise Hopper and normal distribution 0.11 noise to Walker2d.
For each dimension of BipedalWalker, we added either noise based on the normal distribution or noise for the labels. The tuning results for the experiment are shown in Table 5 The "Types of noise" column in the table lists the noise addition operations described below, and the "Parameters" column lists the noise parameters.
"Normal" ... Sadd + s" -U N(,, o²)
"Label" ... Sadd &- min(max(s* + 0), 1), v is randomly sampled from [-1,0, 0, 1] with probability PPa.Pb.P4 Pb: Pel
Table 5: Additive noise types and parameters for each dimension.
Dimension
Types of noise
Parameters
in state
1
Normal
0.00
2
Normal
((0,0.01)
3
Normal
((0,0.01)
4
Normal
(.000.01)
5
Normal
NO, 0.0001)
6
Normal
(O, 0.0001
7
Normal
(.0.0.01)
8
Normal
(.0,0.01)
9
Label
00.1, 0.1]
10
Normal
(0,, 0.0001)
11
Normal
(00,0.0001)
12
Normal
(.0.0.01)
13
Normal
0.0,0.01)
14
Label
00.1, 0.1]
15
Normal
((0,0.01)
16
Normal
((0,0.01)
17
Normal
(.000.01)
18
Normal
((0,0.01)
19
Normal
((000.01)
20
Normal
(.0,0.01)
21
Normal
(0000.01)
22
Normal
(.0.0.01)
23
Normal ((000.01)
24
Normal
((0,0.01)
14

---

Under review as a conference paper at ICLR 2022
B.2 SENSITIVITY TO NOISE SCALE
To analyze the sensitivity of the learning results to the noise scale, we conducted experiments in which we changed the standard deviation of the normal distribution (0,, o²) used generate the noise. Walker2d in the MuJoCo environment was used for the experiments.
The results are shown in Figure 8 It can be seen that the noise should have certain level of magnitude and should not be too small or too large. The appropriate noise level is considered to be lask-dependent.
1000
500
Score
-500
11000
10
10
10
10*
Figure 8: Relationship between noise scale and in the Walker2d task. The X-axis represents the standard deviation of the normal distribution (logarithmic scale), and the Y-axis represents the RL score (sum of true rewards) based IRL results. The bars in the graph indicate the standard deviation of the scores.
B.3 DETAILS OF BEIIAVIORAL CLONING
There are various implementation methods for behavioral cloning. We used supervised learning as the simplest behavioral cloning method; specifically, recording the expert's state and action se- quences and using the state as input and the action as output.
More specifically, we trained multilayer perceptron [(rr; 0) for tasks with a discrete action space using the softmax cross-entropy function shown in Eq. (26) as the loss. The number inputs the network was (S], the number of outputs was equal that of action options. The other settings of the network were the same as those IRL.
(()) = - (a, log softmax (f(s;; 0))]
(26)
In the evaluation, the argmax policy Eq. (27 was used:
ag = argmax, (s;; 0)
(27)
For tasks with a continuous action space, the multilayer perceptron was trained by using the squared loss shown in Eq. (28). The number of inputs in the network was [S], and the number of outputs was AA].
L( (ssaa)~D (a. - ((s; ())
(28)
In the evaluation, the output of the network (s;; 0) was used. Note that the output values were clipped in the defined range of the action in the environment.
B.4 DETAILS OF RED
To measure the RED scores, used the author's implementation /ttpps:/giite com/ RuohanW/ RED We used the same hyperparameters but changed the number of trajectories to 1000.
15


Reviews:
[{"review_id": "iBIyNPFInxn", "reviewer": "Reviewer_uTpc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths ### \n- The problem that the paper tends to address is challenging and is very relevant to the ICLR community. \n- The algorithm proposed seems simple and can be implemented using the off-the-self adversarial one-class algorithms, although tuning noise level may be required as the authors noted. \n- The paper is overall clearly written and easy to follow.\n\n### Concerns ###\n- The approach that the authors propose is very interesting. However, I am not sure whether the analysis of LogReg-IRL in Section 2.3 carries over to the proposed method. The analysis of LogReg-IRL seems to only apply when learning classifiers that distinguish between trajectories generated from uncontrolled and expert-controlled transition probabilities. It’s not shown in the paper what reward and value functions are learned by learning classifiers that distinguish between expert-controlled and noise-corrupted-export-controlled data. Therefore, theoretical analysis of the proposed method is lacking.\n- Do the reward function and transition probability in the experiments satisfy the requirements of LMDP?\n- The experiments need more work. \n   - Some ablation analysis would help to understand what change contributes to the improvement on the BipedalWalker environment among the addition of generator, weight decay, and changed cross-entropy. \n  - LogReg-IRL is affected by baseline trajectories in theory. It would be helpful to compare to related works such as D-REX (Brown et. al. 2019) (an extension of T-REX) that proposes a similar idea of injecting noise to demonstrated trajectories. \n  - Due to the randomness in network initialization and trajectory generation, experiments with more seeds are helpful. More tasks would also better demonstrate the performance of the algorithm. \n  - I appreciated that the authors are honest about the difficulty in turning the noise levels. Could you give some intuition about this? Moreover, what are the state dimensions in Table 5 in Page 13?\n  - It’s exciting to see the visualization of the rewards that SOLO-IRL learned. It will be informative to plot the true reward function too. \n\n ### Other comments / questions ###\n- Eq. (4) (rewriting Eq. (3) using the definition of KL divergence) does not seem correct. And Eq. (5) does not align with the equation in the LogReg-IRL paper. \n- Is assuming the states are finite necessary in e.g. Eq. (4) or it’s just for convenience?\n\n### Post-rebuttal comments ###\nThank the authors very much for responding to my questions and adding more experiments in a short amount of time. However, the results do not seem very promising and I am still concerned with the theoretical grounding of the proposed algorithm. Therefore, I would like to keep my score. \n", "summary_of_the_paper": "The paper proposes a state-only offline ILR algorithm (learning reward function),  SOLO-IRL, by reducing IRL to adversarial one-class classification. Compared to most existing ILR algorithms, the proposed algorithm is more efficient and requires fewer assumptions. It does not require solving RL problems in the inner loop and does not require ranked expert trajectories or assumptions on trajectories generated by uncontrolled transitions probabilities. The authors show the algorithm learns reasonable reward on two simulated control tasks and significantly outperforms the LogReg-IRL algorithm that it extends. \n", "summary_of_the_review": "Because of the concerns discussed in the Main Review tab, I am leaning towards rejecting the paper for now. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8Hhga-ITu-", "reviewer": "Reviewer_phAF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\nThis work demonstrates an interesting and insightful extension to IRL frameworks. The authors created an efficient IRL method that can be trained in a completely offline manner. By replacing the binary classification with adversarial one-class classifiction from LogReg-IRL, they remove the difficulty of picking a good baseline trajectory that LogReg-IRL requires. Furthermore, this idea of adversarially learned one-class classifier draws on the idea of anomaly detection problem to learn a good discriminator; this can potentially have benefits for a stronger baseline trajectory during training. Indeed, this was demonstrated in the results in the BipedalWalker task, where the random policy could not provide an appropriate baseline trajectory for the expert trajectory to progress to the goal.\n\n### Weaknesses\nThe main weakness of this work is the sparsity of the experimental section. A baseline that should be compared to is behavioral cloning (BC). The setting of this work is under an offline setting using only expert trajectories which is the same for BC; thus, I am curious if BC performs on-par with SOLO-IRL? The authors do not explicitly mention an advantage of their method over simple BC (although I would imagine the reward function may be an advantage), and thus I would like to see it's performance compared to BC. The authors demonstrate that the reward function is better shaped than LogReg-IRL, but they do not showcase how it can be used. It would be interesting to see if the learned reward can be used to transfer to like tasks? \n\nOne additional concern is the sparsity in tasks and baselines in the experimental section. It would be interesting to see more than 1 baseline and tasks other than BipedalWalker/CartPole (since CartPole is quite simple). The authors also mentioned that adjusting the noise in SOLO-IRL is difficult. It would be nice to see an ablations section to understand how sensitive/hard to tune that is, as compared to selecting baseline trajectories?\n\n### Miscellaneous\nA more extensive background of the IRL field would be nice. A few citations to include:\n - MaxEnt-IRL, Ziebart et al. 2008\n - Guided Cost Learning, Finn et al. 2016 \n - Generative adversarial imitation learning, Ho et al. 2016\n - Behavioral Cloning from Observation, Torabi et al. 2018", "summary_of_the_paper": "This work introduces a new IRL framework, SOLO-IRL, that learns a reward function using only expert trajectories. This has the benefit of being trained in an offline manner, which speeds up the training process. SOLO-IRL builds on top of the work of Uchibe, 2018 and exploits the fact that a discriminator can replace the binary classification used in LogReg-IRL. They improve on LogReg-IRL by overcoming the difficulty of selecting an appropriate baseline trajectories by proposing to use adversarial one-class classification (Sabokrou et al., 2018). The authors empirically demonstrate their results on the CartPole and BipedalWalker tasks, and show superior performance over LogReg-IRL.", "summary_of_the_review": "The authors introduce an interesting, new framework for IRL that can learn a reward function in an offline manner. Furthermore, they draw insights from anomaly detection problem and improve upon the LogReg-IRL. They demonstrate, empirically, that their method can learn a better reward function and achieve higher performance on two OpenAI Gym tasks. However, the experimental section for this work is very sparse; if the authors can address the concerns listed above, in particular adding BC as a baseline, I would be willing to increase my rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "87AY6Gft6z", "reviewer": "Reviewer_7Yke", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is an interesting paper, but I’m concerned that the experiments don’t compare to prior methods that also learn a reward function purely from expert demonstrations without any unlabeled data:\n - [Random Expert Distillation](https://arxiv.org/abs/1905.06750)\n - [Disagreement-Regularized Imitation Learning](https://openreview.net/forum?id=rkgbYyHtwB)\n - [O-NAIL](https://arxiv.org/abs/2008.03525)\n\nI’m also concerned that the experiments only show that the proposed method outperforms prior work on one task: BipedalWalker. I would consider raising my score if the paper included experiments with different tasks and compared to at least one of the prior methods listed above.\n\nUpdate\n-----\nThank you to the authors for adding the experimental comparisons to BC and RED on the Hopper and Walker2D simulated locomotion tasks. Unfortunately, it seems that the BC and RED baselines substantially outperform the proposed method on those two tasks, so I will keep my original score.\n", "summary_of_the_paper": "The paper proposes an adversarial inverse reinforcement learning algorithm that learns purely from expert demonstrations, and does not require any online interaction with the environment or a dataset of unlabeled interactions with the environment. The key idea is to synthesize negative examples (i.e., examples of non-expert behavior) using a denoising autoencoder trained on the positive examples (i.e., expert demonstrations). Experiments on the BipedalWalker simulated locomotion task show that the proposed method learns a reward function such that an RL agent trained to maximize the learned rewards achieves higher true rewards than a prior method.", "summary_of_the_review": "Lacking comparisons to relevant prior methods and evaluations on diverse tasks", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of inverse reinforcement learning by relying on only demonstrations and no interaction (like imitation learning). The reviewers liked the premise but had major concerns with evaluation and baselines. The paper initially received reviews tending to reject. One of the questions was about missing behavior cloning baseline which the authors added in rebuttal. But the BC baseline seems to be really competitive (in fact, better in 3 out of 4 envs) as compared to the proposed approach. In conclusion, all reviewers still believed that their concerns regarding insufficient evidence for justifying approach and missing comparisons to other prior work still stand. AC agrees with the reviewers' consensus that the paper is not yet ready for acceptance.
Decision: Reject