Learning from Demonstrations with Energy based Generative Adversarial Imitation Learning
Authors: Kaifeng Zhang
Venue:  (2021)
Keywords: Learning from Demonstrations
Energy based Models
Inverse Reinforcement Learning
Imitation Learning

Abstract: Traditional reinforcement learning methods usually deal with the tasks with explicit reward signals. However, for vast majority of cases, the environment wouldn't feedback a reward signal immediately. It turns out to be a bottleneck for modern reinforcement learning approaches to be applied into more realistic scenarios. Recently, inverse reinforcement learning has made great progress in making full use of the expert demonstrations to recover the reward signal for reinforcement learning. And generative adversarial imitation learning is one promising approach. In this paper, we propose a new architecture for training generative adversarial imitation learning which is so called energy based generative adversarial imitation learning (EB-GAIL). It views the discriminator as an energy function that attributes low energies to the regions near the expert demonstrations and high energies to other regions. Therefore, a generator can be seen as a reinforcement learning procedure to sample trajectories with minimal energies (cost), while the discriminator is trained to assign high energies to these generated trajectories. In detail, EB-GAIL uses an auto-encoder architecture in place of the discriminator, with the energy being the reconstruction error. Theoretical analysis shows our EB-GAIL could match the occupancy measure with expert policy during the training process. Meanwhile, the experiments depict that EB-GAIL outperforms other SoTA methods while the training process for EB-GAIL can be more stable.

Under review as a conference paper at ICLR 2021
LEARNING FROM DEMONSTRATIONS WITH ENERGY BASED GENERATIVE ADVERSARIAL IMITATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Traditional reinforcement learning methods usually deal with the tasks with explicit reward signals. However, for vast majority of cases, the environment wouldn't 't feedback reward signal immediately. turns out to be a bottleneck for modern reinforcement learning approaches to be applied into more realistic scenarios. Recently, inverse reinforcement learning (IRL) has made great progress in making full use of the expert demonstrations to recover the reward signal for reinforcement learning. And generative adversarial imitation learning is one promising approach. In this paper, propose imitation learning which is so called energy based generative adversarial imitation learning (EB-GAIL). It views the discriminator an energy function that attributes low energies to the regions near the expert demonstrations and high energies to other regions. Therefore, generator can be seen as reinforcement learning procedure sample trajectories with minimal energies (cost), while the discriminator is trained to assign high energies to these generated trajectories. In detail, EB-GAIL. uses an auto-encoder architecture in place of the discriminator, with the energy being the reconstruction error. Theoretical analysis shows our EB-GAIL could match the occupancy measure with expert policy during the training process. Meanwhile, the experiments depict that EB-GAIL outperforms other SoTA methods while the training process for EB-GAIL can be more stable.
architecture for training generative adversarial
INTRODUCTION
Motivated by applying reinforcement learning algorithms into more realistic tasks, we find that most realistic environments cannot feed an explicit reward signal back to the agent immediately. becomes a bottleneck for traditional reinforcement learning methods to be applied into more realistic scenarios. So how to infer the latent reward function from expert demonstrations is of great significance. Recently, lot of great work have been proposed to solve this problem. They are also successfully applied in scientific inquiries, such Stanford autonomous helicopter Abbeel al. (2006) Abbecl et al. (2007) Ng et al. (2004) Coates et al. 2 Abbeel et al. (2008)) Abbeel et al. (2010). as well as practical challenges such navigation Ratiff al. (2006) Abbeel al. (2008b)
Ziebart et al. (2008) Ziebart et al. (2010) and intelligent building controls Barrett & Linder 2015).
The goal of imitation learning is mimic the expert behavior from expert demonstrations without access to reinforcement signal from the environment. The algorithms in this field can be divided into two board categories: behavioral cloning and inverse reinforcement learning. Behavioral cloning formulate this problem supervised learning problem which aims at mapping state action pairs from expert trajectories to policy. These methods suffer from the problem of compounding errors (covariate shift) which only learn the actions of the expert but not reason about what the expert is trying to achieve. By the contrast, inverse reinforcement learning recovers the reward function from expert demonstrations and then optimize the policy under such unknown reward function.
In this paper, we propose masey generative adversarial imitation learning which views the discriminator energy function without explicit probabilistic interpretation. The energy function computed by the discriminator can be viewed as a trainable cost function for the generator, while the discriminator is trained to assign low energy values to the regions of expert demonstrations, and

---

Under review as a conference paper at ICLR 2021
higher energy values outside these regions. We use auto-encoder to represent the discriminator, and the reconstruction error is thought to be the energy. There are many other choices to learn the energy function, but auto-encoder is quite efficient.
Our main contributions are summarized as follows.
An EB-GAIL framework with the discriminator using auto-encoder architecture in which the energy is the reconstruction error.
Theoretical analysis shows that the policy rolls out the trajectories that are indistinguishable from the distribution of the expert demonstrations by matching the occupancy measure with the expert policy.
Experiments show that EB-GAIL outperforms several SoTA imitation learning algorithms while the training process for EB-GAIL. can he more stable.
2 BACKGROUND
In this section, we'l 'l'll briefly introduce the basic concepts in direct reinforcement learning Sutton & Barto (998) Sugiyama (2015), inverse reinforcement learning Ngg & Russell (2000), imitation learning Bagnell (2015), and energy based models LeCun et al. (2006).
2.1 DIRECT REINFORCEMENT LEARNING
Reinforcement Learning Sutton & Barto 1998) Sugiyama 2015), which is usually used for sequential decision making problems, can help us to learn from the interactions with the environment. The process for direct reinforcement learning is that each time step, the agent receives a state 8 and chooses an action ag from the action space A, following a stochastic policy π (a, (8)). After that, the environment transit new state st+1 and gives scalar reward re back the agent. This process continues until the agent reaches a terminal state.
In this case, the training model can be thought as a Markov decision process (MDP), which is tuple (S, T, 7,, D, R). In this tuple, S is the state space; A is the action space: T = Ps. is probability matrix for state transitions, owing to the environment dynamics; 7 E (0. 1] is discount factor; D is the initial-state transition distribution; and Rt S + A is the reward function, which is assumed to be bounded in absolute value by 1. In addition, we also define that a policy æ is mapping from states to probability distributions over actions, which is also called stochastic policy.
For a certain task, the goal of direct reinforcement learning is to maximize the total future reward. For simplification, we define the value function to be prediction of the total future reward, which can be shown as discounted future reward: V = Σx= R2 Besides, we also define the action value function as Q(s, a) E[ 8, a; = a). which is the expected return for selecting action a in state &. According to the Bellman Optimality, an optimal action value function Q" Q²'(s, a) is the maximum action value achievable by any policy for state and action a.
2.2 INVERSE REINFORCEMENT LEARNING
The goal of inverse reinforcement learning is to infer the reward signal with respect to the expert demonstrations which are assumed to be the observations of optimal behaviors Ng Russell (2000).
In the past decade, lot of great work have been proposed towards enlarging the ability of reward function representation. Take some traditional methods for example, 2010, FIRL Levine & Popvić (2010) was proposed to learn set of composites features based on logical conjunctions with nonlinear functions for the reward signal. Later, non-parametric methods based on Gaussian Process(GP)) Rasmussen & Williams (2006) are proposed to enlarge the function space of latent reward to allow for non-linenn lineerit in Levine Popvić (2010). For undertaking the learning of an abstract structure with smaller data sets, Jin et al. Jin & Spanos (2015) combined deep belief networks and gaussian process to optimize the existing algorithm GP-IR..
To solve the substantial noise in the sensor measurements, some work have been applied here such as bayesian programming, probabilistic graphical model and so on. For example, in 2007, Ramachandran et al. proposed Bayesian nonparametric approach Ramachandran & Amir 2007
2

---

Under review as a conference paper at ICLR 2021
to construct the reward function features in IRL, which is so-called Bayesian IRL. Later, Choi et al. Choi & Kim (2014) extend this algorithm by defining a prior on the composite features, which are defined to be the logical conjunctions of the atomic features.
By assuming that each trajectory can be generated by multiple locally consistent reward functions, sspectation-mxxiiizaaa
Nguyen et al. used functions and the stochastic transitions between them in order to jointly improve the likelihood the expert's demonstrated trajectories Nguyen et al. (2015). As result, the most likely partition of trajectory into segments that are generated from different locally consistent reward functions selected by EM can be derived. Experiments show that the algorithm outperforms the SoTA EM clustering with maximum likelihood IRL.
(EM) algorithm learn the different reward
2.3 IMITATION LEARNING
Imitation learning is study of algorithms that can mimic the experts' demonstrations or teachers' behavior. Unlike inverse reinforcement learning, the goal of imitation learning is to obtain a policy from teachers' behaviors rather than to recover the reward function for some certain tasks.
The algorithms in imitation learning can be classified into two categories: behavioral cloning, and inverse reinforcement learning. One fatal weakness for behavioral cloning is that these methods can only learn the actions from the teacher rather than learn the motivation from teachers' behaviors. To solve this problem, inverse reinforcement learning was proposed recover the reward function for decision making problems. By assuming that the teachers' behavior is optimal, these methods tend to recover the reward function in Markov decision process. So when combined with direct reinforcement learning methods, inverse reinforcement learning can realize the process for mimicking the teachers' behaviors.
2.4 ENERGY BASED MODEL.
The essence of the energy based model LeCun et al. (2006) is to build function that maps each point of an input space to a single scalar, which is called "energy"" The learning phase is a data driven process that shapes the energy surface in such a way that the desired configurations get assigned low energies, while the incorrect ones are given high energies. Supervised learning falls into this framework: for each & in the training set, the energy of the pair (z, y) takes low values when is the correct label and higher values for incorrect y's. Similarly, when modeling alone within an unsupervised learning setting, lower energy is attributed to the data manifold. The term contrastive sample is often used refer to data point causing energy pull up, such the incorrect y's in supervised learning and points from low data density regions in unsupervised learning.
Denoted that the energy function s, the connection between probability and energy can be built through Gibbs distribution:
P(ylx) = exp(-Be(y,,)) x))
Jyer (Bs
the denominator here is the partition function which represents the total energy in the data space and is an arbitrary positive constant. The formulation of this connection might seem arbitrary, but other formulation can be obtained by re-defining the energy function.
ENERGY BASED GENERATIVE ADVERSARIAL. IMITATION LEARNING (EB-GAIL)
3.1 METHODOLOGY
The output of the discriminator goes through an objective functional in order to shape the energy function, assigning low energy to the regions near the expert demonstrations and higher energy to the other regions. In this work, we use an auto-encoder to represent the discriminator and the reconstruction error of the auto-encoder is assumed to be the energy. Meanwhile, we use margin loss function to train EB-GAB BB-GAIL while one loss function is to train the discriminator and the other loss function is assumed to be the reward function for the reinforcement learning procedure.
3

---

Under review as a conference paper at ICLR 2021
Given positive margin margin, state-action pair Xe sampled from expert demonstrations, and a state-action aatee-action pair Xs rolled out by a trained policy, the discriminator loss function Dp is formally defined by:
LD D(x)) + [margin D
(2)
where [+] = ma)
Meanwhile, the reward function for the reinforcement learning procedure is:
r(x)) (xx)
(3)
Maximizing the total reward for the reinforcement learning is similar to minimizing the second term of Lp- In practice, observe that the loss function effectively avoid gradient vanishing and mode collapse problems.
3.2
USING AUTO-ENCODER
initial policy
RL
Enc
Dec
mse
energy
expert demonstrations
Discriminator
Figure 1: The architecture for EB-GAIL.
In this work, propose that the diseriminator is structured auto-encoder, which assigns low energy to the regions near the expert demonstrations and high energy to the other regions.
The discriminator is defined as:
D(x) []De((En(())) x||
(4)
Algorithm Energy based Generative Adversarial Imitation Learning (EB-GAIL)
Require: Initial parameters of policy, discriminator 0g Wh-
Expert trajectories Tg Tg- Choose a value for margin.
Ensure:
1: for i = 0 to N do
2: Sample trajectories: Ti Te, with current policy TT.
3: Sample state-action pairs Xi T; and X with same batch size.
4: Update wi to Wi41 by decreasing with the gradient:
Ex Dw, (s, a)] + Exx V [margón Dw, (s)
5: Take policy step from 0; to 0,+1, using the TRPO update rule with the reward function DD,, (8, a), and the objective function for TRPO is:
E., Er.1(- Dre, (s, a))] (H(o)))
6: end for
Figure depicts the architecture for EB-GAIL. The reinforcement learning component is trained to roll out trajectories t, which are the sequences of state-action pairs. The discriminator D takes either expert generated state-action pairs, and estimates the energy value accordingly. Here, assume
4

---

Under review as a conference paper at ICLR 2021
the discriminator D produces non-ne non-negative values. There are also many other choices for defining the energy function et al. (2006), but the auto-encoder is an efficient one.
Algorithm I depicts the procedure for training EB-GAIL. The first step is sample the state- action pairs from expert demonstrations XE ~ Rg. The second step is to sample the state-action pairs from the trajectories rolled out by current policy X; 7i with the same batch size. Then update the discriminator by decreasing with the gradient: Ex; [V.,, Dwr (s, a)] Ex v., maargin- Du, (s, +]. The fourth step is to update the policy assuming that the reward function is - -ww (8, a).
We runs these steps with N iterations until the policy converges.
3.3 THEORETICAL ANALYSIS
In this section, we introduce theoretical analysis for EB-GAIL. We show that EB-GAIL. reaches Nash equilibrium, then the policy perfectly match the occupancy measure with the expert policy. This section is done in non-parametric setting. We also assume that and G have infinite capacity.
Firstly, we provide a definition for occupancy measure for inverse reinforcement learning.
Definition 1. (occupancy measure) The agent rolls out the trajectories which can be divided into state acate-action pairs (s, a) with policy æ. This leads to the definition of occupancy measure p (s, a) and p (s) as the density of occurrence states or state-action pairs:
P
P 0. s/T)
1=
(5)
= <
Σ
P(s: s| (s),s)
t=0
where we assume = 1 for simplicity in the following paragraph.
A basic result is that the set of valid occupancy measures D = {ox 11} can be written as feasible set of affine constraints: if Po(s) is the distribution of starting states and P(s'ls, (ss)s,,) is the
dynamics model, then D = {p 2 and P(s, a) =s)(s) + Ev P(s|s', a)p(s As E S}.
Proposition 1. (Theorem 2 in Syed al. (2008) If D, then is the occupancy measure for Ta(ass)
p(s.a) E.' p(a.aT). and np is the only policy whose occupancy measure is p.
Here we are justified in writing up to denote the unique policy for an occupancy measure Now, let us define an IRL primitive procedure, which finds a reward function such that the expert performs better than all other policies, with the reward regularized by
IRL (TT)
arg min Φ(r) (max ((T) (s)r(s,))) + Engb(s,)) rER
(6)
We are interested in the policy given by running reinforcement learning procedure on the reward function which the output of IRL.
And now we are ready to acquire the policy learned by RL under the reward function recovered by IRL.
Proposition (Proposition 3.2 in Ho & Ermon (016))
RL o IRL (Tg) = arg min zell
H(()) + pp - Prg)
Remark 1. Proposition 2 tells that ø-regularized inverse reinforcement learning seeks policy whose occupancy measure is close to the expert's, s, which is measured by the convex function o²
We present that the discriminator is to minimize da (Pr,Pmg)= H(T), where do(PT: P= Φ* (pr Pae) by modifying the IRL regularizer d, so that (g Pww) smoothly penalizes violations in difference between the occupancy measures. And the generator is to maximize the total reward for policy, which is F F D(s, ((s,a) a)) (we assume 7 for simplicity).
5

---

Under review as a conference paper at ICLR 2021
Proposition 3. Our choice (EB-GAIL) of is:
6(p PTE) min Ex (s,, a)] + En (margin D(s,a))
(8)
The proof of this proposition is in Appendix A.I.1
Remark Proposition tells that our paper, the & is determined by the loss functional, which is the margin based loss function. And we can further prove that this formulation of ø effectively help the algorithm match the oceupancy measure with expert policy during the training process.
Considered the loss functionals in section 3.1 and proposition we define:
f p(p - PT P - dx:dxe
V(G, D)
XXXR
{EEE (xxE) + Er (margin - 11x PaPng oa daPrg
XXXX
Cn
UG = / D(xi) Padxi
(10)
-r(x:)
Obviously, train the discriminator 1) minimize the quantity V and the generator G minimize the quantity U. A Nash equilibrium of EB-GAIL is a pair (G", D*) that satisfies:
V(G", & V(G", D) U(G", S D")
WD
(11)
AG
Theorem 1. If the system (EB-GAIL) reaches a Nash equilibrium, then Pr. = Prg. and V(G",D") m, where represents margin.
The proof of this theorem is in Appendix A.1.2
Remark 3. The proof follows the idea that won't violate the equation (get conclusion that m S (G"", D*) S m). And therefore, the occupancy measure can be matched (p~ = PTg) when the system reaches a Nash equilibrium. Theorem tells that using EB-GAIL loss functionals can help the trained policy to match the occupancy measure with the expert policy and V(G, D) will converge to margin(m) which indicate that the discriminator will converge while the system reaches Nash equilibrium.
Theorem 2. A Nash equilibrium of this system exists and is churacterized by (1)px Pag. and (2) there exists a constant 7 E [0, m) such that D* (x) 7 almost everywhere, where m represents margin.
The proof of this theorem is in Appendix A.1.3
Remark 4. Theorem Saarrem@tell that if the Nash equilibrium of EB-GAIL exists, then it is characterized by that the trained policy could match the occupancy measure with expert policy (the trajectories rolled by the trained policy will be indistinguishable with the expert demonstrations) and D* (x) ? almost everywhere which indicate that the discriminator could converge.
4 EXPERIMENTS
4.1
TASKS
The tasks in the presented benchmark can be divided into two categories: basic tasks and locomotion tasks. We briefly describe them in this section. We choose to implement all tasks using physics simulators rather than symbolic equations, since the former approach is less error-prone and permits easy modification of each task. Tasks with simple dynamics are implemented using Box2D, an ooen-source, freely available 2D physics simulator. Tasks with more complicated dynamics, such as locomotion, are implemented using MuJoCo, a 3D physics simulator with better modeling of contacts.

---

Under review as a conference paper at ICLR 2021
4.1.1 BASIC TASKS
We implement five basic tasks that have been widely analyzed in reinforcement learning and imitation
learning literature: Cart-Pole Balancing Stephenson (1908) (onnaldon (1960) Widrow (1964) Michie
Chambers (1968); Cart-Pole Swing Up Kimura Kobayashi (1999) Doya (2000); Mountain Car
Moore (990); Acrobot Swing DeJong & Spong ( (1994) Murray & Hauser (991] Doya (2000);
and Double Inverted Pendulum Balancing Furuta et al. (1978). These relatively low dimensional
tasks provide quick evaluations and comparisons of imitation learning algorithms.
4.1.2 LOCOMOTION TASKS
In this category, implement seven locomotion tasks of varying dynamics and difficulty: Swimmer
Purcell 1977 Coulom (2002) Levine & Koltun (2013) Schulman et al. (2015a). Hopper Murphy &
Raibert 1984 Erez al. (201 Levine Koltun (2013 |Schulman et al. (2015a). Walker Raibert
Hodgins (1991) Frez et al. (2011 Levine & Koltun (20133 3) Schulman et al. (2015a), Half-Cheetah
Wawrzynski (2007) Heess et al. (2015), Ant et al. (201 5b), Simple Humanoid Tassa et al.) (S01l)mahhumma et al. (2015b), and Full Humanoid Tassa et al. (2012). The goal for all these tasks is Tassaa
to move forward as quickly as possible. These tasks are more challenging than the basic tasks due to high degrees of freedom. In addition, great amount of exploration is needed to learn to move forward without getting stuck at local optima. Since we penalize for excessive controls as well as falling over, during the initial stage of learning, when the robot is not yet able move forward for a sufficient distance without falling, apparent local optima exist including staying at the origin or diving forward slowly.
4.2 BASELINES
In this section, we will introduce the the baseline methods used in detail.
Behavioral Cloning (BC): learning mapping from state space to action space, with supervised learning methods. Specifically, the algorithm cannot get more information from the expert any more except the expert demonstrations. So the problem of compounding errors will occur and the performance might be poor. But in fact, the performance quite promising.
Guided Cost Learning (GCL): the algorithm of Finn et al. (2016), which is actually sampling based maximum entropy inverse reinforcement learning method with neural networks as the cost function. The reward function search process and policy update process are in the inner loop of the algorithm. So will need more computation resources.
Generative Adversarial Imitation Learning (GAIL): the algorithm Ho & Ermon (016)) which use a GAN architecture for policy improvement and reward function fitting. GAIL use the discriminator to compute the reward for some stateacction pair, and then using the reward function to update the policy.
4.3 TRAINING SETTING
We used all the algorithms to train policies of the same neural network architecture for all tasks: two hidden layers of 100 units each, with tanh nonlinearities in between. All networks were always initialized randomly at the start of each trial. For each task, we gave BC, GCL, GAIL and our EB-GAIL exactly the same amount of environment interaction for training. We ran all algorithms 10 times over different random seeds in all environments. More information is depicted in Appendix
4.4 RESULTS AND DISCUSSION
Figure 2 depicts the results which are the scaled rewards for different imitation learning methods. We set the expert trajectory reward is 1.0 and random policy's 's reward is 0.0. In basic tasks, BC achieves comparable results compared with GCL and GAIL. And our proposed method EB-GAIL achieves expert performance these tasks. Obviously, our EB-G EB-GAIL outperforms other SoTA methods on this evaluation metric. In locomotion tasks, GCL and GAIL achieve higher rewards than BC except for HalfCheetah and Ant. In these tasks, EB-GAIL obtains better performance than BC, GCL. and
7

---

Under review as a conference paper at ICLR 2021
Marber oo
Marber orouucion
CartPole Balancing
CartPole Swing Up
Mountain Car
Acrobot
Morke eforccoorss
Double Inverted
Marber drouuciooss
Swimmer
Hopper
Walker
Pendulum Balancing
Mar doogucsii
HalfCheetah
Ant
Simple Humanoid
Full Humanoid
Figure 2: Comparisons for EB-GAIL and other SoTA methods on different tasks.
GAIL even with limited expert trajectories (we only use 1 ~ 10 trajectories in basic tasks and 4 ~ 25 trajectories in locomotion tasks except for humanoid tasks). And our proposed method EB-GAIL still achieves expert performance and outperform all the compared algorithms. These results definitely show our -B-L outperforms other SoTA methods on basic tasks and MuJoCo environments. Meanwhile, the deviation for our proposed EB-GAIL is also roughly lower than other SoTA methods. It shows the stability of training our EB-GAIL.
Experiment results tells that our EB-GAIL achieves expert performance basic tasks and locomotion tasks, and it basically outperforms other So'TA methods. Meanwhile the training process for EB-GAIL more stable than other SoTA methods.
CONCLUSION
In this paper, present energy based method for generative adversarial imitation learning, which is so-called EB-GAIL. It views the discriminator as an energy function that attributes low energies the regions near the expert demonstrations and high energies other regions. To learn the energy of the state-action space, we use the mean square error of an auto-encoder to represent the energy function (there are still many other choices for us to represent the energy function, but the auto-encoder is quite efficient). Theoretical analysis depicts the system could match the occupancy measure with expert policy when the system reaches Nash equilibrium. Meanwhile it also tells that if the Nash equilibrium of EB-GAIL exists, then it is characterized by that the trained policy could match the occupancy measure with expert policy and the discriminator could converge to a number between 0 and the margin. Experiment results show that our EB-GAIL achieves SoTA performance, for learning a policy that can imitate and even outperform the human experts. Meanwhile, the training procedure for (B-GAIL is more stable than other SoTA methods. As we demonstrated, our method is also quite sample efficient by learning from limited expert demonstrations. We hope that our work can further be applied into more realistic scenarios.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel, Varun Ganapathi, and Andrew Y. Ng. A learning vehicular dynamics, with application to modeling helicopters. In Advances in Neural Information Processing Systems, 2006.
Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Y. Ng. An application of reinforcement learning to aerobatic helicopter flight. In Advances in Neural Information Processing Systems, 2007.
Pieter Abbeel, Adam Coates, Timothy Hunter, and Andrew Ng. Autonomous autorotation of an ro helicopter. In International Symposium Robotics, 2008a.
Pieter Abbeel, Dmitri Dolov, Andrew Y. Ng, and Sebastian Thrun. Apprenticeship learning for motion planning with application to parking lot navigation. In JEEE/RS] International Conference on Intelligent Robots and Systems, 2008b.
Pieter Abbeel, Adam Coates, and Andrew Y. Ng. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research, 2010.
J. Andrew Bagnell. An invitation to imitation. Technical report, 2015.
Enda Barrett and Stephen Linder. Autonomous hva control, a reinforcement learning approach. Machine Learning and Knowledge Discovery in Databases, 2015.
Jaedeug Choi and Ken-Eung&e Kim. Bayesian nonparametric feature construction for inverse reinforce-
ment learning. In International Joint Conference on Artificial Intelligence, 2014.
Adam Coates, Pieter Abbeel, and Andrew Y. Ng. Learning for control from multiple demonstrations. In International Conference on Machine Learning, 2008.
Rémi Coulom. Reinforcement learning using neural networks, with applications to motor control. PhD) thesis, Institut National Polytechnique de Grenoble-INPG, 2002.
Gerald Delong and Mark Spong. Swinging up the acrobot: An example intelligent control. In Proceedings of 1994 American Control Conference-ACC"94 volume 2, pp. 2158-2162. IEEE,
1994.
PEK Donaldson. Error decorrelation: a technique for matching a class of functions. In Proceedings of the Third International Conference Medical Electronics, pp. 173-178. 1960.
Kenji Doya. Reinforcement learning in continuous time and space. Neural computation, 12(1):
219-245, 2000.
Tom Erez, Yuval Tassa, and Emanuel Todorov. Infinite horizon model predictive control for nonlinear periodic tasks. Manuscript under review, 2011.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference machine learning, pp. 49-58, 2016.
K Furuta, T Okutani, and H Sone. Computer control of double inverted pendulum. Computers & Electrical Engineering, 5(1):67-84, 1978.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944-2952, 2015.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2016.
Ming Jin and Costas Spanos. Inverse reinforcement learning via deep gaussian process. In 1r1iv:1512.080650 2015.
H Kimura and S Kobayashi. Stochastic real-valued reinforcement learning to solve nonlinear control problem. In IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference Systems, Man, and Cybernetics (Cat. No. 99CH37028), volume pp. 510-515. IEFF, 1999.

---

Under review as a conference paper at ICLR 2021
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial energy mergy-based learning. Predicting structured data, 1:0, 2006.
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine
Learning, pp. 1-9, 2013.
Sergey Levine and Zoran Popvić. Feature construction for inverse reinforcement learning. In Advances in Neural Information Processing Systems, 2010.
Donald Michie and Roger A Chambers. Boxes: An experiment in adaptive control. Machine intelligence, 2(2):137-152, 1968.
Andrew Moore. Efficient memory-based learning for robot control. Technical report, 1990.
Richard Murray and John Edmond Hauser. A case study in approximate linearization: The acrobat example. Electronics Research Laboratory, College of Engineering, University of California, 1991.
Seshashayee S Murthy and Mare Raibert. 3d balance in legged locomotion: modeling and simulation for the one-legged case. ACM SIGGRAPH Computer Graphics, 18(1):27-27, 1984.
Andrew Y. Ng and Stuart Russell, Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
Andrew Y. Ng, Adam Coates, Mark Diel, Varun Ganapathi, Jamie Schulte, Ben Tse, Eric Berger, and Eric Liang. Inverted autonomous helicopter flight via reinforcement learning. In International Symposium Experimental Robotics, 2004.
Quoc Phong Nguyen, Kian Hsiang Low, and Patrick Jaillet. Inverse reinforcement learning with locally consistent reward functions. In Advances in Neural Information Processing Systems, 2015.
Edward M Purcell. Life low reynolds number. American journal of physics, 45(1):3-11, 1977.
Mare H Raibert and Jessica K Hodgins. Animation of dynamic legged locomotion. Proceedings of the 18th annual conference Computer graphics and interactive techniques, pp. 349-358, 1991.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In International Joint Conference on Artificial Intelligence, 2007.
C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
Nathan D. Ratliff, Andrew Bagnell, and Martin A. Zinkevich. Maximum margin planning. In International Conference Machine Learning, 2006.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 201
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.022 2015b.
Andrew Stephenson. On induced stability. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 15(86):233-236 1908.
Masashi Sugiyama. Statistical Reinforcement Learning: Modern Machine Learning Approaches. CRC Press, 2015.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
Umar Syed, Michael Bowling, and Robert Schapire. Apprenticeship learning using linear pro- gramming. In International Conference on Machine Learning. 2008
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4906-4913. IEFE, 2012.
10

---

Under review as a conference paper at ICLR 2021
Pawel Wawrzynski. Learning control a 6-degree-of of-freeoomoom walking robot. In EUROCON 2007-The International Conference on" Computer as a Tool", pp. 698-705. IEEE, 2007.
Bernard Widrow, Pattern recognition and adaptive control. IEEE Transactions on Applications and Industry, 83(74):269-277, 1964.
Junbo Zhao, Machael Mathieu, and Yann LeCun, Energy-based generative adversarial network. arXiv preprint arXiv: /609.03/26, 2016.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, 2008
Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference Machine Learning, 2010.

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 OMITTED PROOFS
A.1.1 PROOF OF PROPOSITION
Our choice (EB-GAA (EB-GAIL) of Φ is:
ppp PTg) min Exg (s, a)] + E Ex((aagin D(s, a
(12)
Proof. This proof will take the JS divergence example. And we will prove that the cost regularizer
Eng (nglg(((oaa)) if e<0 otherwise
ΦGa(c)
(13)
+00
where
((x) f - - -0og(1 - e*)
if x x<0 otherwise
(14)
+00
satisfies:
6GA(Px Pag) = max En (log D ((ll) Eng [log(1 D(s,a)))
(15)
Using the logistic loss log(1 + exp(-x))), we see that applying Proposition A.1 in Ho & Ermon (2016), we get:
PP Pag) = - RRolp Pag)
Σ max P (s, Ntlt
1 exp( (+exp((-) (1)) ) Pxx (s, a)og as) as, a)log(
+ exx(r))
1
max Pr a)oog( Σ wett Σ max Par (s, al)o(((())))) + Pag (8, a))ag(1 8(7)), yER
+ Prn (s, (s,
1
(16)
1 exp( (1)) 1
1 exp(-x))
where 8(x) 1/(1 + (x)) is the sigmoid function. Because the range of is (0, 1), we can
write: 6a -PRE)= = Σ
max pr (s, allogd + Ps
8,0
(17)
Σ
(r, + Pan (s, (l)g( (Dss,a)))))aa)) a)),
max
which is the desired expression.
Obviously, in our EB-GAIL, we choose the expression for é is: min Eng (D(, a)] + E, (mmargnn - ())
(18)
This completes the proof.
n
A.1.2 PROOF THEBRREM
Lemma 1. Zhao et al. 2016) Let a, 22 0, 6(x) = ax 6/m - x]+. The minimum of o on [0, +00)
exists and is reached in if a < b, and it is reached in 0 otherwise.
Proof. The function & is defined on [0, +oo), its derivative is defined on [0, (mm) and ('(x) = b if x E (0, m) and (x if (m, +00).
So when a < b, the function is decreasing on [0, m) and increasing on (m, +00). Since it is continuous, it has minimum in m.
On the other hand, if a 2 the function e is increasing on [0, +oo), so 0 is minimum.
This completes the proof.
12

---

Under review as a conference paper at ICLR 2021
Lemma 2. Zhao et al.
(2016) I and q are probability densities, and the function 12(2) 1 if
I E otherwise (a(z) 0, then J (p())<a(()ddz if and only if Jg pp(a)yduaaa dx = 0.
Proof. Let's assume that J. 1)()) = 0. Then
((0)))())(((2)) a()))dr
(1 (P( - q()))dr
plax)da 1 g(x)do + / (o(1))) - )(x))dr
1+ / (Upla)cott) 1p0)) q(x))
(19)
=1 =1
(p(())) g(x)))dx+
(p())= q(x))dr
=0
So Jz (())> (p(z) (x)))dx O and since the term in the integral is always non-negative, ((())>q(a)(p(a (p(x) - ((z)) 0 for almost all x. And p(x) g(x)) = 0 implies p(()>>(()) 0, so ((x)><(() = almost everywhere.
This completes the proof.
Theorem If the system (EB-GAIL) reaches a Nash equilibrium, then Pr = -PrR and V(G", DD)) = m, where m represents margin.
Proof. First we observe that
V(G", [ PaR (xx) / pr(x))mm D(xi)l
(20)
(prx (pTx( Per (xx)mm )(xx) Xx
Lemma shows: (1) D*)x) almost everywhere. To verify it, let us assume that there such that D (x)
Let D(x) min (am(DD"( (x), Then V(G", D) < V(G", D which violates equation 11 (2) The function ø reaches its minimum in if a < and in 0 otherwise. So V(G", D) reaches its minimum when we replace D*)x) by these values. We obtain
exists set of measure
V(G D') m 1 (xee))
(xF))xx
(21)
XR
(1 (x) (xE))dxx
Pr. (xF)dxx + mm/ berg(keossse-teeollttre - pa- )dxE
X
X
+ m
(xee (xE) Px- (xE))
X
The second term in equation
is non-positive, V(G", (*)) < m.
13

---

Under review as a conference paper at ICLR 2021
By putting the ideal generator that generates Pdata into the right side of equation get
[ X
f Pre((x)D"xxe)dxxe
px (x=))D((xxxdde
(22)
Thus by equation 20
I X
Px (xx)) + / Pr. (x.(x=) D''x==) SV(G", XE
(23)
and since (x) m. we get S VV(G", D*).
Thus, m S VV(G", DJ) S m, so V(G", D = m. According to equation 21 we see that can only happen if Ix pr (x)xxxx 0. According to Lemma it true and only if Px Pag
This completes the proof.
A.1.3 PROOF OF THEOREM 2
Nash equilibrium of this system exists and is characterized by (1)ps = Pag. and (2) there exists a constant y [0, m) such that D* (xx) y (almost everywhere).
Proof. The sufficient conditions obvious. The necessary condition л comes from theorem and the necessary condition on (x)x)= I m is from the proof theorem |
Let assume that (x) is not constant almost everywhere and find contradiction. If is not, then there exists a constant C and a set S of non-zero measure such that Vx E (x) (x) < and VX & S, D' (x) C. In addition
choose S such that there exists subset S' S of non-zero measure such that PTg (x) > 0 on S'. We can build a generator policy Po such that Pro (x) I PTg (x) over S and Pro (x) Pag (x) over S', We compute
U(G",D" - U(Go, D*) { (DTE Pro) (x)dxx
(PKS Pmo)(1' CC)dx
(24)
I (PTE Pro) (D) (x) C)dx+
(pEn - PDo C)dx > 0
RNNS
which violates equation
This completes the proof.
A.2 EXPERIMENTS
A.2.1
LOCOMOTION TASKS
Figure depicts locomotion tasks' environments.
A.2.2
TRAINING SETTING
For BC, split given dataset of state-action pairs into 70% training data and 30% validation data. The policy trained with supervised learning, with minibatches of 64 examples, until validation error stops decreasing.
For GCL, we use neural network to represent the reward function, and the learning rate for policy update is 0.0001, the learning rate for reward update is 0.001, with minibatches of 64 examples.
14

---

Under review as a conference paper at ICLR 2021
(b)
(c)
(f)
Figure 3: Illustrations for locomotion tasks: (a) Swimmer; (b) Hopper; (c) Walker; (d) Half-Cheetah;
(e) Ant; (f) Simple Humanoid; and (g) Full Humanoid.
For GAIL, the discriminator networks used hidden layers 100 units each, with tanh nonlineari- ties as its architecture. And the learning rate for policy update is 0.0001, and the learning rate for discriminator update is 0.001, with minibatches of 64 examples.
For our EB-GAIL, the discriminator (auto-engy (nutoe) networks used four hidden layers of 100 units each, with tanh nonlinearities as its architecture. And the learning rate for policy update is 0.0001, the learning rate for discriminator (auto-encoder) update is 0.001, and margin is 5, with minibatches of examples.
15


Reviews:
[{"review_id": "uciHshy3bfe", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a learning framework for imitation learning (IL) that uses an energy-based objective for generative adversarial imitation learning (GAIL).\n\nPros\n+ Clarity. The paper is easy to understand and follow.\n+ Theoretical analysis is sound (not novel however -- see below).\n+ Benchmarking on sufficient number of environments that are standard in the literature. The baselines are however insufficient (see below).\n\nCons\n- The major concern with this work is a severe lack of novelty. There are thousands of follow-ups to GANs. In GAIL, it was shown that GANs can be applied to imitation learning. Repeating a GAN work in the IL context consequently should offer significantly more algorithmic/empirical/theoretical insights than the current work.\n- Relatedly, the paper lacks discussion and acknowledgement of related work. The most-closely related work besides GAIL is EB-GAN (Zhao et al.). This work has been cited but not discussed anywhere. Moreover, the references to this work are in the appendix. Theorem 1 essentially  follows directly from Lemma 1 and 2 (proved by Zhao et al) and Theorem 1 of the original GAN paper (Goodfellow et al.). It is unclear if there is anything novel in the analysis or a conclusion that is particularly novel to imitation learning. The subsequent analysis on occupancy measure matching also follows directly from GAIL.\n- On the empirical end, the baselines are very weak in the context of the current work. Just like the literature on GANs has come a long way for improving training stability and mode coverage, the benefits of these GAN++ frameworks have also been shown in the context of imitation learning. For example, the InfoGAIL work by Li et al. shows the benefits of using the WGAN objective for IL and would have been a stronger baseline for empirical comparisons.\n\nIn summary, the work falls in short justifying the merits of EB-GAIL in the context of our existing understanding of imitation learning due to insufficient experimentation and theoretical/algorithmic analysis.\n\nMinor\n- Please use \\citet and \\citep appropriately for improving readability of references.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_3rxe_So8k5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a discriminator-based approach to inverse reinforcement learning (IRL). The discriminator function is trained to attain large values (\"energy\") on trajectories from the current policy and small values on trajectories from an expert policy. The current policy is then improved by using the negative discriminator as a reward signal. The specific discriminator suggested is an autoencoder loss. The authors continue to provide a proof that assuming their discriminator/generator attain a Nash equilibrium, the occupancy measure of the trained policy matches that of the expert policy. They follow up with demonstrating better performance of their approach compared to certain baselines when tested on a number of tasks on Physics simulators.\n\nI believe this paper has potential. Unfortunately I don't think it is publishable in its current form, mainly due to stylistic issues and small errors collectively grave enough to distract from understanding. The language is often unclear due to the unusual grammar, bibliographic references are inserted into the text without any distinction from the surrounding text, where clearer, the language is often a bit too casual for a scientific publication, and there's other small errors that hinder understanding. For example, I believe the above description (discriminator is trained to be small/negative on expert trajectories and large on trajectories from the current policy) is correct, but it's not entirely clear from the paper as e.g. the listing in Algorithm 1 seems to indicate the opposite (comparing the formula in point 4 there with equation (2))? On reading this, I first wondered what happened for negative values of $D$, but equation (4) seems to suggest $D$ is positive at all times? This should be spelled out more properly, and $D$ should be introduced before it is first used in equation (2). I also couldn't find any description of the neural network architecture used in the experiments (the functional form of the final layer would also have cleared up my understanding of what kind of values $D$ produces).\n\nI'm also unclear about the blanket statement \"One fatal weakness for behavioral cloning is that these methods can only learn the actions from the teacher rather than learn the motivation from teachers’ behaviors.\" Recent breakthroughs such as \"AlphaStar: Mastering the Real-Time Strategy Game StarCraft II\" employing a version of behavioral cloning (specifically, \"kickstarting\") seem to suggest otherwise, so this statement should be given some kind of supporting reference. The same goes for the statement \"These methods suffer from the problem of compounding errors (covariate shift)\"; I have heard the term \"covariate shift\" before but I am not truely certain what it _means_.\n\nIn addition to the above, I suspect that the comparison to known baselines is not necessarily the most interesting setting to evaluate the proposed method. However, that is a minor point given the current state of this paper.\n\nThat said, I believe the proposed method is likely to be a good one and will make a worthwhile contribution to scientific progress on IRL. I would encourage the authors to invest the time to polish this submission, ideally also getting it proofread by a diverse set of colleagues. As far as I could tell, the mathematics seems correct and the underlying idea is sufficiently substantial to be published in at a ML conference once it's clarified and the presentational issues are resolved.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XAg6uWZhPL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strong points:\n1. The authors propose an auto-encoder network to replace the discriminator in the GAIL model. The authors also provide some theoretical analysis to show the proposed measure can match the occupancy measure of the expert policy.\n2. The authors conduct experiments on different tasks to show the proposed method can outperform GAIL, BC and GCL.\n\nWeak points:\n1. Some key baselines are missing. There are a few methods that have been proposed and proved to outperform GAIL and GCL. Here, we just show one example. For instance,\n\nFu, J., Luo, K., & Levine, S. (2017). Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248.\nThese methods should be compared.\n\n2. It is better if the authors can explain why using auto encoder as the discriminator can outperform the original GAIL. \n3. The experiments are conducted under relatively small data (usually a few trajectories). Is there a reason? If sufficient data is provided, will the proposed method still perform the best? Is the proposed method specially designed for this small data scenario? The authors may want to explain these questions in more detail.", "extra_scores": {}, "extra_text": {}}, {"review_id": "EvhLQAAeQOq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe paper proposes a new method, called EB-GAIL to deal with the Inverse Reinforcement Learning problem. EB-GAIL is an architecture for training generative adversarial imitation learning, where the discriminator is an autoencoder with the energy being the reconstruction error.\n\nStrengths \nThe experiments are promising since the algorithm performs well in almost all experiments.\n\nWeakness\n\nThe novelty of the article is unclear. Theoretical results are marginal as they are mostly from previous works.\nIt is hard to understand the proofs. \n- What is $R_{\\phi}$ in equation 16? I think that it is the same quantity as in the Generative Adversarial Imitation Learning paper.\n- What is d in equation 17?\n- I think there is an error in the second line of equation 21: a $\\le$ is $\\ge$\n\nThe paper overall is not well written. The Background section is confusing: it starts with an introduction to RL, then there is a subsection related to IRL, and at the end another subsection to imitation learning, where is another explanation of IRL. I would suggest changing this section into RL, Imitation Learning (IRL + BC). \n\nWhat are the main differences between this paper and some new works as Energy-Based Imitation Learning [1] or Strictly Batch Imitation Learning by Energy-based Distribution Matching[2]?\n\n\nMinor comments\n- the citations are all without parentheses\n- the reward is defined as a function R: S $\\rightarrow$ A\n- the discounted future reward has to be in expectation under the initial-state distribution, the transition model and the policy\n- it is missed a citation to behavioural cloning\n- it is missed a comparison with \n\n\n\nFinal comment\n\nThe document is unclear, some sections are confusing. The idea is interesting, but the paper must be edited before it can be published.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to uses an energy-based objective combined with generative adversarial networks for imitation learning. While most reviewers find the work easy to follow and come with theoretical justifications, albeit mostly followed from previous works, and good coverage of experimental results, all of them raised questions regarding the limited novelty and added contribution of the work, and missing more recent baselines. Please consider address these feedback in your future submissions.
Decision: Reject