Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a learning framework for imitation learning (IL) that uses an energy-based objective for generative adversarial imitation learning (GAIL).

Pros
+ Clarity. The paper is easy to understand and follow.
+ Theoretical analysis is sound (not novel however -- see below).
+ Benchmarking on sufficient number of environments that are standard in the literature. The baselines are however insufficient (see below).

Cons
- The major concern with this work is a severe lack of novelty. There are thousands of follow-ups to GANs. In GAIL, it was shown that GANs can be applied to imitation learning. Repeating a GAN work in the IL context consequently should offer significantly more algorithmic/empirical/theoretical insights than the current work.
- Relatedly, the paper lacks discussion and acknowledgement of related work. The most-closely related work besides GAIL is EB-GAN (Zhao et al.). This work has been cited but not discussed anywhere. Moreover, the references to this work are in the appendix. Theorem 1 essentially  follows directly from Lemma 1 and 2 (proved by Zhao et al) and Theorem 1 of the original GAN paper (Goodfellow et al.). It is unclear if there is anything novel in the analysis or a conclusion that is particularly novel to imitation learning. The subsequent analysis on occupancy measure matching also follows directly from GAIL.
- On the empirical end, the baselines are very weak in the context of the current work. Just like the literature on GANs has come a long way for improving training stability and mode coverage, the benefits of these GAN++ frameworks have also been shown in the context of imitation learning. For example, the InfoGAIL work by Li et al. shows the benefits of using the WGAN objective for IL and would have been a stronger baseline for empirical comparisons.

In summary, the work falls in short justifying the merits of EB-GAIL in the context of our existing understanding of imitation learning due to insufficient experimentation and theoretical/algorithmic analysis.

Minor
- Please use \citet and \citep appropriately for improving readability of references.
review_title: Lack of novelty and rigor

final_score: rating: 5: Marginally below acceptance threshold
review_content: Strong points:
1. The authors propose an auto-encoder network to replace the discriminator in the GAIL model. The authors also provide some theoretical analysis to show the proposed measure can match the occupancy measure of the expert policy.
2. The authors conduct experiments on different tasks to show the proposed method can outperform GAIL, BC and GCL.

Weak points:
1. Some key baselines are missing. There are a few methods that have been proposed and proved to outperform GAIL and GCL. Here, we just show one example. For instance,

Fu, J., Luo, K., & Levine, S. (2017). Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248.
These methods should be compared.

2. It is better if the authors can explain why using auto encoder as the discriminator can outperform the original GAIL. 
3. The experiments are conducted under relatively small data (usually a few trajectories). Is there a reason? If sufficient data is provided, will the proposed method still perform the best? Is the proposed method specially designed for this small data scenario? The authors may want to explain these questions in more detail.
review_title: More experiments are needed

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a discriminator-based approach to inverse reinforcement learning (IRL). The discriminator function is trained to attain large values ("energy") on trajectories from the current policy and small values on trajectories from an expert policy. The current policy is then improved by using the negative discriminator as a reward signal. The specific discriminator suggested is an autoencoder loss. The authors continue to provide a proof that assuming their discriminator/generator attain a Nash equilibrium, the occupancy measure of the trained policy matches that of the expert policy. They follow up with demonstrating better performance of their approach compared to certain baselines when tested on a number of tasks on Physics simulators.

I believe this paper has potential. Unfortunately I don't think it is publishable in its current form, mainly due to stylistic issues and small errors collectively grave enough to distract from understanding. The language is often unclear due to the unusual grammar, bibliographic references are inserted into the text without any distinction from the surrounding text, where clearer, the language is often a bit too casual for a scientific publication, and there's other small errors that hinder understanding. For example, I believe the above description (discriminator is trained to be small/negative on expert trajectories and large on trajectories from the current policy) is correct, but it's not entirely clear from the paper as e.g. the listing in Algorithm 1 seems to indicate the opposite (comparing the formula in point 4 there with equation (2))? On reading this, I first wondered what happened for negative values of $D$, but equation (4) seems to suggest $D$ is positive at all times? This should be spelled out more properly, and $D$ should be introduced before it is first used in equation (2). I also couldn't find any description of the neural network architecture used in the experiments (the functional form of the final layer would also have cleared up my understanding of what kind of values $D$ produces).

I'm also unclear about the blanket statement "One fatal weakness for behavioral cloning is that these methods can only learn the actions from the teacher rather than learn the motivation from teachers’ behaviors." Recent breakthroughs such as "AlphaStar: Mastering the Real-Time Strategy Game StarCraft II" employing a version of behavioral cloning (specifically, "kickstarting") seem to suggest otherwise, so this statement should be given some kind of supporting reference. The same goes for the statement "These methods suffer from the problem of compounding errors (covariate shift)"; I have heard the term "covariate shift" before but I am not truely certain what it _means_.

In addition to the above, I suspect that the comparison to known baselines is not necessarily the most interesting setting to evaluate the proposed method. However, that is a minor point given the current state of this paper.

That said, I believe the proposed method is likely to be a good one and will make a worthwhile contribution to scientific progress on IRL. I would encourage the authors to invest the time to polish this submission, ideally also getting it proofread by a diverse set of colleagues. As far as I could tell, the mathematics seems correct and the underlying idea is sufficiently substantial to be published in at a ML conference once it's clarified and the presentational issues are resolved.
review_title: Interesting result suffering from unclear presentation

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
The paper proposes a new method, called EB-GAIL to deal with the Inverse Reinforcement Learning problem. EB-GAIL is an architecture for training generative adversarial imitation learning, where the discriminator is an autoencoder with the energy being the reconstruction error.

Strengths 
The experiments are promising since the algorithm performs well in almost all experiments.

Weakness

The novelty of the article is unclear. Theoretical results are marginal as they are mostly from previous works.
It is hard to understand the proofs. 
- What is $R_{\phi}$ in equation 16? I think that it is the same quantity as in the Generative Adversarial Imitation Learning paper.
- What is d in equation 17?
- I think there is an error in the second line of equation 21: a $\le$ is $\ge$

The paper overall is not well written. The Background section is confusing: it starts with an introduction to RL, then there is a subsection related to IRL, and at the end another subsection to imitation learning, where is another explanation of IRL. I would suggest changing this section into RL, Imitation Learning (IRL + BC). 

What are the main differences between this paper and some new works as Energy-Based Imitation Learning [1] or Strictly Batch Imitation Learning by Energy-based Distribution Matching[2]?


Minor comments
- the citations are all without parentheses
- the reward is defined as a function R: S $\rightarrow$ A
- the discounted future reward has to be in expectation under the initial-state distribution, the transition model and the policy
- it is missed a citation to behavioural cloning
- it is missed a comparison with 



Final comment

The document is unclear, some sections are confusing. The idea is interesting, but the paper must be edited before it can be published.
review_title: Review for Learning from Demonstrations with Energy based Generative Adversarial Imitation Learning 

Meta-review: This work proposes to uses an energy-based objective combined with generative adversarial networks for imitation learning. While most reviewers find the work easy to follow and come with theoretical justifications, albeit mostly followed from previous works, and good coverage of experimental results, all of them raised questions regarding the limited novelty and added contribution of the work, and missing more recent baselines. Please consider address these feedback in your future submissions.
Decision: Reject