Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a large-scale empirical study for implementation choices on adversarial imitation learning. The contributions of this work can be summarized as follows:

a. implement a high-configurable adversarial imitation learning algorithm and conduct a large-scale empirical study

b. from the empirical study, the authors argue that they find two important results:

1. a choice of discriminator regularizers is not critical 

2. artificial demonstrations are not a good proxy for human demonstrations
main_review: This paper presents interesting results about algorithmic choices on adversarial imitation learning by conducting rigorous large-scale empirical studies. The paper is clearly written and findings, such as discriminator regularization techniques (e.g. gradient penalty, .. ) are not as crucial as we considered in literature, are interesting. Yet, I have following concerns and comments about this paper:

1. The authors argue that synthetic demonstrations are not a good proxy for human data. 
But I concern this argument is too obvious to be one of main findings of this empirical study. 

2. The authors may argue that unexpected human behaviors make significant differences in optimal hyperparameter settings in Figure 4. But it is not clear to me where those differences come from. Isn’t it simply because of the difference in performance? According to Table 2 in Appendix F, performances of door-expert-v0 and door-human-v0 are significantly different (2882 and 796 respectively), hammer-expert-v0 and hammer-human-v0, too (12794 and 3071 respectively).

3. Hyperparameter selections should be more carefully considered especially for human data. The agent getting higher performance on the evaluation environment is not always better, because the goal of imitation learning is imitating the given expert demonstration.

4. Since expert demonstrations define each problem, varying the quality of expert datasets could be a valuable extension of this empirical study. For example, D4RL dataset contains different versions of each task such as hopper-expert-v0 and hopper-medium-v0, etc. 
. 

Although I have some concerns on this paper, lessons from this work can be helpful to future imitation learning researchers and practitioners. Hence, I vote for acceptance to this paper.

limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents an extensive study on hyperparameters and design choices for adversarial imitation learning, including the choice of divergence (reward function), RL algorithm and whether marginals $p(s,a)$, $p(s)$, $p(s,s')$ or $p(s,a,s')$ are to be matched. Evaluations are conducted on MuJoCo and Adroit environments --- the latter experiments also made use of human demonstrations.
The main findings are that demonstrations collected by an RL agent are not good proxies for human demonstrations and that more standard discriminator regularizers like dropout perform similarly well as gradient penalty.

main_review: 
Significance
----------------
Studies like these can be very helpful for researchers and practitioners. The paper was interesting to read, but afterall I did not get any major new insights. The fact that the differences between human demonstrations and demonstrations from an RL agent affect the performance of the algorithm and the choice of parameters was to be exected. But of course, learning about which parameters might be preferable in a human-demonstration setting is still useful, even though I think that the number of experiments is too small to conclude that JS divergence performs better than KL for example.

Originality
--------------
I don't think that a similar study on adversarial imitation learning exists, so in that sense the work is original. However, it does not present any intriguing ideas and also the interpretations of the results seem a bit shallow. 

Quality
-----------
Overall, the study was soundly conducted. However, I do have a few criticisms:

* All experiments seem to use a fixed number of demonstrations. That would be a major oversight, given that some algorithms that work well for many demonstrations (for example behavioral cloning) might not work very well for few demonstrations. Evaluating how design choices relate to the number of demonstrations would have been very interesting. On a minor note, also evaluations w.r.t. to noisy / mixed data might have produced more meaningful insights.

* The performance is evaluated based on the agent's return. I think it would be important to also evaluate the performance w.r.t. the different divergences, since these are the actual objectives of the algorithms. It is not at all clear, whether lower divergences always achieve higher rewards.

* The authors say that they plan to release the code, however, I am not overly optimistic since not even the reviewers have been given access to the code. Also, I would expect a study to publish the underlying data, which would be very useful, for example, to conduct the missing evaluations w.r.t. the different divergences (maybe even MMD).

Clarity
--------
The presentation of the results is good and the paper is neatly structured.
limitations_and_societal_impact: Limitations and potential negative societal impact were adequately addressed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The work conducts a detailed empirical analysis of a range of design decisions in the domain of adversarial imitation learning. The imitation learning framework is sufficiently general to encompass prior work like GAIL, AIRL, various extensions to these algorithms (e.g. regularizers), different RL algorithms used for the generator training. These decisions are tested in a comprehensive hyperparameter sweep involving over 500k trained agents across eight continuous control tasks. The results include many observations that will be useful to practitioners, such as the importance of observation normalization. It also in some cases casts doubt on received wisdom, such as the various AIL-specific methods for discriminator regularization, which seem to be little better than standard SL approaches like dropout.
main_review: With the proliferation of adversarial imitation learning algorithms and the sadly often not-so-careful evaluation in these papers, testing what actually works is timely and important. The paper is clearly written and the experiments are carefully conducted. I expect practitioners to find this a helpful aid to tuning their imitation learning algorithm. For those active in AIL research, this paper serves both as a useful wake-up call (with, in effect, several negative results contradicting prior work) and as a guide to areas where further research is needed (e.g. more human data benchmarks).

Although I am overall excited by this paper, I did find some of the experiments hard to interpret and potentially confounded. In particular:
  1. The performance measure that approximates area under the learning curve differs from the metric used in much prior work, of episode return at convergence. I think both of these metrics could be appropriate depending on the context. Perhaps you could also report episode return at the end of the training, at least for the experiments where this most significantly changes results? In particular, this would allow the reader to establish whether seemingly negative results about prior work are due to the differing performance measure (perhaps prior methods are better at convergence but worse during early training) or a true replication failure.
  2. I am glad that you highlighted the issue that variable-length episodes cause in evaluation, an issue that has plagued this field. I was also pleased to see that inclusion of the absorbing state does seem to help with this -- at least the difference between reward function choice in Figure 72 becomes a lot less significant with an absorbing state. But how confident can we be the absorbing state has really de-biased this? I would have favored removing the termination condition from these environments, as discussed in Appendix A of [Christiano et al (2017)](https://arxiv.org/pdf/1706.03741.pdf) and followed in much of the other work in this area. I understand re-running the experiments may not be practical -- perhaps consider a break-down of environments with fixed-length episodes vs variable-horizon episodes given your test suite contains a mix of both?
  3. There are often correlations between the hyperparameters. I think the paper mostly does a good job highlighting these limitations in context, but these caveats have tended to be dropped in summaries. In particular, I think the conclusion that synthetic demos may not predict human demos is significantly undermined by the human demos in question being quite strange, in the sense that the agent *cannot* possibly replicate the human demos. I think you either need to argue that this is a normal or unavoidable situation when working with human data, or modify the environment to allow the agent to transition to an absorbing state early, or at least add some significant hedging to the conclusion.

I also have one question: you write that “Following prior work [14, 31, 46], we subsample expert demonstrations by only using every 20th state-action pair to make the tasks harder.” However, I do not see any mention of this in the GAIL paper [14]. Can you refer me to where in [14] this is discussed?

As said, the paper was generally well written, but I noticed a few areas that were confusing:
- Line 29: “and never these changes were all compared together” – ungrammatical. Perhaps “and these changes have never been compared simultaneously”?
- Line 30: “the low-level”->”low-level”. “might be not even” -> “might not even be”.
- Line 220: “As noticed in prior work” – this made me expect you were going to confirm that result but you actually find it doesn't deplicate, consider rephrasing? E.g. “Prior work has claimed the initialization ...”
- Line 223: “An overfitting” -> “An overfit”?
- Some capitalization issues in references, e.g. "Openai gym"->"OpenAI Gym".

# Update after rebuttal

I thank the authors for their response. It is nice to know the results mostly do not change between final performance rather than AUC, but I agree that including a brief analysis of this in the paper would be worthwhile. 

I continue to have some concerns about the results being confounded by variable-length episodes and the quirks in the human data, but I do sympathize that given the scale of the experiments and the already significant length of the paper, these issues are difficult to correct. I'd suggest the authors consider adding a more explicit discussion of these limitations to the manuscript, however.
limitations_and_societal_impact: Limitations discussed in context but not adequately pushed through to the summary. No real discussion of societal impact of work which is a missed opportunity but I expect it to be robustly positive.
review_title: null

Meta-review: This paper presents an extensive experimental study on implementation choices in adversarial imitation learning. All reviewers were leaning towards acceptance of the paper since many observations made in the paper could be useful to practitioners. I am also recommending acceptance of the paper. 

Decision: Accept (Poster)