Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use auto-encoder for inverse reinforcement learning. The main goal of using the auto-encoder, is to use it as the reward function, which takes the auto-encoder reconstruction error to provide reward signals for the agent. The authors claim that such approach provides more informative signal than existing works, especially adversarial imitation learning approaches. Their experiments demonstrate that this method has a better performance over other baselines, and more robust to demonstration noise.
main_review: Pros:
* The use of auto-encoder and reconstruction error for formulating reward functions is novel.
* The auto-encoder is flexible and has many choices in the literature, and I appreciate the authors' effort of trying different variants on that in ablation study.
* The ablation study on distance function for inverse RL loss function is also impressive, which shows the importance of using auto-encoder.
* I appreciate the effort of visualizing the latent representation of the reward model, though with some concerns on the plot. Please see details below.

Some questions and concerns:
* I wonder why using TRPO as the baseline agent, especially when obtaining the expert agent. 
  * It's well-known that the performance of learning a reward model (and also a police) is affected by how good the expert agent is, so I wonder if you have tried some more recent and better agents in the literature. 
  * For example, the PWIL paper shows a great agent with much higher score on Humanoid (nearly 9000) and results in the imitating agent to achieve 7000+ scores, and their agent was trained by D4PG.
* The description of agent and reward model (and/or auto-encoder) architecture is missing. Please add it for completeness and reproducibility.
* As it has been widely discussed in the literature, there are too many misuses of t-SNE due to the lack of misunderstanding hyper-parameters and possibly more details. I generally recommend the authors to provide more details when doing the t-SNE plots. For example, the authors can provide the value of `perplexity` that used for the figure (you can do that in appendix), and possibly plot more than one plots by varying `perplexity`.
  * Also, please provide details on which latent space is plotting, for both auto-encoder and discriminators.
* *Question*: is it correct to say, a well-trained auto-encoder in the loop (e.g, in Fig. 1) can encode expert samples better with lower reconstruction error, while it doesn't capture the features of generated samples and result in higher error?
* *Question*: is it possible to provide theoretical analysis on why the proposed method can minimize the distance between the state-action pair distributions of expert and agent samples, or other related similarity / distance metrics?
summary_of_the_review: The authors provide a good insight on how the adversarial imitation learning algorithms fail. The sensitivity to minor differences between expert and agent samples does impede the learning of agent because it may be pushed to learn how to imitate the details instead of other more important features, and thus less robust to the noise of data. Given that, the authors introduce auto-encoder to handle this issue, and the experiments confirms the claim by showing its better performance over other baselines.

However, I still have some concerns regarding the writing, mostly the lack of important details. The experiments may need to conduct in another round with a more recent and powerful RL agent, which should be easily available now. I also hope to see if the authors can evaluate their approach in more complex and/or real-world environments. The theoretical understanding of the proposed approach is also worth further investigating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes the Auto-Encoding Inverse Reinforcement Learning (AEIRL) method for better imitation learning, especially in the presence of noisy expert demonstrations. The paper’s key insight is that auto-encoders can eliminate the effects of noise from expert demonstrations and provide a more stable reward signal based on the auto-encoder error. The experiments feature extensive analyses across standard benchmarks compared to a large number of baselines. 
main_review: Paper Strengths: 
* AEIRL intuitively provides several benefits. (1) it denoises potentially noisy expert demonstrations, whereas prior inverse RL methods struggle to work with noisy expert demonstrations. (2) the auto-encoder also provides a more informative reward signal whereas the binary classification objective in works like GAIL, can easily result in overfitting. 
* The method seems easy to implement only requiring changing the discriminator to an auto-encoder.
* The main experiments of the paper are thorough. In both non-noisy and noisy demonstrations they compare on 6 standard benchmark tasks against 5 baselines and two versions of their method, all using 5 seeds. 
* Their method performs convincingly better in the non-noisy setting and mostly better in the noisy setting. 
* There are extensive analyses and ablation studies. I especially found the study of the reward signal in Figure 5 to be informative. This adds qualitative evidence to AEIRL learning a more dense reward function. 

Paper Weaknesses: 
* Why does AEIRL perform relatively better on non-noisy expert demonstrations compared to noisy expert demonstrations? The motivation of the method is that it can work well from noisy inputs, yet the method is outperformed in several of the noisy settings and has a large performance drop. This is the opposite of what I would expect. 
* While the reward is intuitive, can it recover a true reward function, or is the reward coupled with the policy like with GAIL? 
* This method is hard to scale to higher dimensional tasks like image-based control. AEIRL must learn a decoder to reconstruct observations. The reconstruction error could be less reliable as a reward signal in these higher dimension tasks since learning the decoder is harder. 
* What happened on the Ant task with noise? Many of the methods went from very positive rewards to negative rewards. 
* The numbers in Tables 1 and 2 are too close together, making some of them hard to read. 
summary_of_the_review: The method is intuitive and performs strongly across in an extensive imitation learning evaluation. Their analyses are also convincing that their. I, therefore, recommend acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presented the auto-encoding inverse reinforcement learning where the reward function is recovered from expert demonstrations in inverse RL problem. The reward was represented as a surrogate function which used the auto-encoder with the metric of reconstruction error.
main_review: Pros:
1. Auto-encoder based reward function was proposed to denoise the expert data through the encoding-decoding process.
2. The experiments on the MoJoCo showed that the proposed method outperformed state-of-the-art imitation learning on both clean and noisy expert demonstrations.
3. Ablation studies showed that different reward functions and loss functions also received good performance.

Cons:
1. Theoretical justification to support the idea of auto-encoder based surrogate reward function was missing.
2. This method was to minimize the Wasserstein distance between the state action distribution of the policy and the expert demonstrations. But, there was not explanation on Lipschitz constraint or weight clipping.
3. This paper mentioned WGAIL (Arjovsky et al., 2017) many times, but Wasserstein GAN did not use reinforcement learning or imitation learning.
4. Need theoretical discussion about using the variational auto-encoder based reward function and the Jensen-Shannon divergence to replace auto-encoder and Wasserstein distance, respectively.
5. Lack of suitable experimental results to support that auto-encoder based reward function was more informative and denser than the discriminator based one.
summary_of_the_review: A new auto-encoder based reward function was presented. Theoretical and experimental justifications were insufficient.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present an architecture that utilizes autoencoding as an approach to inverse reinforcement learning.  They present a mathematical formalism, pseudocode, and some empirical analyses on MuJoCo tasks.  
main_review: The paper is an interesting approach to IRL, using minimax games and autoencoding to enhance training.  While the approach appears novel (IRL is not my main area), the experimental results, while generally positive, are a little mixed in terms of which approach is preferred (especially VAE vs non-VAE). 

The empirical results on noise-free data are impressive (as are some of the noisy results), but I would have liked to have seen more explanation on why their VAE-based approach outperforms non-VAE on some tasks but not others. I.e., which one should be chosen based on the task? 

I don't understand the significance of Figure 4.  In what way are your approach's latent representations "more indistinguishable" from those of the others?  In particular, I don't see any real difference on the non-noisy data.  Can you quantify this somehow? 

I read the authors' response, but it didn't compel me to change my score. 
summary_of_the_review: The paper is an interesting approach to IRL, using minimax games and autoencoding to enhance training.  While the approach appears novel (IRL is not my main area), the experimental results, while generally positive, are a little mixed in terms of which approach is preferred (especially VAE vs non-VAE).
review_title: null

Meta-review: This work addresses the problem of learning representations from noisy expert demonstrations in in adversarial imitation learning. The authors build on top of GAIL, which utilizes a discriminator to model a "pseudo"-reward from demonstrations. In this work, the discriminator is replaced with an auto-encoder. The authors hypothesis is that using an auto-encoder helps in 2 ways: 1) denoising expert trajectories for more "robust" learning; 2) using the reconstruction error (instead of binary classification loss) to distinguis experts from samples provides more informative signal for reward learning. 

**Strengths**
on a global perspective this work is well motivated
a novel algorithmic variant of GAIL is proposed 
thorough experimental evaluation

**weaknesses**
The manuscript doesn't clearly distinguish between adversarial imitation learning algorithms (like GAIL) and "true" inverse reinforcement learning algorithms. This makes it unclear what the real goal of the proposed method is. The ultimate goal of adversarial IL is to learn a policy (by inferring a pseudo-reward at "train" time which is then never used again), while the primary goal of IRL is to learn a reward function at train time, which can then be used at test time. The manuscript motivates the algorithm by saying it will have a more informative signal for learning reward functions, but the algorithm itself is an adversarial IL algorithm which primary goal is to learn a policy from demonstrations. Overall, makes the evaluation and analysis confusing. Ideally, the authors would have focussed on the question "Does the reconstruction error lead to better policies?" (through better pseudo-reward modeling) - or would have extended an IRL method.

Second, the motivation is that the autoencoder helps with more "robust" learning, but it's unclear to me that the evaluation really shows that learning is more robust (also because "robustness" is not clearly defined)

The experimental evaluation is a bit of a mixed bag, and it's a unclear why the new algorithm performs better on non-noisy data (when compared to baselines), but not less so on the noisy data.

**Summary**
Overall, this work provides a promising direction, however in it's current form the manuscript is not yet ready for publication.
Decision: Reject