Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces a new IRL framework, SOLO-IRL, that learns a reward function using only expert trajectories. This has the benefit of being trained in an offline manner, which speeds up the training process. SOLO-IRL builds on top of the work of Uchibe, 2018 and exploits the fact that a discriminator can replace the binary classification used in LogReg-IRL. They improve on LogReg-IRL by overcoming the difficulty of selecting an appropriate baseline trajectories by proposing to use adversarial one-class classification (Sabokrou et al., 2018). The authors empirically demonstrate their results on the CartPole and BipedalWalker tasks, and show superior performance over LogReg-IRL.
main_review: ### Strengths
This work demonstrates an interesting and insightful extension to IRL frameworks. The authors created an efficient IRL method that can be trained in a completely offline manner. By replacing the binary classification with adversarial one-class classifiction from LogReg-IRL, they remove the difficulty of picking a good baseline trajectory that LogReg-IRL requires. Furthermore, this idea of adversarially learned one-class classifier draws on the idea of anomaly detection problem to learn a good discriminator; this can potentially have benefits for a stronger baseline trajectory during training. Indeed, this was demonstrated in the results in the BipedalWalker task, where the random policy could not provide an appropriate baseline trajectory for the expert trajectory to progress to the goal.

### Weaknesses
The main weakness of this work is the sparsity of the experimental section. A baseline that should be compared to is behavioral cloning (BC). The setting of this work is under an offline setting using only expert trajectories which is the same for BC; thus, I am curious if BC performs on-par with SOLO-IRL? The authors do not explicitly mention an advantage of their method over simple BC (although I would imagine the reward function may be an advantage), and thus I would like to see it's performance compared to BC. The authors demonstrate that the reward function is better shaped than LogReg-IRL, but they do not showcase how it can be used. It would be interesting to see if the learned reward can be used to transfer to like tasks? 

One additional concern is the sparsity in tasks and baselines in the experimental section. It would be interesting to see more than 1 baseline and tasks other than BipedalWalker/CartPole (since CartPole is quite simple). The authors also mentioned that adjusting the noise in SOLO-IRL is difficult. It would be nice to see an ablations section to understand how sensitive/hard to tune that is, as compared to selecting baseline trajectories?

### Miscellaneous
A more extensive background of the IRL field would be nice. A few citations to include:
 - MaxEnt-IRL, Ziebart et al. 2008
 - Guided Cost Learning, Finn et al. 2016 
 - Generative adversarial imitation learning, Ho et al. 2016
 - Behavioral Cloning from Observation, Torabi et al. 2018
summary_of_the_review: The authors introduce an interesting, new framework for IRL that can learn a reward function in an offline manner. Furthermore, they draw insights from anomaly detection problem and improve upon the LogReg-IRL. They demonstrate, empirically, that their method can learn a better reward function and achieve higher performance on two OpenAI Gym tasks. However, the experimental section for this work is very sparse; if the authors can address the concerns listed above, in particular adding BC as a baseline, I would be willing to increase my rating.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an adversarial inverse reinforcement learning algorithm that learns purely from expert demonstrations, and does not require any online interaction with the environment or a dataset of unlabeled interactions with the environment. The key idea is to synthesize negative examples (i.e., examples of non-expert behavior) using a denoising autoencoder trained on the positive examples (i.e., expert demonstrations). Experiments on the BipedalWalker simulated locomotion task show that the proposed method learns a reward function such that an RL agent trained to maximize the learned rewards achieves higher true rewards than a prior method.
main_review: Overall, this is an interesting paper, but I’m concerned that the experiments don’t compare to prior methods that also learn a reward function purely from expert demonstrations without any unlabeled data:
 - [Random Expert Distillation](https://arxiv.org/abs/1905.06750)
 - [Disagreement-Regularized Imitation Learning](https://openreview.net/forum?id=rkgbYyHtwB)
 - [O-NAIL](https://arxiv.org/abs/2008.03525)

I’m also concerned that the experiments only show that the proposed method outperforms prior work on one task: BipedalWalker. I would consider raising my score if the paper included experiments with different tasks and compared to at least one of the prior methods listed above.

Update
-----
Thank you to the authors for adding the experimental comparisons to BC and RED on the Hopper and Walker2D simulated locomotion tasks. Unfortunately, it seems that the BC and RED baselines substantially outperform the proposed method on those two tasks, so I will keep my original score.

summary_of_the_review: Lacking comparisons to relevant prior methods and evaluations on diverse tasks
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a state-only offline ILR algorithm (learning reward function),  SOLO-IRL, by reducing IRL to adversarial one-class classification. Compared to most existing ILR algorithms, the proposed algorithm is more efficient and requires fewer assumptions. It does not require solving RL problems in the inner loop and does not require ranked expert trajectories or assumptions on trajectories generated by uncontrolled transitions probabilities. The authors show the algorithm learns reasonable reward on two simulated control tasks and significantly outperforms the LogReg-IRL algorithm that it extends. 

main_review: ### Strengths ### 
- The problem that the paper tends to address is challenging and is very relevant to the ICLR community. 
- The algorithm proposed seems simple and can be implemented using the off-the-self adversarial one-class algorithms, although tuning noise level may be required as the authors noted. 
- The paper is overall clearly written and easy to follow.

### Concerns ###
- The approach that the authors propose is very interesting. However, I am not sure whether the analysis of LogReg-IRL in Section 2.3 carries over to the proposed method. The analysis of LogReg-IRL seems to only apply when learning classifiers that distinguish between trajectories generated from uncontrolled and expert-controlled transition probabilities. It’s not shown in the paper what reward and value functions are learned by learning classifiers that distinguish between expert-controlled and noise-corrupted-export-controlled data. Therefore, theoretical analysis of the proposed method is lacking.
- Do the reward function and transition probability in the experiments satisfy the requirements of LMDP?
- The experiments need more work. 
   - Some ablation analysis would help to understand what change contributes to the improvement on the BipedalWalker environment among the addition of generator, weight decay, and changed cross-entropy. 
  - LogReg-IRL is affected by baseline trajectories in theory. It would be helpful to compare to related works such as D-REX (Brown et. al. 2019) (an extension of T-REX) that proposes a similar idea of injecting noise to demonstrated trajectories. 
  - Due to the randomness in network initialization and trajectory generation, experiments with more seeds are helpful. More tasks would also better demonstrate the performance of the algorithm. 
  - I appreciated that the authors are honest about the difficulty in turning the noise levels. Could you give some intuition about this? Moreover, what are the state dimensions in Table 5 in Page 13?
  - It’s exciting to see the visualization of the rewards that SOLO-IRL learned. It will be informative to plot the true reward function too. 

 ### Other comments / questions ###
- Eq. (4) (rewriting Eq. (3) using the definition of KL divergence) does not seem correct. And Eq. (5) does not align with the equation in the LogReg-IRL paper. 
- Is assuming the states are finite necessary in e.g. Eq. (4) or it’s just for convenience?

### Post-rebuttal comments ###
Thank the authors very much for responding to my questions and adding more experiments in a short amount of time. However, the results do not seem very promising and I am still concerned with the theoretical grounding of the proposed algorithm. Therefore, I would like to keep my score. 

summary_of_the_review: Because of the concerns discussed in the Main Review tab, I am leaning towards rejecting the paper for now.
review_title: null

Meta-review: This paper studies the problem of inverse reinforcement learning by relying on only demonstrations and no interaction (like imitation learning). The reviewers liked the premise but had major concerns with evaluation and baselines. The paper initially received reviews tending to reject. One of the questions was about missing behavior cloning baseline which the authors added in rebuttal. But the BC baseline seems to be really competitive (in fact, better in 3 out of 4 envs) as compared to the proposed approach. In conclusion, all reviewers still believed that their concerns regarding insufficient evidence for justifying approach and missing comparisons to other prior work still stand. AC agrees with the reviewers' consensus that the paper is not yet ready for acceptance.
Decision: Reject