Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a maximum likelihood approach for IRL. Theoretical results show that the algorithm converges in finite time and that for linear rewards that the ML-IRL problem is dual with MaxEnt-IRL and that there is strong duality. Empirical results show that ML-IRL outperforms state-of-the-art IRL algorithms across several standard benchmarks.
strengths_and_weaknesses: +The strong duality theorem appears novel and draws an interesting connection between ML-IRL and MaxEnt-IRL

+The authors prove the first finite guarantees for IRL with nonlinear reward functions

+The empirical results are promising and improve upon SOA

-Some of the claims of novelty seem to be standard practice and are used in prior work.

-It is unclear why an ML approach to IRL should perform better than other approaches. The authors have nice theory, but in practice the implemented algorithm seems very similar to prior work.

-It is unclear how statistically significant the empirical results are with only 3 seeds and no confidence intervals.
questions: The authors are missing related work that also seeks to make IRL more tractable.

Wang, Ruohan, et al. "Random expert distillation: Imitation learning via expert policy support estimation." International Conference on Machine Learning. PMLR, 2019.
and
Brown, Daniel S., Wonjoon Goo, and Scott Niekum. "Better-than-demonstrator imitation learning via automatically-ranked demonstrations." Conference on robot learning. PMLR, 2020.
Propose approaches that do not require interleaving reward learning and policy learning.
Recent work
Barde, Paul, et al. "Adversarial soft advantage fitting: Imitation learning without policy optimization." Advances in Neural Information Processing Systems 33 (2020): 12334-12344.
proposes an adversarial IL approach that does not require any RL.

One of the stated contributions is to implement  single-loop approach. This doesn't seem like a contribution since most Adversarial IRL methods do the same thing. This idea is well studied, going back to GAIL and Guided cost learning, and probably earlier. I do not think the authors can claim this as novel. Furthermore, the two-timescale approach is common in GANs and GAIL-like algorithms. I don't think this can be claimed as a contribution. 

Equation 3: The theta parameterizes r which informs argmax to get pi. This is confusing with line 119 where the policy is parameterized by theta. Policies are not usually parameterized by rewards.

Line 155: is supposed to be an example of solving a hard optimization problem in the inner loop but it refers to IQ-Learn which does was previously noted to be computationally efficient because it avoids this inner loop mdp solver.

Lemma 1: How is his different from the guided cost learning update. In guided cost learning the partition function is estimated with onpolicy rollouts so it appears quite similar.

Line 188: Why only sample one demo and one policy rollout? Doesn't this have very high variance?

Allowing the algorithm to work with state only demonstrations is nice, but many other algorithms already do this. E.g. 
Brown, Daniel S., Wonjoon Goo, and Scott Niekum. "Better-than-demonstrator imitation learning via automatically-ranked demonstrations." Conference on robot learning. PMLR, 2020.
Torabi, Faraz, Garrett Warnell, and Peter Stone. "Generative adversarial imitation from observation." arXiv preprint arXiv:1807.06158 (2018).
As well as all of the classical IRL algorithms. 
It would be good to motivate more why this is a contribution.

limitations: A discussion of limitations are lacking in the main body of the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper formalizes IRL as a entropy-regularized maximum likelihood problem. The authors show that this problem is dual to maximum entropy IRL, and then formulates this problem as a bi-level optimization. They propose an algorithm that iterates between single steps of evaluation and policy improvement (in contrast to most max entropy IRL methods that have to solve RL problems in an inner loop). For this algorithm, the authors provide a _finite time_ convergence guarantee under some regularity conditions. Finally, they provide an empirical evaluation on standard MuJoCo task, demonstrating that their algorithm outperforms alternative IRL approaches.
strengths_and_weaknesses: **Strengths**
- The paper provides the first finite-time convergence analysis without assuming linear rewards, as far as I can tell. The assumptions of the main theoretical results seem pretty reasonable; this is a solid theoretical contribution.
- The duality between MaxEnt IRL and the proposed maximum likelihood formulation might be interesting to the IRL community and could motivate interesting follow-up work.
- The authors provide a good experimental evaluation that covers most relevant baselines and shows that their algorithm can perform well empirically.
- The authors discuss the novelty of their work and provide particularly appropriate comparisons to results in prior work at multiple point, e.g., for their ML formulation, the proposed algorithm, and the main theoretical result.


**Weaknesses**
- The duality with MaxEnt IRL is not particularly novel and has been observed before in other context (as noted by the authors).
- The results in Table 1 don't show very large differences, and it is hard to evaluate if these are significant differences, given that only 3 random seeds were run.
- The experimental evaluation is limited to MuJoCo locomotion tasks which can sometimes be too simple to draw strong conclusions.
- Overall, the writing could be clearer at times, and the structure of the paper could be improved. For example, the Introduction spends a lot of time discussion prior work before stating the contributions of this paper, which is sub-optimal.
questions: Can you provide standard errors for the results in Table 1?

Why does the state-action version of the algorithm perform worse than the state-only algorithm in the half-cheetah environment?


**Typos:**
 - "inherent nested" -> "inherently nested" (line 4)
- "Mujoco" -> "MuJoCo" (line 19)
- "computational efficient" -> "computationally efficient" (line 53)
limitations: I did not find any discussion of potential broader impact of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel single-loop IRL algorithm to reduce the computational burden of the previous IRL framework. The authors prove that the solution of the proposed framework is equivalent to the maximum entropy IRL framework with a  linearly parameterized reward function. They also provide a convergence guarantee of the proposed algorithm. The experiment result seems promising. 
strengths_and_weaknesses: Strengths:

- The paper is well written. 
- The problem studied in this paper is important and the proposed method is novel.
- The authors provide theoretical guarantees for the proposed framework.

Weaknesses:

- There may need more literature review on the MaxEnt-IRL.
- Although the ML-IRL framework could reduce the computational complexity, it is not obvious why the proposed ML-IRL framework works for the transfer learning tasks.

questions: - Could the authors discuss the MaxEnt-IRL literature?
- Could the authors explain why the ML-IRL framework works for the transfer learning tasks?
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed a new formulation of IRL based on maximum likelihood, which is equivalent to MaxEnt when the reward function is linear.  By leveraging this formulation, the authors propose a new computationally efficient gradient-based iterative algorithm that does not require solving MDP in each iteration. The authors further prove a non-asymptotically rate of how fast the algorithm converges to a stationary point, which is the first non-asymptotic convergence result for ILR with nonlinear reward parameterization. Finally, the authors conduct extensive experiments that show that the proposed algorithm outperforms several state-of-the-art IRL algorithms. 
strengths_and_weaknesses: ### Strength: 

The authors claim that Thm 2 is the first non-asymptotic convergence result for ILR with nonlinear reward parameterization. I believe this is true as a recent paper [1] from ICML 2021 proves a non-asymptotic rate for a gradient-based method with linear reward parameterization (although that paper’s proof does not seem to have the assumption the value function can be accurately estimated).
 
### Weakness: 
 I think the contribution above is important to the IRL community.  However, other contributions of this paper may not be as significant as this one. 

1. The new formulation of IRL. This maximum likelihood formulation is known [2]. The proof seems to be similar to Thm 3 in [3]. 
2. The algorithm. The authors mentioned that the algorithm enjoys computational efficiency and is capable of reward transferring. However, GAIL-type of algorithms that use state-dependent rewards in the discriminator [4] and scalable MaxEnt IRL algorithms [5] are able to achieve those benefits as well. 
3. The experiments. I like the experiments on reward transferring. However, the main goal of proposing the new algorithm and theory is to reduce computation burden, but the experiments focus on the quality of the learned policy / reward after convergence. It’s unclear from the experiments whether the new algorithm is indeed faster. For example, an interesting experiment would be a comparison with MaxEnt IRL as in theory the proposed method is able to converge to a stationary point that’s the same as MaxEnt IRL, but faster.

Furthermore, the clarity of the paper can be improved. Line 118 to 135 is rather confusing and only until the second reading I was able to understand the math. In Line 119,\pi_\theta is parameterized by \theta. But in Line 123, \theta is used to parameterize the reward function. 

[1] Kamoutsi, Angeliki, Goran Banjac, and John Lygeros. "Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrations." International Conference on Machine Learning. PMLR, 2021.

[2] Jain, Vinamra, Prashant Doshi, and Bikramjit Banerjee. "Model-free IRL using maximum likelihood estimation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.

[3] Ziebart, Brian D., J. Andrew Bagnell, and Anind K. Dey. "The principle of maximum causal entropy for estimating interacting processes." IEEE Transactions on Information Theory 59.4 (2013): 1966-1980.

[4] Torabi, Faraz, Garrett Warnell, and Peter Stone. "Generative adversarial imitation from observation." arXiv preprint arXiv:1807.06158 (2018).

[5] Finn, Chelsea, Sergey Levine, and Pieter Abbeel. "Guided cost learning: Deep inverse optimal control via policy optimization." International conference on machine learning. PMLR, 2016.

questions: In the algorithm, it’s assumed that the soft-Q function can be accurately estimated. Can the approximation  in soft-Q function be considered in the theorem as well?

Interestingly, the gradient in Eq.(6) is the same as the gradient to r with fixed pi in Eq. (2) (MaxEnt). And the inner optimization in Eq. (2) is finding the optimal policy given r which is the same as solving Eq. (3a). Therefore, the proposed algorithm can be considered as solving Eq. (2) directly without the ML-IRL formulation. Would the authors provide some explanation on why formulating IRL in ML-IRL is fundamental to the convergence proof?

In the text the authors try to distinguish between MaxEnt IRL and adversarial IRL (e.g. Line 30 - 44). I wonder why that's the case as these two have been shown to be closely related [6, 7].

[6] Finn, Chelsea, et al. "A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models." arXiv preprint arXiv:1611.03852 (2016).

[7] Swamy, Gokul, et al. "Of moments and matching: A game-theoretic framework for closing the imitation gap." International Conference on Machine Learning. PMLR, 2021.

limitations: Yes
review_title: null

Meta-review: metareview: This paper presents a new single loop IRL algorithm that avoids the typical policy/reward optimization loop in IRL algorithms, without sacrificing the accuracy of the learned reward function. This is achieved through the use of stochastic gradients of the likelihood function. The proposed algorithm is proved to converge to a stationary solution with a finite-time guarantee. Experiments on some problems in MuJoCo show that the proposed algorithm can outperform existing solutions. The reviewers all agree that the paper is well-written, the algorithm is sufficiently new, and the experiments are compelling. There are some concerns that experimental evaluation is limited to MuJoCo locomotion tasks which can sometimes be too simple to draw strong conclusions.
confidence: Less certain
Decision: Accept