Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to get around the typical two-stage inverse RL pipeline of (1) learn a reward model and (2) optimize it, by encapsulating both rewards and dynamics in a single Q function, learned by imitation. This reframing of IRL preserves the standard properties of IRL but is simpler to optimize and avoids the need for adversarial training used by other methods. This results in a simple imitation learning method that is competitive with or outperforms reasonable baselines on a few standard benchmarks. The proposed algorithm has other nice features, including the ability to easily recover the learned reward and to be trained both offline and online, and it can be easily rephrased to obtain a method for imitation from observations.
main_review: Positives:
- See my summary. I found the core technical contributions of this work to be novel, and potentially high-impact, as they simplify and reframe IRL to remove a common annoyance: the need for two-stage (or simultaneous two-part) training.
- The theoretical analysis is correct as far as I can tell. Although certain details were overemphasized to my taste (e.g. the choice of regularizers, which didn't make a huge difference empirically), the exposition was quite clean and easy to follow.
- The experiments compare against a representative set of reasonably strong baselines, and include enough domains to suggest that this method will work generally.

Negatives:
- As far as I can tell, none of the experimental domains here require generalization. These domains include very stereotyped initial state distributions, which means that an imitation agent that exactly replays the expert demonstration will solve the task. This likely plays a role in the proposed method's ability to perform well given a single demonstration. And in my opinion it is misleading, because it sidesteps one of the principle challenges in RL and IL, namely learning policies that can be used in the presence of at least mild occupancy drift (see [1] and [2] for discussions). Ideally, imitation learning methods should be evaluated in settings where the expert initial state distribution does not completely cover the states the agent can start in, such as the DeepMind Control Suite (as discussed in [2]), procedural tasks like CoinRun (as used in [3]) or tasks with more challenging perception (such as the door-opening task in [4]).  
- The proposed baselines are not as strong as they could be. In particular, Discriminator-actor-critic [5] or strongly regularized GAIL variants (such as [6]) are closer to what are used in practice than vanilla GAIL and typically give much better results. And another recently published method [4] produces similar results on Gym with single demonstrations, but is not mentioned or compared to.

Minor:
- I found the point made in footnote 2 to be obscure as phrased. Can you rephrase this to explain exactly how this relates to (6) and what this has to do with arbitrary experts?

Despite the limitations of the experimental evaluation, I believe the proposed algorithm is a solid and potentially quite impactful contribution to the IRL and imitation learning literature. I found the exposition insightful and I admire the simplicity of the proposed result. 

I am willing to raise my score if the limitations of the experimental evaluation are addressed.

[1] Mania et al - NeurIPS 2018 - Simple random search of static linear policies is competitive for reinforcement learning

[2] Jaegle et al - ICML 2021 - Imitation by Predicting Observations

[3] Edwards et al - ICML 2019 - Imitating Latent Policies from Observation

[4] Dadashi et al - ICLR 2021 - Primal Wasserstein Imitation Learning

[5] Kostrikov et al - ICLR 2019 - Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning

[6] Peng et al - ICLR 2019 - Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper suggests an alternative approach, equivalent to adversarial IRL (that learns both the policy and the reward in an adversarial fashion), but in a direct way (learning the Q-function that encompasses both the reward and the policy). As in adversarial methods (GAIL, AIRL etc) they use the maximum entropy assumption and more explicitly, they assume that the optimal policy is greedy wrt an optimal soft-Q-function. 

Using this relation between the optimal policy and the Q-function, they restrict the set of solutions to a manifold in which the objective only depends on the Q-function. They just have to minimize this objective (ensuring on the side that the policy stays in the manifold, i.e. it stays greedy) to obtain an optimal policy and Q-function. Their solution, in theory equivalent to the ones from an adversarial setting, is expected to outperform them as they avoid the learning difficulties of adversarial learning. Optionally, they can recover a reward function from the Q-function by simply inverting the Bellman operator. 

main_review: This paper proposes a novel IL/IRL algorithm, that can be applied both in online and offline settings, that is easy to adapt for continuous states/action spaces (approximating the value function within a separate critic network). The method is simpler to implement (close to SAC) and proved to converge. 

I would argue to accept it, my only concern being the lack of experiments: 
Mujoco: Did the authors try Humanoid? 
Atari: only 3 games, I understand that these environments are longer to train and require more engineering, but presenting just 3 envs looks like they tried more but had less good results on the other. I believe this is not the case, but having more games would really improve the paper (even if the results are not that good in other games, the paper is already great).
Reward reconversion: having results from more environments (at least the other mujoco) would be great to convince that the reward is correlated. 

However, the theory part is nice enough and the already given results are promising, so I will go for an accept. But if the authors can provide more results in the paper revision I will even increase my score.

Details:
A closely related IRL work inverses soft-policy improvement also to recover the soft-Q-function, but assuming the observed trajectories are generated by a soft-Q learner and not an expert:
Learning from a Learner (ICML 2019) by Jacq, Geist, and Pietquin.

In Eq.1, the notation $\pi^*$ is confusing, as it should be used for the policy that is greedy wrt the optimal soft-Q function ($\pi^*(a\vert s) \propto \exp(Q^*(s,a))$).

Same for $J^*(Q)$, that would be the optimized objective after convergence.

limitations_and_societal_impact: This paper has no societal impact.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper presents an IL/IRL method that avoids estimating the reward during optimization by alternating between updates of the policy and Q-function instead, which builds on recent insights from ValueDice. Indeed, the optimization problem for learning the Q-function is very similar to the one used by ValueDice, when optimizing the reverse KL. Notable differences compared to ValueDice are the generalization to a large class of divergences (including f-divergences), non-adversarial formulation, and a procedure to recover rewards.
In particular, I think that the modification for recovering a state-only reward function is very interesting.
The algorithm only requires mild modifications of SAC (mainly by using a different loss for the Q-function) and shows promising results in experiment evaluations.
main_review: Quality
======
Overall, the approach seems technically sound and promising. Section 3 establishes the foundations for framing inverse reinforcement learning in terms of the Q function.  Section 4 exploits the insight that we do not need to reverse the order of $\text{IRL} \circ \text{RL}$ (which was proposed by GAIL) to make the optimization tractable, if we directly learn the Q-function instead of the reward function, since the Q-function is already an implicit representation of the optimal policy, making the inner reinforcement learning trivial. 

The second important important insight is that the Q-function for a state-only reward function can be learned, simply by replacing the expectation over expert actions with an expectation over agent actions, which can be shown using importance sampling.

However, I have a question regarding the state-only rewards. It is not clear how a reward function of the form $r(s)$ is derived from the learned Q-function. From what I understand the learned models are all function of states and actions, and while the inverse Bellman update should result in a state-only reward, it is not clear to me how this step can be performed in practice.

Novelty / Significance
===================
* The presented derivations and algorithm are novel and very interesting. In particular, the derivation of the loss for the Q-function for general divergences and the modification for state-only rewards seem significant. Furthermore, the experimental results are promising.

* I think that Section 3 in itself does not contain that many new insights because it well known that the advantage function is a shaped [1] reward function, which also holds for MaxEnt RL, where maximizing the expected advantage + entropy regularization directly corresponds to minimizing the RKL to the distribution specified by A (or Q when disregarding normalization). Also, the transformation from the reward objective to Q-objective via the "inverse" Bellman operator strongly resembles the change-of-variables applied by DualDice/ValueDice. However, applying these ideas to Eq. 3 is an important contribution of the current submission. 

* The idea to simplify IRL/IL by directly learning the Q-function is not novel, and was introduced already in the early times of IRL (e.g. Opt-Q [2]) and became more traction with recent deep learning methods such as ValueDice.

* The Q-update for the RKL looks very similar to the ValueDice objective, which I think should be stated more clearly. From what I can tell, the differences to ValueDice are 1) The use of the f-gan-formulation of the RKL instead of Donsker-Varadhan (e.g. using the expectation instead of the log-expectation in the first term), 2) Learning the soft-Q-function rather than the Q-function, 3) the actor update. Here 1) is rather insignificant as this option was also stated by Kostrikov et al. 2020. More similar is the connection to O-NAIL [3], which also uses the (DV or F-Gan) objective for the soft-Q-function in combination with the soft-actor-critic update.  Here the only difference seems to be that they provide an additional reward based on the last policy to incur a KL penalty between updates.


Clarity
======
The presentation of theory and algorithm is excellent. 


References
==========
[1] Ng, Andrew Y., Daishi Harada, and Stuart Russell. "Policy invariance under reward transformations: Theory and application to reward shaping." Icml. Vol. 99. 1999.

[2] Dvijotham, Krishnamurthy, and Emanuel Todorov. "Inverse optimal control with linearly-solvable MDPs." ICML. 2010.

[3] Arenz, Oleg, and Gerhard Neumann. "Non-Adversarial Imitation Learning and its Connections to Adversarial Methods." arXiv preprint arXiv:2008.03525 (2020).
limitations_and_societal_impact: limitations and societal impact have been sufficiently addressed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work aims at improving the sample efficiency in terms of expert demonstrations in IL. In order to do so, it transforms the usual max-min objective of IRL into a single maximization objective by expressing both the reward and the policy in terms of the Q-function. The methods are supported with proofs and tested on both discrete and continuous action environments for the online setting and on simpler environments for the offline setting.
main_review: 2. Strong points

The method seems well supported by the theoretical derivations and the experiments. It is tested on many different settings (offline/online, continuous/discrete actions, large/small observation space) and outperforms the baselines. It can also be used for IRL as it retrieves rewards close to the ground truth ones (at least in simple environments). The method seems to be implemented easily which would make it practical.


3. Weak points

   1- The related work section could be improved. To my knowledge, Klein et al. (2012) were the first to propose to represent the policy in terms of the Q-value to directly learn a reward function in Off-line IRL. Also, it would be interesting to discuss more in-depth the relationship of the proposed method with ASAF which also proposed an analytical solution to the maximization step of the min-max IRL objective enabling to directly learn the Q-values parametrizing the policy.  

   2- The derivation of the method could be made clearer, in particular how to get to Eq. (5) and how it yields the objectives of the method Eqs. (9) and (10).


4. Recommendation

I recommend accept because the method seems efficient, practical, and well supported both theoretically and experimentally but I think the paper can be improved in terms of clarity and discussion with related methods. Additionally, some points remain to be clarified (see 5. Additional Remarks)

------------------------------
 ####  UPDATE
The author's response addressed my concerns and provided additional experiments so I decided to raise my score. 

------------------------------

5. Additional Remarks 

   1- In the continuous case, when SAC is used to improve the policy, the method falls back to a min-max objective. How do you explain that it does not experiment with the same drawbacks as adversarial methods such as ValueDICE and GAIL?

   2- I would be interested in seeing how the performance on Atari varies in terms of the number of expert demonstrations (like Fig 2.).  

   3- I would be interested in seeing the performance of agents trained on the learned IRL reward as correlation is not always a good indicator of the quality of the reward. For example, the hardcoded reward of SQIL will train good agents and the one of GAIL can't (it is an apprenticeship algorithm and not an IRL one) yet SQIL has a lower correlation than GAIL (line 723 Appendix D).

   4- The methods you compare against on Atari were proposed for low dimension observations (MuJoCo), are there other IL/IRL methods you could compare against that were proposed for high dimensional envs?


6. Minor Remarks

   1- I am confused at line 490 and line 82: if $Q=\mathcal{B}Q$ and $Q=\mathcal{B}r$ then we would have $Q=r$ which does not make sense. Moreover $\mathcal{B}$ operates on $Q$ not $r$

   2- line 81-82, $Q$ is unique provided that $r$ is given but also $\pi$. It might be nice to differentiate $Q$ and optimal $Q$ (using for example $Q^*$ in Eqs. (1), (2), etc.) 

   3- line 271-272 "with role similar to $r$" is confusing in the case of ASAF as the discriminator is not used to learn a reward in order to train the imitation policy like GAIL or AIRL (or the hard-coded reward of SQIL)


References

Klein, Edouard, et al. "Inverse reinforcement learning through structured classification." NIPS 2012. 2012.
limitations_and_societal_impact: yes
review_title: null

Meta-review: This paper received very positive reviews initially. The reviewers liked the originality of the work, it's theoretical soundness and the good performance of the algorithm. They had some concerns and wanted to see more about generalization capacities, stronger baselines and a larger set of test environments. The discussion was rich enough to convince the reviewers to raise their scores. The authors made additional experiments and addressed all the concerns of the reviewers. 
Decision: Accept (Spotlight)