Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper revisits the generative adversarial network guided cost learning (GAN-GCL)  algorithm presented last year. The authors argue learning rewards from sampled trajectories has a high variance. Instead, they propose to learn a generative model wherein actions are sampled as a function of states. The same energy model is used for sampling actions: the probability of an action is proportional to the exponential of its reward. To avoid overfitting the expert's demonstrations (by mimicking the actions directly instead of learning a reward that can be generalized to different dynamics), the authors propose to learn rewards that depend only on states, and not on actions. Also, the proposed reward function includes a shaping term, in order to cover all possible transformations of the reward function that could have been behind the expert's actions. The authors argue formally that this is necessary to disentangle the reward function from the dynamics. Th paper also demonstrates this argument empirically (e.g. Figure 1).

This paper is well-written and technically sound. The empirical evaluations seem to be supporting the main claims of the paper. The paper lacks a little bit in novelty since it is basically a variante of GAN-GCL, but it makes it up with the inclusion of  a shaping term in the rewards and with the related formal arguments. The empirical evaluations could also be strengthened with experiments in higher-dimensional systems (like video games). 

"Under maximum entropy IRL, we assume the demonstrations are drawn from an optimal policy p(\tau) \propto exp(r(tau))" This is not an assumption, it's the form of the solution we get by maximizing the entropy (for regularization).

review_title: A variante of the GAN-GCL for Inverse RL (IRL) is presented and evaluated. The difference with the original algorithm is the fact that the sampling happens at the level of stat-actions instead of full trajectories, to reduce its variance. Empirical results clearly show the advantage of this method.

final_score: rating: 6: Marginally above acceptance threshold
review_content: SUMMARY:
This paper considers the Inverse Reinforcement Learning (IRL) problem, and particularly suggests a method that obtains a reward function that is robust to the change of dynamics of the MDP.

It starts from formulating the problem within the MaxEnt IRL framework of Ziebart et al. (2008). The challenge of MaxEnt IRL is the computation of a partition function. Guided Cost Learning (GCL) of Finn et al. (2016b) is an approximation of MaxEnt IRL that uses an adaptive importance sampler to estimate the partition function. This can be shown to be a form of GAN, obtained by using a specific discriminator [Finn et al. (2016a)].

If the discriminator directly works with trajectories tau, the result would be GAN-GCL. But this leads to high variance estimates, so the paper suggests using a single state-action formulation, in which the discriminator f_theta(s,a) is a function of (s,a) instead of the trajectory. The optimal solution of this discriminator is to have f(s,a) = A(s,a) — the advantage function.
The paper, however, argues that the advantage function is “entangled” with the dynamics, and this is undesirable. So it modified the discriminator to learn a function that is a combination of two terms, one only depends on state-action and the other depends on state, and has the form of shaped reward transformation.


EVALUATION:

This is an interesting paper with good empirical results. As I am not very familiar with the work of Finn et al. (2016a) and Finn et al. (2016b), I have not verified the detail of derivations of this new paper very closely. That being said, I have some comments and questions:


* The MaxEnt IRL formulation of this work, which assumes that p_theta(tau) is proportional to exp( r_theta (tau) ), comes from
[Ziebart et al., 2008] and assumes a deterministic dynamics. Ziebart’s PhD dissertation [Ziebart, 2010] or the following paper show that the formulation is different for stochastic dynamics:

Ziebart, Bagnell, Dey, “The Principle of Maximum Causal Entropy for Estimating Interacting Processes,” IEEE Trans. on IT, 2013.

Is it still a reasonable thing to develop based on this earlier, an inaccurate, formulation?


* I am not convinced about the argument of Appendix C that shows that AIRL recovers reward up to constants.
It is suggested that since the only items on both sides of the equation on top of p. 13 depend on s’ are h* and V, they should be equal.
This would be true if s’ could be chosen arbitrararily. But s’ would be uniquely determined by s for a deterministic dynamics. In that case, this conclusion is not obvious anymore.

Consider the state space to be integers 0, 1, 2, 3, … .
Suppose the dynamics is that whenever we are at state s (which is an integer), at the next time step the state decreases toward 1, that is s’ = phi(s,a) = s - 1; unless s = 0, which we just stay at s’ = s = 0. This is independent of actions.
Also define r(s) = 1/s for s>=1 and r(0) = 0.
Suppose the discount factor is gamma = 1 (note that in Appendix B.1, the undiscounted case is studied, so I assume gamma = 1 is acceptable).

With this choices, the value function V(s) = 1/s + 1/(s-1) + … + 1/1 = H_s, i.e., the Harmonic function.
The advantage function is zero. So we can choose g*(s) = 0, and h*(s) = h*(s’) = 1.
This is in contrast to the conclusion that h*(s’) = V(s’) + c, which would be H_s + c, and g*(s) = r(s) = 1/s.
(In fact, nothing is special about this choice of reward and dynamics.)

Am I missing something obvious here?

Also please discuss how ergodicity leads to the conclusion that spaces of s’ and s are identical. What does “space of s” mean? Do you mean the support of s? Please make the argument more rigorous.


* Please make the argument of Section 5.1 more rigorous.
review_title: Using the deterministic-MDP formulation of MaxEnt IRL is a concern

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper provides an approach to learning reward functions in high-dimensional domains, showing that it performs comparably to other recent approaches to this problem in the imitation-learning setting. It also argues that a key property to learning generalizable reward functions is for them to depend on state, but not state-action or state-action-state. It uses this property to produce "disentangled rewards", demonstrating that they transfer well to the same task under different transition dynamics.

The need for "state-only" rewards is a useful insight and is covered fairly well in the paper. The need for an "adversarial" approach is not justified as fully, but perhaps is a consequence of recent work. The experiments are thorough, although the connection to the motivation in the abstract (wanting to avoid reward engineering) is weak.

Detailed feedback:

"deployed in at test-time on environments" -> "deployed at test time in environments"?

"which can effectively recover disentangle the goals" -> "which can effectively disentangle the goals"?

"it allows for sub-optimality in demonstrations, and removes ambiguity between demonstrations and the expert policy": I am not certain what is being described here and it doesn't appear to come up again in the paper. Perhaps remove it?

"r high-dimensional (Finn et al., 2016b) Wulfmeier" -> "r high-dimensional (Finn et al., 2016b). Wulfmeier".

"also consider learning cost function with" -> "also consider learning cost functions with"?

"o learn nonlinear cost function have" -> "o learn nonlinear cost functions have".

" are not robust the environment changes" -> " are not robust to environment changes"?

"We present a short proof sketch": It is unclear to me what is being proven here. Please state the theorem.

"In the method presented in Section 4, we cannot learn a state-only reward function": I'm not seeing that. Or, maybe I'm confused between rewards depending on s vs. s,a vs. s,a,s'. Again, an explicit theorem statement might remove some confusion here.

"AIRLperforms" -> "AIRL performs".

Figure 2: The blue and green colors look very similar to me. I'd recommend reordering the legend to match the order of the lines (random on the bottom) to make it easier to interpret.

"must reach to goal" -> "must reach the goal"?

"pointmass" -> "point mass". (Multiple times.)

Amin, Jiang, and Singh's work on efficiently learning a transferable reward function seems relevant here. (Although, it might not be published yet: https://arxiv.org/pdf/1705.05427.pdf.)

Perhaps the final experiment should have included state-only runs. I'm guessing that they didn't work out too well, but it would still be good to know how they compare.

review_title: Learns environment-independent rewards; reasonable next step in adversarial IRL

Meta-review: The AIRL is presented as a scalable inverse reinforcement learning algorithm. A key idea is to produce "disentangled rewards", which are invariant to changing dynamics; this is done by having the rewards depend only on the current state. There are some similarities with GAIL and the authors argue that this is effectively a concrete implementation of GAN-GCL that actually works.  The results look promising to me and the portability aspect is neat and useful!

In general, the reviewers found this paper and its results interesting and I think the rebuttal addressed many of the concerns. I am happy that the reproducibility report is positive which helped me put this otherwise potentially borderline paper into the 'accept' bucket.
Decision: Accept (Poster)