Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a receding horizon IRL algorithm to address scalability for reward learning in high-dimensional control tasks. By using model-based planning and optimizing the learned reward function to match demonstrations locally, the authors show that their approach can perform comparably or better than prior works.
strengths_and_weaknesses: + Strong empirical results on a variety of domains

+ Good results on robustness to noise in demonstrations


- Relevant prior work is missing or not discussed in enough detail.

- I did not see any results on the computational time required for RHIRL vs other methods compared to
questions: Missing very related work 
MacGlashan and Littman. "Between imitation and intention learning." Twenty-Fourth International Joint Conference on Artificial Intelligence. 2015.
They are, to my knowledge, the first to formulate a receding horizon IRL algorithm. The authors should add this reference and adjust their claims accordingly.

Authors claim that prior work isn't robust to noise, but MaxEnt IRL and Bayesian IRL are both robust to noisy demos.

Other related work is 

Levine, Sergey, and Vladlen Koltun. "Continuous inverse optimal control with locally optimal examples." arXiv preprint arXiv:1206.4617 (2012).

and 

Kalakrishnan, Mrinal, et al. "Learning objective functions for manipulation." 2013 IEEE International Conference on Robotics and Automation. IEEE, 2013.

which both consider local optimization to make IRL more tractable.

Guided cost learning also performs local optimization over many steps. How is the proposed approach different?

Line 251: Authors claim that no one else has looked at visual control tasks for IRL, but there has been prior work in this area including [12] and [27] (cited in the submission) as well as 

Uchibe, Eiji. "Model-free deep inverse reinforcement learning by logistic regression." Neural Processing Letters 47.3 (2018): 891-905.

Section 4.4 How is task horizon chosen for the other tasks not considered in this ablation? How does it compare with the full horizon?


If you do finite horizon how do you deal with the cost to go at the last state? This is often learned for MPC style approaches. 

Also, the learned cost function may be very different from the true cost. 
Jain, Avik, et al. "Optimal Cost Design for Model Predictive Control." Learning for Dynamics and Control. PMLR, 2021.
showed that learning shaped rewards for MPC worked better than using the true rewards.  Does this cause problems when the rewards are learned from data? I.e. is the reward significantly affected by the limitations of MPC and not suitable for optimization with other methods?
Adding discussion about this would be good.

When K=1 does RHIRL reduce to BC?

Many of the figures are too small and make reading difficult.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes "Receding Horizon IRL" (RHIRL), a method for IRL that is designed to (1) expose a trade off between optimality and scalability, and (2) achieve scale even in the presence of noise in the demonstrations. While many of the core subtleties of RHIRL are framed around (1)---including the use of a receding horizon---the presence of noise in the demonstrator is one of the key novelties compared to other similar approaches to IRL. In particular, the paper assumes that the demonstrator's behavior is chosen, then the observed control is actually sampled from a Gaussian with known covariance (though the paper is careful to note that knowing this covariance is not critical). Thus, the key difficult of the IRL problem posed is to recover the reward (or cost) function that the demonstrator was optimizing, even in spite of the presence of this noise in behavior. Additionally, the trajectories only include state information. To my reading of the work, the key insight comes in Section 3.3, which articulates a _local_ method for matching the parameters of the unknown reward (cost) function $\theta$ to the observed trajectory. As noted in the section, this is in contrast to many prior approaches to IRL that take a global view on this optimization. One Theorem is presented relating the KL between $p_E$ and $p_Q$ (the optimal control policy under $\theta$) to the TV between $p_E(x)$ and $p_{RHC}(x; \theta)$. Specifically, if this KL is bounded by $\epsilon$ for _every_ time step $t \in [0:T-1]$, then $D_{\text{TV}}(p_E(x) || p_{\text{RHC}}(x;\theta) < T \sqrt{\epsilon / 2}$. I will say more about this result below, but in short, the paper claims we can thus conclude that the error grows linearly with $T$. Then, Section 4 discusses experiments contrasting RHIRL to other IRL and BC methods in a variety of standard benchmarks. Results are presented in Figure 3, 5 and Tables 1 and 2---we find a clear benefit of using RHIRL across every domain in Figure 3 compared to standard approaches. Table 1 and 2 suggest this trend holds even as we scale the noise on the demonstrator's side, while Figure 5 explores the impact of modifying $K$, the size of the receding horizon (and the knob that controls the optimality vs. scalability trade off).
strengths_and_weaknesses: [STRENGTHS]

This paper is well-written and largely well-motivated. I found the core of the method to be relatively simple to state, and the context provided in the introduction was effective at differentiating this method relative to many other approaches to IRL, imitation learning, and behavior cloning. I did find a few small aspects of the setup harder to motivate, but I will discuss that below. The paper is well organized, plots are well laid out (I found the table-like structure of Figures 2 and 3 to be a particularly effective way to convey a lot of information in a compact way). Experiments are well designed, and the results reported provide support for the conclusions drawn. For these reasons, I found this paper easy to read, and support the conclusion that RHIRL is a useful algorithm for IRL under the right assumptions.



[WEAKNESSES]

However, there were a few small aspects of the setup I had a hard time motivating. In particular, it is state early on that, unlike other approaches to IRL with a sub-optimal demonstrator, RHIRL will assume there is some Gaussian noise added to the demonstrator's behavior. This is in contrast to say, T-REX, that assumes the demonstrator is intentionally optimizing some unknown reward (cost) function, but does not know _how_, precisely, to achieve optimal behavior: "The existing works mentioned above address (i). RHIRL addresses (ii)." I appreciate the value of this distinction, and found it very useful to frame this paper. However, to me, I had a hard time motivating the setting: why would we be more interested in the case where a demonstrator adds Gaussian noise to their controls, rather than one in which the demonstrator is sub-optimal in a structurally significant way, as in the cited works that tackle "(i)"? This is my primary question about the work, which I restate below under "questions" for completeness.

Second, I found this paper to deviate heavily from standard RL notation. The notation was largely consistent (and of course is based in control, rather than RL), but as this is an IRL paper, I would find it cleaner to stick with typical RL notation. That is, to start from rewards rather than costs, and to use actions rather than controls. I suppose the connection to Gaussian noise in the demonstrator is more natural in an optimal control problem, so ultimately of course I leave this to the authors' discretion.

Lastly, one small typo: in Table 2, it looks like bold is misattributed to RHIRL in the first column of Ant.

**Overall**, I found the experimental results were well presented, the experiments were interesting, and the overall conclusions are sensible. I believe this paper will be of interest to others in the community, though I note my own skepticism about some of the motivation of the problem formulation as mentioned above. This is the primary question I would like the authors to address.
questions: Q1: My primary question, as stated above, is about the motivation of the setting. As described in the introduction, many other approaches to IRL focus on sub-optimal demonstrators that are _systematically_ sub-optimal, due to uncertainty about the environment or bounded compute. In contrast, this paper motivates the case where a demonstrator's actions are perturbed due to Gaussian noise. I found this latter setting harder to motivate than the former, and wondered why we should consider it rather than the more typical case of systematic sub-optimality.

Q2: In Theorem 3.1, I believe switching from KL to TV is actually making the result less clear, to me. The core of the result is translating T independent bounds over $p_E$ and $p_Q$ into a single T-step bound over $p_E$ and $p_{\text{RHC}}$. However, by moving through Pinsker's to turn the KL into TV, it makes it slightly less apparent which aspects of the resulting bound are from aggregating over the T independent bounds to form a single T-step bound, rather than which elements come from translating KL over to TV. Is there a reason to invoke TV here? Otherwise I might recommend sticking with KL all the way through (or alternatively assuming the TV is bounded by $\epsilon$ as the antecedent).
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors propose an inverse reinforcement learning (IRL) algorithm based on trajectory optimization. As opposed to prior methods that either do single-step supervised learning (like behavioral cloning) or require reasoning about infinite horizons (like GAIL or MaxEnt IRL), the authors try to match the expert’s behavior over a short time horizon. They do this by using the MPPI algorithm with a parameterized cost function and optimize the cost function parameters to minimize the KL divergence between the control sequence from MPPI and the (unobserved) control sequences from the expert. Thanks to the structure of the cost function and KL divergence, they can efficiently optimize this objective with access only to the expert state trajectories. The authors then demonstrate that RHIRL outperforms various IRL baselines on several control tasks, including a racing task where the observation is an image.
strengths_and_weaknesses: The proposed algorithm is a neat extension of trajectory optimization for inverse reinforcement learning, and I’m not aware of any prior method doing this. 

The writing for the most part is clear. I think the authors make a nice comparison of prior IRL algorithms and where their proposed algorithm RHIRL sits among the prior work. I think the presentation in Section 3, though, is a bit spotty. For instance, the authors say they work with deterministic dynamics (line 95) but also say the dynamical system has “noisy control” (line 88).  I think it would be cleaner instead to connect to prior IRL work that assumes noisy experts, particularly those that also maximize entropy [2, 3]. In light of that, RHIRL seems to assume an optimal expert that has some Gaussian noise, which can happen under an LQG/iLQG formulation [1,4] of a maximum entropy RL problem.

One other thing I am confused about is how they make the cost function “robust” to noise. In particular, they factor the cost function into a sum of a state-dependent term and a quadratic control term. Since the noise is assumed to only enter in the controls, this structure is supposed to make the learned cost function robust to control noise, something that is corroborated in the experiments. However, do we expect this structure to generalize to other IRL problems where the noise isn’t just additive Gaussian?

The results in Section 4.4 also seem a bit weak since it is only run on the pendulum, which is a simple problem that RHIRL solves easily. There aren't any clear patterns in Fig 5, either. I would suggest running that evaluation on a more difficult problem, like Hopper or Walker2D.

Some other comments:
- The style of Fig. 1a should match that of Fig. 1b-d.
- Parentheses for exp need to be fixed in Equation (11).
- Misreferenced equations occur frequently. For example, lines 157 and 160 should reference (4).
- No anonymized code included.

[1] Todorov, E., & Jordan, M. I. (2002). Optimal feedback control as a theory of motor coordination. Nature neuroscience, 5(11), 1226-1235.\
[2] Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2008, July). Maximum entropy inverse reinforcement learning. In Aaai (Vol. 8, pp. 1433-1438).\
[3] Ho, J., & Ermon, S. (2016). Generative adversarial imitation learning. Advances in neural information processing systems, 29.\
[4] Levine, S., & Koltun, V. (2013, May). Guided policy search. In International conference on machine learning (pp. 1-9). PMLR.
questions: Confusion about Theorem 3.1: If K = 1 (horizon of 1 step), we should run into the same quadratic error as in behavioral cloning, but we only get linear error. Why does RHIRL not run into this issue?\
How long (in wall-clock time) does it take to run this algorithm compared to the baselines?

limitations: As implied in the “Questions”, the authors did not indicate how long in wall-clock time their algorithm takes. MPPI’s main bottleneck is the simulator, and that can make the algorithm end up running quite slowly.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a method for inverse RL based on optimizing short-horizon, open-loop action sequences. Similar to MPPI, the proposed method learns a policy by sampling a bunch of these open-loop action sequences, weights each by its (learned) reward function, and then updates the mean control sequence. After updating the policy, the proposed method updates the reward function using gradient descent. Empirically, the proposed method outperforms three imitation learning baselines on 6 tasks, and ablation experiments show that it is relatively robust to noise in the demonstrations.
strengths_and_weaknesses: Strengths
* The proposed method is relatively simple.
* I appreciate that the paper explains how the paper fits into the larger body of prior work by relating the tradeoffs made by shorter horizon planning.
* I appreciate that the paper shows results on the image-based car-racing task, showing that the method isn't limited to low-dimensional settings.
* The ablation experiments are great for studying the robustness of the method.

Weaknesses
* I'm a bit unsure whether the proposed method requires more assumptions than prior work. For example, the trajectory sampling in L6 of Alg 1 seems to require reset access to the simulator, which most prior methods avoid.
* The proposed method seems fairly similar to prior work on imitation learning [1,2, 3].

[1] https://arxiv.org/pdf/1206.4617.pdf

[2] http://proceedings.mlr.press/v48/finn16.pdf

[3] https://arxiv.org/pdf/2201.06539.pdf


questions: Questions:
*  Most inverse RL and imitation learning methods can be simply described as a combination of (1) an inverse RL objective (e.g., distribution matching for MaxEnt IRL, a max-margin objective for max margin IRL) and (2) and RL algorithm (e.g., MPC, MPPI, LQR, SAC). Is the proposed method equivalent to doing MaxEnt IRL with MPPI, with a special parametrization of the cost function (Eq. 2)?
* One of the central questions raised by the paper was the tradeoff vs local vs global methods. I didn't fully understand this point, and was hoping that the authors could elaborate a bit. Is the trade-off caused by using $K$ step planning instead of planning over infinite-length horizons? Or is it caused by planning over open loop sequences rather than closed loop sequences (i.e., contingency planning)?
* Does the method require reset access to the simulator?
* How does the proposed method compare to more recent+competitive baselines, such as (well-tuned) behavioral cloning and DAC [1]?
* L37 "trades off optimality" -- Does this mean that the proposed method doesn't converge to the reward-maximizing policy?
* Do the baselines also factor the reward function as in Eq. 2?

**Summary**: Overall, I think the paper is generally strong. The proposed method seems theoretically sound, and empirically seems to outperform some baselines. My current score reflects uncertainty about three main aspects of the paper:
1. I'm not clear about the difference between the proposed method and MaxEnt IRL + MPPI
2. I'm not sure whether the proposed method requires resetting the environment, in ways that the alternative approaches don't
3. It's not clear how the proposed method would compare to more recent/competitive imitation learning methods.

If these concerns are addressed, I would definitely consider increasing my score.

Minor comments
* L16 "on line" --> "online
* L19 "etc" -- Don't use "etc" in technical writing.
* L21 "in practice" -- Cite.
* L30 -- L34 -- I didn't understand this part of this paragraph.
* Fig 1 -- I didn't understand this figure. It might be easier to visualize just one initial state for each example.
* L67 "challenges in scalability" -- I'd recommend elaborating on this point.
* L70 "major obstacle" -- Cite.
* L96 "full proof" -- Proof of what?
* Eq 2 -- Where is $J$ used in the subsequent derivation?
* Fig 4 -- I didn't find this figure very informative, and recommend that it be replaced by a figure that explains the mechanics of the method (e.g., visualize a bunch of rollouts, visualize their rewards, visualize how this changes the mean control and the reward function)


[1] https://arxiv.org/abs/1809.02925
limitations: Yes, the second paragraph of the conclusion has a great discussion of the trade-off between scalability vs optimality induced by the proposed method.
review_title: null

Meta-review: metareview: This paper presents a new algorithm for inverse reinforcement learning (IRL) that uses receding horizon to locally match expert demonstrations, which generally results in better performance compared to global methods, especially with long horizons. The algorithm is shown to outperform some state of the art alternatives on a few simulated robotic environments. 
The paper is overall well-written, technically strong, and the proposed idea is interesting. The main issue, raised by one of the reviewers, is the assumption that the algorithm has access to a resettable dynamics model, but it seems like this assumption is also made by other model-free methods that are compared here, such as GAIL. 
confidence: Less certain
Decision: Accept