Receding Horizon Inverse Reinforcement Learning
Authors: Yiqing Xu
Wei Gao
David Hsu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Inverse Reinforcement Learning

Abstract: Inverse reinforcement learning (IRL) seeks to infer a cost function that explains the underlying goals and  preferences of  expert demonstrations. This paper presents Receding Horizon Inverse Reinforcement Learning (RHIRL), a new IRL algorithm for high-dimensional, noisy, continuous systems with black-box dynamic models. RHIRL addresses two key challenges of IRL: scalability and robustness. To handle high-dimensional continuous systems, RHIRL matches the induced optimal trajectories with expert demonstrations locally in a receding horizon manner and ``stitches'' together the local solutions to learn the cost; it thereby avoids the ``curse of dimensionality''. This contrasts sharply with  earlier algorithms that match with expert demonstrations globally over the entire high-dimensional state space. To be robust against imperfect expert demonstrations and control noise, RHIRL learns a state-dependent cost function ``disentangled'' from system dynamics under mild conditions. Experiments on benchmark tasks show that RHIRL outperforms several leading IRL algorithms in most instances. We also prove that the cumulative error of RHIRL grows linearly with the task duration. 

Receding Horizon Inverse Reinforcement Learning
Yiqing Xu
Wei Gao
David Hsu 1.2
School of Computing
Smart Systems Insitute
National University of Singapore
(xuyiqingggaowei00 gaowei90, Syyhsu--ccmpp nus edu. sg
Abstract
Inverse reinforcement learning (IRL) seeks to infer cost function that explains the underlying goals and preferences expert demonstrations. This paper presents receding-horizon horizon inverse reinforcement learning (RHIRL), an IRL algorithm for high-dimensional, noisy, continuous systems with black-box dynamic models. RHIRL addresses key challenges of IRL: scalability and robustness. To han- dle high-dimensional continuous systems, RHIRL matches the induced optimal trajectories with expert demonstrations locally in receding horizon manner and "stitches" together the local solutions to learn the cost; it thereby avoids the "curse of dimensionality". This contrasts with earlier algorithms that match with expert demonstrations globally over the entire high- doth-dimensionll state space. To be ro- bust against imperfect expert demonstrations and control noise, RHIRL learns a state-dependent cost function "disentangled" from system dynamics under mild conditions. Experiments on benchmark tasks show that RHIRL outperforms several leading IRI. algorithms in most instances. We also prove that the cumulative error RHIRL grows linearly with the task duration.
Introduction
Reinforcement learning (RL) has made exciting progress in range of complex tasks, including real-time game playing [24], visuo-motor control of robots [35], and many other works. The success, however, often hinges on carefully crafted cost function [17, 25]. which is major impediment to the wide adoption of RI in practice. Inverse reinforcement learning (IRI.) [26) addresses this need by learning cost function that explains the underlying goals and preferences of expert demonstrations. This work focuses on two key challenges in IRL,, scalability and robustness.
Classic IRL algorithms commonly consist of two nested loops. The inner loop approximates the optimal control policy for hypothesized cost function, while the outer loop updates the cost function by comparing the behavior of the induced policy with expert demonstrations. The inner loop must solve the (forward) reinforcement learning or optimal control problem, which in itself a challenge for complex high- sigh--imensional systems. Many interesting ideas have been proposed for IRL, including, e.g., maximum entropy learning [39, 41]. guided cost learning [7], and adversarial learning [8]. See Figure 1 for illustrations. They try to match globally optimal approximate policy with expert demonstrations over the entire system state space or sampled approximation of it. This impractical for nigh-dimensional continuous systems and is fundamental impediment to scalability, Like RL, IRL suffers from the same "curse of dimensionality". To scale up. receding-horizon IRL (RHIRL) computes locally optimal policies with receding horizons rather than a globally optimal policy and then matches them with expert demonstrations locally in succession (Figure ld). The local approximation and matching substantially mitigate the impact of high-dimensional space and improve the sample efficiency of RHIRL, at the cost of a local rather than a global solution. So RHIRL trades off optimality for scalability and provides alternative to current approaches.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure I: A comparison of RHIRL and selected IRL algorithms. They all try match the policy induced by the learned cost with expert demonstrations. (a) MaxEnt matches the exact feature count over the entire system state space. (b) REIRL, GCL, and GAN-GCL match the approximate feature count over sampled state trajectories globally over the entire task duration. (c) GAIL, AIRL, discriminate the sampled state state-action distributions. (d) RHIRL matches control sequences locally along demonstrated trajectories in a receding horizon manner.
Another important concern of IRL is noise in expert demonstrations and system control. Human experts may be imperfect for various reasons and provide good, but still suboptimal demonstrations. Further, the system may fail to execute the commanded actions accurately because of control noise. We want to learn a cost function that captures the expert's 's intended actions rather than the imperfectly executed actions. While earlier work has investigated the question of learning from sub-optimal or failed demonstrations [3, 31, 38]. there is a subtle, but critical difference between (i) demonstrations intended to be suboptimal and (ii) optimal demonstrations corrupted by noise. The existing work [3, 31, 38] addresses (i); RHIRL addresses (ii). To learn the true intentions from the noise corrupted demonstrations, RHIRL relies on simplifying assumption: the cost function is linearly separable with two components, one stae-dependen and one control control-dependent Many interesting systems in practice satisfy the assumption, least, approximately [28]. RHIRL then learns the state-dependent cost component, which is disentangled from the system dynamics [8] and agnostic to noise.
2 Related Work
IRL can be viewed as indirect approach imitation learning. It learns cost function, which induces optimal policy whose behavior matches with expert demonstrations. In contrast, behavior cloning (BC) is direct approach. It assumes independence among all demonstrated state-action pairs and learns a policy that maps states to actions through supervised learning of state-action pairs from expert demonstrations. The simplicity of BC is appealing. However, it typically requires large amounts of data to learn well and suffers from covariate shift [30]. IRL is more data-efficient.. Further, produces a cost function, which explains the expert demonstrations and potentially transfers to other systems with different dynamics. These benefits, however, come at the expense of greater computational complexity.
Classic IRL algorithms learn a cost function iteratively in a double-loop: the outer loop updates a hypothesized cost function, and the inner loop solves the forward RL problem for an optimal policy
and matches it with expert demonstrations. Various methods have been proposed [1, 7, 39, 41]. but they all seek globally optimal solution over the entire state space (Figure la) [39, 41] or the entire task duration (Figure 1b) [1, 7J. As a result, they face significant challenges in scalability and must make simplifications, such as locally linear dynamics [7]. Recent methods use the generative adversarial network (GAN) [10] to learn a discriminator that differentiates between the state or state-action distribution induced by the learned cost and that from expert demonstrations [5, 9, 13, 14, 18, 21, 27, 29]. We view this global matching as major obstacle to scalability. In addition, GAN training is challenging and faces difficulties with convergence.
RHIRL stands in between BC and the aforementioned IRI. algorithms by trading off optimality for scalability. BC performs local matching at each demonstrated state-action pair, treating all of them independently. Most existing IRL algorithms perform global matching over the entire state space or sampled trajectories from RHIRL follows the standard IRL setup. To tackle the challenge of scalability for high-dimenn dinensiimnsional continuous systems, RHIRL borrows ideas from receding horizon control (RHC) [19]. It solves for locally optimal control sequences with receding horizons and learns the cost function by "stitching" together a series of locally optimal solutions to match the global state distribution of expert demonstrations (Figure Id). Earlier work has explored the idea RHC in IRL. [23], but relies on handcrafted reward features and is limited to discrete, low-dimensional tasks. More recent work learns reward features automatically [20]. However, it focuses on lane navigation for autonomous driving, exploting known analytic model of dynamics and noise-free,

---

perfect expert demonstrations. RHIRL aims at general high- continuous tasks with noisey expert demonstrations.
Another challenge to IRI is suboptimal expert demonstrations and system control noise. Several methods learn an auxiliary score ranking to reweigh the demonstrations, in order to approximate the underlying optimal expert distribution [4, 5, 38j. RHIRL does not attempt to reconstruct the optimal expert demonstrations. It explicitl models the human errors in control actions as additive Gaussian noise and matches the noisy control with expert demonstrations, in order to learn from the intended, rather than the executed expert actions. Modelling the human error as additive Gaussian noise is natural choice technically [22, 34, 40], but compromises on realism. Human errors in sequential decision making may admit other structural regularities, as result of planning, resource constraints, uncertainty, or bounded rationality[15], Studying the specific forms of human errors in the IRI. context requires insights beyond the scope of our current work, but forms promising direction for future investigation.
3 Receding Horizon Inverse Reinforcement Learning
3.1 Overview
Consider a continuous dynamical system:
Fe+ [ vt);
where Ia R" is the state, U, RT is the control at time t, and the initial system state at t 0 follows a distribution J.. To account for noise in expert demonstrations, we assume that ig is a random variable following the Gaussian distribution N(ua) [u, Σ), with mean 2 and covariance Σ. We can control uy directly, but not U2 because of noise. The state- transstranntionn function f captures the system dynamics. RHIRL represents f as a black-box simulator and does not require its analytic form. Thus, we can accommodate arbitrary complex nonlinear dynamics. To simplify the presentation, we assume that the system dynamics is deterministic. We sketch the extension stochastic dynamics at the end of the section, the full proof is given in Appendix C.
In RL, are given cost function and want to find control policy that minimizes the expected total cost over time under the dynamical system. In IRL, we are not given the cost function, but instead, a set DD of expert demonstrations. Each demonstration is trajectory of states visited by the expert over time: r = (Io, 21, F2, ...., 1)) for a duration of T steps.
We assume that the expert chooses the actions to minimize an unknown cost function and want to recover this cost from the demonstrations. Formally, suppose that the cost function is parameterized by 0. RHIRL aims to learn cost function that minimizes the loss C(0; D), which measures the difference between the demonstration trajectories and the optimal control policy induced by the cost function with parameters 0.
RHIRL performs this minimization iteratively, using the gradient ac/00 to update 0. In iteration t. let x, be the system state at time t and D, be the set of expert sub-trajectories starting at time and having duration of maximum K steps. We use the current cost to perform receding horizon control (RHC) at 2t, with time horizon K, and then update the cost by comparing the resulting state trajectory distribution with the demonstrations in D,. Earlier IRL work requires the simulator to reset to the states from the initial state distribution [1, 6, 7, 8, 13, 18). To perform receding horizon optimization, RHIRL requires a simulator to reset to the initial state J, for the local control sequence optimization at each time step i. = O....T - 1, a common setting in optimal control (e.g., [36, 37, 20]). See Algorithm for a sketch.
First, sample M control sequences, each of length K (line 6). We assume that the covariance Σ known. If it is unknown, we set it to be identity by default. For our experiment results reported in Table and Table 2, Σ is unknown and is approximated by a constant factor of the identity matrix (grid search is performed to determine this constant factor). We show through experiments that the learned cost function is robust over different noise settings (Section 4.3). Next, we apply modei-predictive path integral (MPPI) control [37] at xT, MPPI provides an analytic solution for the optimal control sequence distribution and the associated state sequence distribution, which allow us to estimate the gradient ac/ /0 efficiently through importance sampling (lines 7-8) and update cost function parameters 0 (line 9). Finally, we execute the computed optimal control (line 10-12)
3

---

Algorithm RHIRL
Initialize randomly.
2:
for io 1,2,3, 3,...... do
3: Sample from
4: Initialize control sequence U of length K to (0,0, (O.)....).
for 0, 1, ... 2', do
6:
Sample M control sequences V for j = 1, .......du M. according to ((VU,,))
7:
Compute the importance weight wy for each V₁, using the state cost S(V, 24, 0). See equation (11). Compute 86 [(;; Dr, 21) using D, and V with weight wly, for j 1, ..,, M. See equation (12).
9:
0
4440 0
10:
Σ5=1 Wy,
11:
Sample 12 from (v/u, E), where u is the first element in the control sequence U.
12: 21+1 [(x, (t).
Remove from U. Append 0 the U.
14:
end for
15:
end for
and update the mean control input for the next iteration (line 13). We would like to emphasize that we only use the simulator to sample rollouts and evaluate the current cost function. Fixed expert demonstrations D are given upfront. Unlike DAgger, we do not query the expert online during training.
Our challenge is uncover cost function that captures the expert's intended controls (uo, U1....y 1y-1), even though they were not directly observed, because of noise, and do so in scalable and robust manner for high-dimensional, noisy systems.
We develop three ideas: structuring the cost function, matching locally with expert demonstrations, and efficient computation of the gradient 0C/00, which are described next.
3.2 Robust Cost Functions
To learn a cost function robust against noise, we make simplifying assumption that linearly separates the one-step cost into components:
cost g(æ; parameterized by 0 and quadratic control cost u TT' Σ u. Despite the simplification, this cost function models wide variety of interesting systems practice [28]. It allows RHIRL learn state cost g(æ; f), independent of control noise (Section 3.4). and thus generalize over different noise distributions (Section 4.3).
Suppose that V (vo, 01.-.-5 (K-1) is control sequence of length K, conditioned the input U= (Mo, U1,-... "1,-..,Uk--.............. 11) We apply V to the dynamical system in (1) with start state Tp and obtain a state sequence (ro, .1,,..,,000 ,,.,.,,x/ xx) with F f( 1; Uk-1) for 1: 1,2,..., K. Define the total
cost of V as
K Σ k=0
K
J(V, To: 0):
g (x 0)
Σ
T
Uk:
2
k=0
where À > 0 is constant weighting the relative importance between the state and control costs. For convenience, define also the total state cost of as
K
S(V, zoi 0) =
g 0).
(3)
While S is defined in terms of the control sequence V, it only depends the corresponding state trajectory T. This is very useful, as the training data contains state and not control sequences explicitly.
3.3 Local Control Sequence Matching
To minimize the loss L. each iteration RHIRL applies RHC with time horizon under the current cost parameters 0 and computes locally optimal control sequences of length K. In contrast, classic IRI. algorithms, such MaxEnt [41], perform global optimization over the entire task duration T in the inner loop. While RHC sacrifices global optimality, it is much more scalable and enables RHIRL

---

to handle high-dimensional continuous systems. We use the hyperparameter K to trade off optimality and scalability.
Specifically, we use MPPI [37] to solve for an optimal control sequence distribution at the current start state 2y in iteration f. The main result of MPPI suggests that the optimal control sequence distribution Q minimizes the "free energy" of the dynamical system and this free energy can be calculated from the cost of the state trajectory under Q. Mathematically, the probability density (V*V)*) can be expressed as a function of the state cost Sv; 0), with respect to Gaussian "base" distribution B(U, Σ) that depends on the control cost:
(2 (V E, my 0) - 7 Pe (V E) Texp(-X 2. 0))
1
where Z is the partition function. For the quadratic control cost in (2), we have U3 (0. ........ [37]. Intuitively, the expression in (4) says that the probability density of V* is the product of two factors, one penalizing high control cost and one penalizing high state cost. So controls with large values or resulting in high-cost states occur with probability exponentially small.
Equation (4) provides the optimal control sequence distribution under the current cost. Suppose that the control sequences for expert demonstrations D, follow distribution E. We define the loss [(; Dr, Nz) as the KL-divergence between the two distributions:
C(0; D., xx) Das (p (V(zz) ) (V E,,;;
(5)
which RHIRL seeks to minimize in each iteration. While the loss defined in terms of control sequence distributions, the expert demonstrations D provide state information and not control information. However, each control sequence V induces a corresponding state sequence 7 for given start state zo. and r determines the cost of V according to (2). We show in the next subsection that ac/08 00 can be computed efficiently using only state information from D. This makes RHIRL appealing for learning tasks in which control labels are difficult or impossible to acquire.
3.4 Gradient Optimization
To simplify notations, we remove the explicit dependency on x, and D, and assume that in iteration I of RHIRI, all control sequences are applied with ve as the start state and expert demonstrations come from D.. We have
Oc 0
/ Pa (V)
(V( dV PV Σ;0)
00
00
= (V(
00 S(V:0) dv - (e(VIU,,E;0) S(V
a
(6)
00
The first line follows directly from the definition, and the derivation for the second line is available in Appendix A.
We estimate the two integrals in (6) through sampling. For the first integral, we can use the expert demonstrations samples. For the second integral, cannot sample Pa directly, the optimal control distribution Q is unknown in advance. Instead, we sample from a known Gaussian distribution with density p P(VIU, Σ. and apply importance sampling so that
(v [V]]=Epprii (x
The importance weight w(V) is given by
w(V) Po (VIUn, E;0) (VIU,, E; PVVIU, Σ)
(V E)
(8)
(VIU, E) ,(V
To simplify the first ratio in (8), we substitute in the expression for Pe from (4):
Po (V E;
Pn (V Σ) Z exp (-
We then simplify the second ratio, as both are Gaussian distributions with the same covariance matrix Σ:
Pe (V 2) (VVU,E
exp
k-1
Σ
(ux uZ) (v)-)ux))
(10)
k=
5

---

where 2s and uk. k ... respectively. Similarly, u k = 0, ...., K are the mean controls for the base distribution, and they are all O in our case. Substituring (9) and (10) into (8), we have
K-1 -
are the mean controls and the sampled controls from pVV Σ),
w(V) exp(- x S(V:0) +
Σ k=
(11)
For each sampled control sequence V, evaluate the expression in (11) and normalize over all samples to obtain w(V).
To summarize, we estimate 0c/00 through sampling:
i 00 C(0, D1, v2)
1 a
1 S(V,, Tz; M Σ w ((1)) SVV, D
8
(12)
N
À 00
where =
are the control sequences for the expert demonstrations Dj and V 1, .., M are the sampled control sequences. Equation (12) connects OC/00 with 0S/00. The state cost function S is represented as a shallow neural network, and its derivative can be obtained easily through back-propagation.
evaluate 8 S(V, zt; 0), we do not actually use the expert control sequences, as they are unknown. We use the corresponding state trajectories in Dj directly, as the state cost depends only on the visited states. See equation (3).
Finally, we approximate the optimal mean control sequence according to (7):
U = Ep [
wVV)VV
(13)
The first element in the control sequence U is the chosen control for the current time step We remove the first element from U and append zero at the end. The new control sequence is then used as the mean for the sampling distribution in the next iteration.
3.5 Analysis
Since RHIRL performs local optimization sequentially over many steps, one main concern is error accumulation over time. For example, standard behavior cloning has one-step error and cumulative error ( over I steps, because of covariate shift [30]. The DAgger algorithm reduces the error to (TT) by querying the expert repeatedly during online learning [30]. We prove similar result for RHIRL, which uses offline expert demonstrations only. In iteration t of RHIRL, let Ps (V/ (2a) be the K-step expert demonstration distribution and Pa ( (V. B. Σ, Zy; 0) be the computed K-step optimal control distribution for some fixed cost parameters O. RHIRL minimizes the KL-divergence between these two distributions each iteration. Let (1)) be the state marginal distribution of expert demonstrations and (ar 0) be the state marginal distribution of the computed RHC policy under over the entire task duration T. Intuitively, we want the control policy under the learned cost to visit states similar to those of expert demonstrations in distribution. In other words, Paac (a 0) and Pa (a(x)) are close.
Theorem 3.1. If Dai (V.la,) II Div (1), 0)) < T ee/2.
< c for all t = 0, 1,.,,.... T 1, then
The theorem says that RHIRL.'s cumulative error, measured total variation distance DTy between (a(x)) and Pasc(2; ()) grows linearly with T'. The proof consists of three steps. First, in each iteration f, if the KL-divergence between two control sequence distributions are bounded by so is the KL-divergence between the two corresponding state distributions induced by control sequences. Next, we show that the KL-divergence between the state distributions over two successive time steps are bounded by the same Finally, we switch from KL-divergence to total variation distance and apply the triangle inequality to obtain the final result. Note that our local optimization's 's objective is defined in KL divergence, while the final error bound is in TV distance. We switch the distance measures to get the best from both. Minimizing the KI. divergence leads to strong local optimization result, but KI. itself is not a proper metric. Therefore, we further bound the KI. divergence by TV distance to obtain proper metric bound for the final result. The full proof is given in Appendix B. Since

---

Pendulum-v0
LumarL.ander-v2
Hopper-v2
Walker2d-v2
Ant-v2
CarRacing-vl
[S]:
3-D
8-D
11-D
17-D
111-D
96 96 (image)
A|:
2-D
3-D
6-D
8-D
3-D 1000
T:
100
250
1000
1000
1000
Figure Benchmark tasks. |S] and |AA] denote the dimensions of the state space and the action space. respectively. ' denotes the task horizon.
RHC performs local optimization in each iteration, we cannot guarantee global optimality. However, the theorem indicates that unlike standard behavior cloning, the cumulative error of RHIRL grows linearly and not quadratically T. This shows one advantage of IRL. over behavior cloning from the theoretical angle.
Given a control policy with resulting state marginal distribution pv (x), we can calculate expected total cost of V by integrating the one-step cost over py. Now suppose that the one-step cost bounded. Theorem 3.1 then implics that the regret in total cost, compared with the expert policy, also grows linearly in T.
3.6 Extension to Stochastic Dynamics
Suppose that the system dynamics is stochastic: Te+1 f(r,, Uz, wt), where is random variable that models the independent system noise. RHIRL still applies, with modifications. We redefine the total cost functions J(V, zo; 0) and 2o: 0) by taking expectation over the system noise distribution. When calculating the importance weight (V) in (11), sample over the noise distribution to estimate the expected total state cost. Finally, we may need more samples when estimating the gradient in (12), because of the increased variance due the system noise. The full derivation of the extension to stochastic dynamics given in Appendix C. The experiments in the current work all have deterministic dynamics. We leave experiments with the extension future work.
4 Experiments
We investigate two main questions. Does RHIRL scale up to high-dimensional continuous control tasks? Does RHIRL learn a robust cost function under noise?
4.1 Setup
We compare RHIRL with two leading IRL, algorithms, namely AIRL [8] and f-RL [27], and one imitation learning algorithm, GAIL [13]. In particular, f-IRL is recent algorithm that achieves leading performance on high-dimensional control tasks. We use the implementation of AIRL, GAIL, and f-IRL from the f-ILL's official repository along with the reported hyperparameters [27], whenever possible. We also perform hyperparameter search on grid to optimize the performance of every method on every task. The specific hyperparameter settings used reported in Appendix D.2.
Our benchmark set consists six continuous control tasks (Figure 2) from OpenAI Gym [2], with increasing sizes of state and action spaces. For the most complex task, CarRacing, the input consists of 96 x 96 raw images, resulting in an enormous state space that poses great challenges [11]. To our knowledge, RHIRL is the first few IRL algorithms to attempt such high-dimensional space. For fair comparison, we customize all tasks to the fixed task horizon settings (Figure See Appendix D.2 for details task parameter settings.
use WorldModel [11] to generate expert demonstration data for CarRacing and use SAC [12] for the other tasks. We add Gaussian noise to the input controls and collect expert demonstrations different control noise levels. The covariance of the control noise is unknown to all methods, including, in particular, RHIRL.

---

Drive in the center of the lame.
Adjustt the direction at sharp turn.
Chouse the shartest path to make
Align the lane center aller the
the
turn.
Figure 3: Driving behaviors learned by RHIRL for CarRacing-vil -O,, with pixel-level raw image input.
To measure the performance of the learned cost function and policy, we score its induced optimal policy using the ground-truth cost function. For ease of comparison with the literature, we use negated cost values, i.e., rewards, in all reported results. Higher values indicate better performance. Each experiment is repeated 10 times to estimate the performance variance.
4.2 Scalability
We compare RHIRL with other methods, first in noise-free environ- ments (Figure 4) and then with increasing noise levels (Table 1).
Table 1: Performance comparison of RHIRL and other methods. The per- formance is reported as the ratio of the learned policy's average return and the expert's 's average return. The absolute average returns and the standard deviations are reported in Appendix D.4. Negative ratios are clipped to 0. The two numbers under the name of each environment indicate the dimensions of the state space and the action space, respectively.
Figure shows the learn- ing curve of each meth- ods in noise-free en- vironments. Overall, RHIRL converges faster and achieves higher turn, especially for tasks with higher state space dimensions. This im- proved performance sug- gests that the benefit of local optimization adopted by RHIRL out- weighs its potential lim- itations.
No Noise
Mild Noise
High Noise
Σ
Σ 0.2
0.5
Pendulum 3,
Expert
-154.69 67.61
-156.50 + 70.72
-168.54 80.89
RIIRL
f-IRL.
1.07
1,06
0.93
AIRL
1.05
0.94
0.91
GAIL
0,88
0.89
0,80
Lunarlander
Expert
232.00 + 86.12
222.65 d: 56.35
164.52 =:: 16.79
RHIRL
1.05
f-RL
0.76
0.63
0.74
AIRL
0.74
0.60
0.58
GAIL
0.72
0.56
0.60
Hopper 11,
Export
3222, 48 = 390.65
3159.72 72 = 520.00
2887, 72 483.93
RHIRL
0.95
0.98
0.96
1IRL
0,96
0.82
0.43
AIRL
0.01
0.01
0.01
GAIL.
0.82
0.50
0.24
Walker2d 17,6
Expert
4999.47 d: $5.99
4500.43 J: 114.48
3624.48 J. 95.05
RHIRL
0.99
0.99
0.95
Table shows the fi- nal performance all methods at various noise levels. RHIRL clearly outperforms AIRL and GAIL in all experiments. So focus our discus- sion on comparison with f-IRL. In noise-free en- vironments, RHIRL and f-IRL perform comparably on most tasks. On CarRacing, the most challenging task, RHIRL per- forms much better. RHIRL manages to learn the critical driving behaviors illustrated in Figure 3, despite the high-dimensional image input. However, RHIRL does not manage learn to drive fast enough. That is the main reason why it under-perform: the expert. In comparison, f-IRL only learns follow straight lane after large number of environment steps, and still fails to make a sharp turn after 3.0 x 107 environment steps. In the noisy environments, the advantage of RHIRL over f-IRL more pronounced even on some the low-dimensional tasks, because RHIRL accounts for the control noise explicitly in the cost function.
AIRL
0.00
0.00
0.00
GAIL
0.50
0.64
0,51
Ane
Expert
5759.22 + 173.57
3257.37 +:: 5-5.9595
252.62 + 91.44
111,
RHIRL
0.86
0.93
0.91
AIRI.
0.17
0.33
0.00
GAIL
0.48
0.40
0.00
CarRacing 96 x 96.
Expert RHIRL
903.25 :: 0.23
702.01 : 0.3
281.12 : 0.34
0.40
0.29
0.19
F.I.
AIRL GAIL
0.00
0.00
0.00
0.00
0.01
0.00

---

Penduluu-vv
LunatLanderContinuuussz
Hopper-v2
400
3000
2000
-500
-200
-750
11000
-400
0-
0.0
0.6 Steps
Steps
Steps
Walker2dFH-v0
Ant-v2
CarRacing-v0
6000
7500
4000
5000
400
2500
2000
200
-2500
0.4
0.6
0.8
1.0
Number of Env Steps
1e7
Number of Env Steps
le6
Number of Env Steps
107
RHIRL
AIRL
GAIL
f-IRL
Figure 4: Learning curves for RHIRL and other methods.
4.3 Robustness
Next evaluate RHIRL and other methods for robustness under noise. robust cost function encodes the expert's s true intent. It is expected to yield consistent performance over dif- ferent noise levels, regardless of noise in expert demonstrations.
Table 2: Robustness RHIRL and other methods under
noise. The performance is measured the ratio between the average return of an re-optimized policy in a noisy environ- ment and the expert's average return in the corresponding noise-free environment. The absolute average returns and the standard deviations reported in Appendix D.4. The negative ratios are clipped to 0.
For each task, a cost function is learned in noise-free environment and is then used to re-optimize a pol- icy in the corresponding noisy vironments. Specifically for GAIL, since it learns policy and does recover the associated cost function, we directly apply the learned policy in noisy environments.
Noise Level 3?
0.2
0.5
Pendulum 3.
Expert
-154.69 + 67.61
RHIRL
1.06
1.07
1.06
f-RRL
0.90
0.85
AIRL
1.05
0.79
0.67
GAIL
0.71
LunarLander 8,2
Expert
232.00 = 86.12
RHIRL
1.05
0.89
0.76
FIRL
0.76
0.53
0.44
AIRL
0.74
0.14
0.10
GAII
0.72
0.44
0.34
Table 2 shows that noise causes per- formance degradation in all methods. However, RHIRL is more robust in comparison. For simple tasks, Pendu- lum and Lunarlander, RHIRL and IRL perform consistently well across different noise levels, while GAIL and AIRL fail to maintain their good per- formance, when the noise level in- creases. For the more challenging tasks, Hopper and Walker, RHIRL's performance degrades mildly, and f- IRL suffers more significant perfor- mance degradation. It is worth noting that the expert demonstrations used to train the transferred cost function are from the perfect system. Therefore,
Hopper 11,3
Expert RHIRL
3222.48 390.65
0.95
0.80
0.67
IRL
0.96
0.65
0.62
AIRL
0.01
GAIL
0.82
0.07
0.06
Walker 17,
Expert RHIRL
4999.47 55.99
fIRL
0.99
0.60
0.22
AIRL
0.00
0.28
0.36
GAIL
0.50
0.02
0.02
Ant 111,8
Expert
$759.22
173.57
RHIRL
0.86
0.55
0.15
FIRL
0.87
0.35
0.08
AIRL
GAIL 0.17
0.15
0.00
0.00
CarRacing
Expert RHIRL
903.25 0.23
96 96,
0.40
0.29
0.12
AILL
0.09
0.02
0.00
AIRL
0.00
GAIL
0.00
some expert actions and states may longer be optimal or feasible in highly noisy environment.

---

Moreover, the cost function trained in the perfect system cannot reason about the long-tern conse- quences of an action high noise environment. Therefore, it is challenging for the learned cost function to be robust to highly noisy environment, capturing the true intention of the expert is difficult.
4.4 Effect of Receding Horizon K
RHIRL uses the receding horizon K to trade off optimal- ity and efficiency. We hope to ablate the effect of K on Hopper-v2 show how different Ks affect the final per- formance and sample complexity. The task horizon for Hopper-v2 1000 steps, i.c. T 1000. We run RHIRL with the receding horizon K E {5, 20, 100). The results are illustrated in Figure 5. When K is small, RHIRL. improves its performance quickly but converges to the suboptimal solution. For K
3000
1983
RHIRL's performance shoots up after the first few iterations to 1000, then it quickly converges
Nustoc
final score of 1100. When in- creases, though the performance improves slightly slower than = 5, can continue to learn and reach a score of 3071.68. At K 20, it takes fewer than 1e6 env steps to stabilize to a score greater than 3000. However, when too large, the learning becomes much slower. When K = 100, it takes more than 1e7 env steps to stabilize to a score larger than 3000, which 10 times more than when K = 20. On the other hand, K 100 achieve final score of 3083, which slightly more than that of K 20. This ablation study shows that our receding horizon K can tradeoff optimality and efficiency: using a smaller K allows to learn faster at the expense of sub-optimal solution, while using large K may make the learning inefficient. Seeking suitable K can balance the requirement for optimality and efficiency.
Figure 5: The effect of receding horizon K on the performance of RHIRL on the Hopper-v2 task.
Conclusion
RHIRL is scalable and robust IRL algorithm for high-dimensional, noisy, continuous systems. Our experiments show that RHIRL outperforms several leading IRL algorithms on multiple benchmark tasks, especially when expert demonstrations are noisy.
RHIRL's 's choice of local rather than global optimization is an important issue that deserves further investigation. Overall, we view this as an interesting trade-off between scalability and optimality. While this trade-off well known in reinforcement learning, optimal control, and general optimization problems, it is mostly unexplored in IRL. Further, local optimization may tie the learned cost with the optimizer. would be interesting to examine whether the learned cost transfers easily other domains with different optimizers. We are keen to investigate these important issues and their implications to IRL as our next step.
Acknowledgments This research is supported in part by the Agency of Science, Technology and Research, Singapore, through the National Roboties Program (Grant No. 192 25 00054), the National Research Foundation (NRF), Singapore and DSO National Laboratories under the Al Singapore 222--RP--220-0666 and the National University of Singapore (AcRF
Program (AISG Award No: grant A-0008071 (188071-00000) Any opinions, findings and conclusions recommendations expressed in this material are those of the authors and do not reflect the views of NRF Singapore.
References
[1] A. Boularias, J. Kober, and J. Peters. Relative entropy inverse reinforcement learning. In Proc. Int. Conf. on Artificial Intelligence & Statistics, 2011.
[2] G. Brockman, V. Cheung, I.. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. arXiv preprint
2016.
10

---

[3] D. S. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In Proc. Int. Conf. on Machine Learning, 2019.
[4] D. S. Brown, W. Goo, and S. Niekum. Better-than-demonstiie imitation learning via automatically-ranker demonstrations. Proc. Conference on Robot Learning, 2019.
[5] L. Chen, R. R. Paleja, and M. C. Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. Proc. Conference on Robot Learning, 2020.
[6] C. Finn, P. Christiano, P. Abbeel, and S. Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. In Advances in Neural Information Processing Systems, 2016.
[7] C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In Proc. Int. Conf. on Machine Learning, 2016.
[8] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. In Proc. Int. Conf. on Learning Representations, 2018.
[9] S. K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective imitation learning methods, In Proc. Conference on Robot Learning, 2019.
[10] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems, 2014.
[11] D. Ha and J. Schmidhuber. World models. In Advances in Neural Information Processing Systems, 2018
[12] Haarnoja, Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. Int. Conf. on Machine Learning. 2018.
[13] J. Ho and Ermon. Generative adversarial imitation learning. Advances in Neural Informa- tion Processing Systems, 2016.
[14] H. Hoshino, K. Ota, A. Kanezaki, and R. Yokota. Opirl: Sample efficient off- inverse reinforcement learning via distribution matching. Proc. IEEE Int. Conf. on Robotics & Automation, 2021.
[15] D. Kahneman and A. Tversky. Prospect theory: An analysis of decision under risk. Economet- rica, 47(2):263-291, 1979.
[16] L. Ke, S. Choudbury, M. Barnes, W. Sun, G. Lee, and S. Srinivasa. Imitation learning as f-divergence minimization. In Algorithmic Foundations of Robotics XIV-Proc. Ini. Workshop on the Algorithmic Foundations of Robotics (WAFR), 2020.
[17] W. B. Knox, A. Allievi, H. Banzhaf, F. Schmitt, and P. Stone. Reward (mis)design for au- tonomous driving, arXiv preprint M.XXi:2204..3900 2021.
[18] I. Kostrikov, K. K. Agrawal, Dwibedi, Levine, and J. Tompson. Discriminator-actorrriiic Addressing sample inefficiency and reward bias in adversarial imitation learning. In Proc. Int. Conf. on Learning Representations, 2019.
[19] W.-H. Kwon and Han. Receding Horizon Control: Model Predictive Control for State Models. Springer-Verlag Verlag London, 2005.
[20] K. Lee, D. Isele, E. A. Theodorou, and S. Bae. Spatiotemporal costmap inference for MPC via deep inverse reinforcement learning. In IEEE Robotics & Automation Letters, 2022.
[21] L. Lee, B. Eysenbach, Parisotto, E. Xing, Levine, and R. Salakhutdinov. Efficient exploration via state marginal matching. ArXiv, abs/1906.05274, 2019.
[22] S. Levine and V. Koltun. Guided policy search. In Proc. Ins. Conf. on Machine Learning, 2013.

---

[23] J. MacGlashan and M. L. Littman. Between imitation and intention learning. In Proc. Int. Jut. Conf. on Artificial Intelligence, 2015.
[24] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Ried- miller. Playing atari with deep reinforcement learning. In Advances in Neural Information Processing Systems, 2013.
[25] A. Ng. D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proc. Int. Conf. on Machine Learning, 1999.
[26] A. Y. Ng and J. Russell. Algorithms for inverse reinforcement learning. Proc. Int. Conf. on Machine Learning, 2000.
[27] T. Ni, H. Sikchi, Y. Wang, T. Gupta, L. Lee, and B. Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Proc. Conference on Robot Learning, 11 2020.
[28] W. Powell. Approximate Dynamic Programming: Solving the Curses of Dimensionality. Wiley- Interscience, 2007.
[29] A. H. Qureshi, B. Boots, and M. C. Yip. Adversarial imitation via variational inverse reinforce- ment learning. In Proc. Int. Conf. on Learning Representations, 2019.
[30] S. Ross, G. J. Gordon, and J. A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proc. Int. Conf. on Artificial Intelligence Statistics, 2011.
[31] K. Shiarlis, J. V. Messias, and S. Whiteson. Inverse reinforcement learning from failure. In Proc. Int. Conf. on Autonomous Agents & Multiagent Systems, 2016.
[32] Spencer, S. Choudhury, A. Venkatraman, B. Ziebart, and A. Bagnell. Feedback in imitation learning: The three regimes of covariate shift, 2021.
[33] E. A. Theodorou and E. Todorov. Relative entropy and free energy dualities: Connections to path integral and kl control. In Proc. IEEE Conference Decision & Control,
[34] E. Todorov and M. Jordan. Optimal feedback control as theory of motor coordination. Nature neuroscience, 5:1226335, 12 2002.
[35] U. Viereck, A. Pas, K. Saenko, and R. Platt. Learning visuomotor controller for real world robotic grasping using simulated depth images. In Proc. Conference on Robot Learning, 2017.
[36] G. Williams, B. Goldfain, P. Drews, K. Saigol, J. M. Rehg, and E. A. Theodorou. Robust sampling based model predictive control with sparse objective information. Robotics: Science and Systems XIV, 2.
[37] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpe for model-based reinforcement learning. In Proc. IEEE Int. Conf. on Robotics & Automation, 2017.
[38] Y. H. Wu, N. Charoenphakdee, H. Bao, V. Tangkaratt, and M. Sugiyama. Imitation learning from imperfect demonstration. In Proc. Int. Conf. on Machine Learning, 2019.
[39] M. Wulfmeier, P. Ondruska, and I. Posner. Maximum entropy deep inverse reinforcement learning. In Advances in Neural Information Processing Systems, 2016.
[40] B. D. Ziebart, A. Bagnell, and K. Dey. Modeling interaction via the principle of maximum causal entropy. In Proc. Int. Conf. on Machine Learning, 2010.
[41] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Proc. AAAI Conf. on Artificial Intelligence, 2008.
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes], code is included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "rCST3pz8ElD", "reviewer": "Reviewer_SQCF", "summary": "In this work, the authors propose an inverse reinforcement learning (IRL) algorithm based on trajectory optimization. As opposed to prior methods that either do single-step supervised learning (like behavioral cloning) or require reasoning about infinite horizons (like GAIL or MaxEnt IRL), the authors try to match the expert’s behavior over a short time horizon. They do this by using the MPPI algorithm with a parameterized cost function and optimize the cost function parameters to minimize the KL divergence between the control sequence from MPPI and the (unobserved) control sequences from the expert. Thanks to the structure of the cost function and KL divergence, they can efficiently optimize this objective with access only to the expert state trajectories. The authors then demonstrate that RHIRL outperforms various IRL baselines on several control tasks, including a racing task where the observation is an image.", "questions": "Confusion about Theorem 3.1: If K = 1 (horizon of 1 step), we should run into the same quadratic error as in behavioral cloning, but we only get linear error. Why does RHIRL not run into this issue?\\\nHow long (in wall-clock time) does it take to run this algorithm compared to the baselines?\n", "limitations": "As implied in the “Questions”, the authors did not indicate how long in wall-clock time their algorithm takes. MPPI’s main bottleneck is the simulator, and that can make the algorithm end up running quite slowly.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed algorithm is a neat extension of trajectory optimization for inverse reinforcement learning, and I’m not aware of any prior method doing this. \n\nThe writing for the most part is clear. I think the authors make a nice comparison of prior IRL algorithms and where their proposed algorithm RHIRL sits among the prior work. I think the presentation in Section 3, though, is a bit spotty. For instance, the authors say they work with deterministic dynamics (line 95) but also say the dynamical system has “noisy control” (line 88).  I think it would be cleaner instead to connect to prior IRL work that assumes noisy experts, particularly those that also maximize entropy [2, 3]. In light of that, RHIRL seems to assume an optimal expert that has some Gaussian noise, which can happen under an LQG/iLQG formulation [1,4] of a maximum entropy RL problem.\n\nOne other thing I am confused about is how they make the cost function “robust” to noise. In particular, they factor the cost function into a sum of a state-dependent term and a quadratic control term. Since the noise is assumed to only enter in the controls, this structure is supposed to make the learned cost function robust to control noise, something that is corroborated in the experiments. However, do we expect this structure to generalize to other IRL problems where the noise isn’t just additive Gaussian?\n\nThe results in Section 4.4 also seem a bit weak since it is only run on the pendulum, which is a simple problem that RHIRL solves easily. There aren't any clear patterns in Fig 5, either. I would suggest running that evaluation on a more difficult problem, like Hopper or Walker2D.\n\nSome other comments:\n- The style of Fig. 1a should match that of Fig. 1b-d.\n- Parentheses for exp need to be fixed in Equation (11).\n- Misreferenced equations occur frequently. For example, lines 157 and 160 should reference (4).\n- No anonymized code included.\n\n[1] Todorov, E., & Jordan, M. I. (2002). Optimal feedback control as a theory of motor coordination. Nature neuroscience, 5(11), 1226-1235.\\\n[2] Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2008, July). Maximum entropy inverse reinforcement learning. In Aaai (Vol. 8, pp. 1433-1438).\\\n[3] Ho, J., & Ermon, S. (2016). Generative adversarial imitation learning. Advances in neural information processing systems, 29.\\\n[4] Levine, S., & Koltun, V. (2013, May). Guided policy search. In International conference on machine learning (pp. 1-9). PMLR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bQ0cGYUDjg", "reviewer": "Reviewer_pMQy", "summary": "This paper proposes \"Receding Horizon IRL\" (RHIRL), a method for IRL that is designed to (1) expose a trade off between optimality and scalability, and (2) achieve scale even in the presence of noise in the demonstrations. While many of the core subtleties of RHIRL are framed around (1)---including the use of a receding horizon---the presence of noise in the demonstrator is one of the key novelties compared to other similar approaches to IRL. In particular, the paper assumes that the demonstrator's behavior is chosen, then the observed control is actually sampled from a Gaussian with known covariance (though the paper is careful to note that knowing this covariance is not critical). Thus, the key difficult of the IRL problem posed is to recover the reward (or cost) function that the demonstrator was optimizing, even in spite of the presence of this noise in behavior. Additionally, the trajectories only include state information. To my reading of the work, the key insight comes in Section 3.3, which articulates a _local_ method for matching the parameters of the unknown reward (cost) function $\\theta$ to the observed trajectory. As noted in the section, this is in contrast to many prior approaches to IRL that take a global view on this optimization. One Theorem is presented relating the KL between $p_E$ and $p_Q$ (the optimal control policy under $\\theta$) to the TV between $p_E(x)$ and $p_{RHC}(x; \\theta)$. Specifically, if this KL is bounded by $\\epsilon$ for _every_ time step $t \\in [0:T-1]$, then $D_{\\text{TV}}(p_E(x) || p_{\\text{RHC}}(x;\\theta) < T \\sqrt{\\epsilon / 2}$. I will say more about this result below, but in short, the paper claims we can thus conclude that the error grows linearly with $T$. Then, Section 4 discusses experiments contrasting RHIRL to other IRL and BC methods in a variety of standard benchmarks. Results are presented in Figure 3, 5 and Tables 1 and 2---we find a clear benefit of using RHIRL across every domain in Figure 3 compared to standard approaches. Table 1 and 2 suggest this trend holds even as we scale the noise on the demonstrator's side, while Figure 5 explores the impact of modifying $K$, the size of the receding horizon (and the knob that controls the optimality vs. scalability trade off).", "questions": "Q1: My primary question, as stated above, is about the motivation of the setting. As described in the introduction, many other approaches to IRL focus on sub-optimal demonstrators that are _systematically_ sub-optimal, due to uncertainty about the environment or bounded compute. In contrast, this paper motivates the case where a demonstrator's actions are perturbed due to Gaussian noise. I found this latter setting harder to motivate than the former, and wondered why we should consider it rather than the more typical case of systematic sub-optimality.\n\nQ2: In Theorem 3.1, I believe switching from KL to TV is actually making the result less clear, to me. The core of the result is translating T independent bounds over $p_E$ and $p_Q$ into a single T-step bound over $p_E$ and $p_{\\text{RHC}}$. However, by moving through Pinsker's to turn the KL into TV, it makes it slightly less apparent which aspects of the resulting bound are from aggregating over the T independent bounds to form a single T-step bound, rather than which elements come from translating KL over to TV. Is there a reason to invoke TV here? Otherwise I might recommend sticking with KL all the way through (or alternatively assuming the TV is bounded by $\\epsilon$ as the antecedent).", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[STRENGTHS]\n\nThis paper is well-written and largely well-motivated. I found the core of the method to be relatively simple to state, and the context provided in the introduction was effective at differentiating this method relative to many other approaches to IRL, imitation learning, and behavior cloning. I did find a few small aspects of the setup harder to motivate, but I will discuss that below. The paper is well organized, plots are well laid out (I found the table-like structure of Figures 2 and 3 to be a particularly effective way to convey a lot of information in a compact way). Experiments are well designed, and the results reported provide support for the conclusions drawn. For these reasons, I found this paper easy to read, and support the conclusion that RHIRL is a useful algorithm for IRL under the right assumptions.\n\n\n\n[WEAKNESSES]\n\nHowever, there were a few small aspects of the setup I had a hard time motivating. In particular, it is state early on that, unlike other approaches to IRL with a sub-optimal demonstrator, RHIRL will assume there is some Gaussian noise added to the demonstrator's behavior. This is in contrast to say, T-REX, that assumes the demonstrator is intentionally optimizing some unknown reward (cost) function, but does not know _how_, precisely, to achieve optimal behavior: \"The existing works mentioned above address (i). RHIRL addresses (ii).\" I appreciate the value of this distinction, and found it very useful to frame this paper. However, to me, I had a hard time motivating the setting: why would we be more interested in the case where a demonstrator adds Gaussian noise to their controls, rather than one in which the demonstrator is sub-optimal in a structurally significant way, as in the cited works that tackle \"(i)\"? This is my primary question about the work, which I restate below under \"questions\" for completeness.\n\nSecond, I found this paper to deviate heavily from standard RL notation. The notation was largely consistent (and of course is based in control, rather than RL), but as this is an IRL paper, I would find it cleaner to stick with typical RL notation. That is, to start from rewards rather than costs, and to use actions rather than controls. I suppose the connection to Gaussian noise in the demonstrator is more natural in an optimal control problem, so ultimately of course I leave this to the authors' discretion.\n\nLastly, one small typo: in Table 2, it looks like bold is misattributed to RHIRL in the first column of Ant.\n\n**Overall**, I found the experimental results were well presented, the experiments were interesting, and the overall conclusions are sensible. I believe this paper will be of interest to others in the community, though I note my own skepticism about some of the motivation of the problem formulation as mentioned above. This is the primary question I would like the authors to address.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bac0pxz3eQX", "reviewer": "Reviewer_VXKi", "summary": "This paper proposes a method for inverse RL based on optimizing short-horizon, open-loop action sequences. Similar to MPPI, the proposed method learns a policy by sampling a bunch of these open-loop action sequences, weights each by its (learned) reward function, and then updates the mean control sequence. After updating the policy, the proposed method updates the reward function using gradient descent. Empirically, the proposed method outperforms three imitation learning baselines on 6 tasks, and ablation experiments show that it is relatively robust to noise in the demonstrations.", "questions": "Questions:\n*  Most inverse RL and imitation learning methods can be simply described as a combination of (1) an inverse RL objective (e.g., distribution matching for MaxEnt IRL, a max-margin objective for max margin IRL) and (2) and RL algorithm (e.g., MPC, MPPI, LQR, SAC). Is the proposed method equivalent to doing MaxEnt IRL with MPPI, with a special parametrization of the cost function (Eq. 2)?\n* One of the central questions raised by the paper was the tradeoff vs local vs global methods. I didn't fully understand this point, and was hoping that the authors could elaborate a bit. Is the trade-off caused by using $K$ step planning instead of planning over infinite-length horizons? Or is it caused by planning over open loop sequences rather than closed loop sequences (i.e., contingency planning)?\n* Does the method require reset access to the simulator?\n* How does the proposed method compare to more recent+competitive baselines, such as (well-tuned) behavioral cloning and DAC [1]?\n* L37 \"trades off optimality\" -- Does this mean that the proposed method doesn't converge to the reward-maximizing policy?\n* Do the baselines also factor the reward function as in Eq. 2?\n\n**Summary**: Overall, I think the paper is generally strong. The proposed method seems theoretically sound, and empirically seems to outperform some baselines. My current score reflects uncertainty about three main aspects of the paper:\n1. I'm not clear about the difference between the proposed method and MaxEnt IRL + MPPI\n2. I'm not sure whether the proposed method requires resetting the environment, in ways that the alternative approaches don't\n3. It's not clear how the proposed method would compare to more recent/competitive imitation learning methods.\n\nIf these concerns are addressed, I would definitely consider increasing my score.\n\nMinor comments\n* L16 \"on line\" --> \"online\n* L19 \"etc\" -- Don't use \"etc\" in technical writing.\n* L21 \"in practice\" -- Cite.\n* L30 -- L34 -- I didn't understand this part of this paragraph.\n* Fig 1 -- I didn't understand this figure. It might be easier to visualize just one initial state for each example.\n* L67 \"challenges in scalability\" -- I'd recommend elaborating on this point.\n* L70 \"major obstacle\" -- Cite.\n* L96 \"full proof\" -- Proof of what?\n* Eq 2 -- Where is $J$ used in the subsequent derivation?\n* Fig 4 -- I didn't find this figure very informative, and recommend that it be replaced by a figure that explains the mechanics of the method (e.g., visualize a bunch of rollouts, visualize their rewards, visualize how this changes the mean control and the reward function)\n\n\n[1] https://arxiv.org/abs/1809.02925", "limitations": "Yes, the second paragraph of the conclusion has a great discussion of the trade-off between scalability vs optimality induced by the proposed method.", "rating": 3, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The proposed method is relatively simple.\n* I appreciate that the paper explains how the paper fits into the larger body of prior work by relating the tradeoffs made by shorter horizon planning.\n* I appreciate that the paper shows results on the image-based car-racing task, showing that the method isn't limited to low-dimensional settings.\n* The ablation experiments are great for studying the robustness of the method.\n\nWeaknesses\n* I'm a bit unsure whether the proposed method requires more assumptions than prior work. For example, the trajectory sampling in L6 of Alg 1 seems to require reset access to the simulator, which most prior methods avoid.\n* The proposed method seems fairly similar to prior work on imitation learning [1,2, 3].\n\n[1] https://arxiv.org/pdf/1206.4617.pdf\n\n[2] http://proceedings.mlr.press/v48/finn16.pdf\n\n[3] https://arxiv.org/pdf/2201.06539.pdf\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9IHRFRu1JW4", "reviewer": "Reviewer_xWf5", "summary": "This paper proposes a receding horizon IRL algorithm to address scalability for reward learning in high-dimensional control tasks. By using model-based planning and optimizing the learned reward function to match demonstrations locally, the authors show that their approach can perform comparably or better than prior works.", "questions": "Missing very related work \nMacGlashan and Littman. \"Between imitation and intention learning.\" Twenty-Fourth International Joint Conference on Artificial Intelligence. 2015.\nThey are, to my knowledge, the first to formulate a receding horizon IRL algorithm. The authors should add this reference and adjust their claims accordingly.\n\nAuthors claim that prior work isn't robust to noise, but MaxEnt IRL and Bayesian IRL are both robust to noisy demos.\n\nOther related work is \n\nLevine, Sergey, and Vladlen Koltun. \"Continuous inverse optimal control with locally optimal examples.\" arXiv preprint arXiv:1206.4617 (2012).\n\nand \n\nKalakrishnan, Mrinal, et al. \"Learning objective functions for manipulation.\" 2013 IEEE International Conference on Robotics and Automation. IEEE, 2013.\n\nwhich both consider local optimization to make IRL more tractable.\n\nGuided cost learning also performs local optimization over many steps. How is the proposed approach different?\n\nLine 251: Authors claim that no one else has looked at visual control tasks for IRL, but there has been prior work in this area including [12] and [27] (cited in the submission) as well as \n\nUchibe, Eiji. \"Model-free deep inverse reinforcement learning by logistic regression.\" Neural Processing Letters 47.3 (2018): 891-905.\n\nSection 4.4 How is task horizon chosen for the other tasks not considered in this ablation? How does it compare with the full horizon?\n\n\nIf you do finite horizon how do you deal with the cost to go at the last state? This is often learned for MPC style approaches. \n\nAlso, the learned cost function may be very different from the true cost. \nJain, Avik, et al. \"Optimal Cost Design for Model Predictive Control.\" Learning for Dynamics and Control. PMLR, 2021.\nshowed that learning shaped rewards for MPC worked better than using the true rewards.  Does this cause problems when the rewards are learned from data? I.e. is the reward significantly affected by the limitations of MPC and not suitable for optimization with other methods?\nAdding discussion about this would be good.\n\nWhen K=1 does RHIRL reduce to BC?\n\nMany of the figures are too small and make reading difficult.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Strong empirical results on a variety of domains\n\n+ Good results on robustness to noise in demonstrations\n\n\n- Relevant prior work is missing or not discussed in enough detail.\n\n- I did not see any results on the computational time required for RHIRL vs other methods compared to", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept