Robust Imitation via Decision-Time Planning
Authors: Carl Qi
Pieter Abbeel
Aditya Grover
Venue:  (2021)
Keywords: imitation learning
reinforcement learning
inverse reinforcement learning

Abstract: The goal of imitation learning is to mimic expert behavior from demonstrations, without access to an explicit reward signal. A popular class of approaches infers the (unknown) reward function via inverse reinforcement learning (IRL) followed by maximizing this reward function via reinforcement learning (RL). The policies learned via these approaches are however very brittle in practice and deteriorate quickly even with small test-time perturbations due to compounding errors. We propose Imitation with Planning at Test-time (IMPLANT), a new algorithm for imitation learning that utilizes decision-time planning to correct for compounding errors of any base imitation policy. In contrast to existing approaches, we retain both the imitation policy and the rewards model at decision-time, thereby benefiting from the learning signal of the two components. Empirically, we demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments and excels at zero-shot generalization when subject to challenging perturbations in test-time dynamics.

Under review as a conference paper at ICLR 2021
ROBUST IMITATION VIA DECISION-TIME PLANNING
Anonymous authors Paper under double-blind review
ABSTRACT
The goal of imitation learning is mimic expert behavior from demonstrations, without access to an explicit reward signal. A popular class of approach infers the (unknown) reward function via inverse reinforcement learning (IRL) followed by maximizing this reward function via reinforcement learning (RL). The policies learned via these approaches however very brittle in practice and deteriorate quickly even with small test-time perturbations due to compounding errors. We propose Imitation with Planning al Test-time (IMPLANT), new algorithm for imitation learning that utilizes decision-ting planning to correct for compounding errois of any base imitation policy. In contrast to existing approaches, we retain both the imitation policy and the rewards model at decision-time, thereby bene- fiting from the learning signal of the two components. Empirically, we demon- strate that IMPLANT significantly outperforms benchmark imitation learning ap- proaches on standard control environments and excels at zero-shot generalization when subject to challenging perturbations in test-time dynamics.
INTRODUCTION
The objective of imitation learning is optimize agent policies directly from demonstrations of expert behavior. Such a learning paradigm sidesteps reward engineering, which is a key bottleneck for applying reinforcement learning (RL) in many real-world domains, e.g., autonomous driving, robotics. In the presence of finite dataset of expert demonstrations however, key challenge with current approaches that the learned policies can quickly deviate from intended expert behavior and lead to compounding errors at test-time (Osa et al. 2018). Moreover, it has been observed that imi- tation policies can be brittle and drastically deteriorate in performance with even small perturbations to the dynamics during execution (Christiano et al., 2016 de Haan et al. 2019).
predominant class of approaches to imitation learning is based on inverse reinforcement learning (IRL) and involve successive application of two steps: (a) an IRL step where the agent infers the (unknown) reward function for the expert, followed by (b) an RL step where the agent maximizes the inferred reward function via policy optimization algorithm. For example, many popular IRL approaches consider an adversarial learning framework (Goodfellow et al.) (014). where the reward function is inferred by a discriminator that distinguishes expert demonstrations from roll-outs of an imitation policy [IRL step] and the imitation agent maximizes the inferred reward function to best match the expert policy [RL step] (Ho & Ermon 2016 Fu et al. (017).. In this sense, reward inference is only an intermediary step towards learning the expert policy and is discarded post- training of the imitation agent.
We introduce Imitation with Planning at Test-time (IMPLANT), a new algorithm for imitation learn- ing that incorporates decision-ting -ime planning within an IRL algorithm. During training, we can use any standard IRL approach to estimate reward function and a stochastic imitation policy, along with an additional value function. The value function can be learned explicitly or often byprod- uct of standard RL algorithms that involve policy evaluation, such as actor-critic methods (Konda) & Tsitsiklis 2000, Peters & Schaal 2008). At decision-time, we use the learned imitation policy in conjunction with closed-loop planner. For any given state, the imitation policy proposes a set of candidate actions and the planner estimates the returns for each of actions by performing fixed- horizon rollouts. The rollout returns are estimated using the learned reward and value functions. Finally, the agent picks the action with the highest estimated return and the process is repeated at each of the subsequent timesteps.

---

Under review as a conference paper at ICLR 2021
Conceptually, IMPLANT aims counteract the imperfections due to policy optimization in the RL step by using the reward function (along with a value function) estimated in the IRL step for decision-time planning. We demonstrate strong empirical improvements using this approach over benchmark imitation learning algorithms in a variety settings derived from the MuloCo-based benchmarks in OpenAl Gym (Todorov al. [2012] Brockman al. 2016). In default evaluation setup where train and test environments match, we observe that IMPLANT improves by 16.5% on
average over the closest baseline.
We also consider transfer setups where the imitation agent is deployed in test dynamics that differ from train dynamics and the test dynamics are inaccessible to the agent during both training and decision-time planning. In particular, we consider the following three setups: (a) "causal confusion" where the agent observes nuisance variables the state representation during training (de Haan et al. 2019), (b) motor noise which adds noise in the executed actions during testing (Christiano et al, 2016), and (c) transition noise which adds noise to the next state distribution during testing. In all these setups, observe that IMPLANT consistently and robustly transfers to test environments with improvements 35.2% on average over the closest baseline.
PRELIMINARIES
Problem Setup. We consider the framework of Markov Decision Processes (MDP) (Puterman 1990). An MDP is denoted by a tuple M = A, Po, 7)), where S is the state space, A is the action space, T : S x A S Rz0 are the stochastic transition dynamics, po S - R20 the initial state distribution,
S x A + R is the reward function, and We assume an infinite horizon setting. At any given state $ E S, an agent makes decisions via a stochastic policy π S x A + Rzn- We denote trajectory to be sequence of state-action pairs r = (8o, On, 81, 01,'' ...). ). Any policy я, along with MDP parameters, induces a distribution over trajectories, which can be expressed as (+(T)= = p(00) I <(a (st)T(((+ (1 0z). The return of a trajectory is the discounted sum of rewards R(T) = Ei= 7'r(s;,a:)
[0, 1) is the discount factor.
In reinforcement learning (RL), the goal to learn a parameterized policy Te that maximizes the expected returns w.r.t. the trajectory distribution. Maximizing such an objective requires interaction with the underlying MDP for simulating trajectories and querying rewards. However, in many high- stakes scenarios, the reward function is not directly accessible and hard to manually design.
In imitation learning, we sidestep the availability of the reward function. Instead, we have access to finite set of D trajectories TE ka. demonstrations) that are sampled from expert policy TE. Every trajectory E TE consists of finite length sequence of state and action pairs T = (80, 0o, 81, "1;'..) ), where 80 Ps()), a (-(s)) and 81+1 T a.). Our goal is to learn a parameterized policy He which best approximates the expert policy given access to Tg. Next, we discuss the two major families of techniques for imitation learning.
2.1 BEHAVIORAL CLONING
Behavioral cloning (BC) casts imitation learning as a supervised learning problem over state-action pairs provided in the expert demonstrations (Pomerleau (191). In particular, we learn the policy parameters by solving a regression problem with states 8, and actions a as the features and target labels respectively. Formally, we minimize the following objective:
(Bc = Σ
||a (s
(1)
(sT.0.)ETR
In practice, BC agents suffer from distribution shift in high dimensions, where small deviations in the learned policy quickly accumulate during deployment and lead to significantly different trajectory distribution relative to the expert (Ross & Bagnell, 2010; Ross et al. 2011).
2.2 INVERSE REINFORCEMENT LEARNING
An alternative indirect approach imitation learning is based inverse reinforcement learning (IRL). Here, the goal is to infer reward function for the expert and subsequently maximize the in- ferred reward to obtain policy. For brevity, focus adversarial imitation learning approaches
2

---

Under review as a conference paper at ICLR 2021
to IRL (Goodfellow et al. 2014). These approaches represent the state-of- the-art in imitation learn- ing and are also relevant baselines for our empirical evaluations.
Generative Adversarial Imitation Learning (GAIL) is IRL algorithm that formulates imitation learning as an "occupancy measure matching" objective w.r.t. a suitable probabilistic divergence (Ho) & Ermon 2016). GAIL consists of two parameterized networks: (a) policy network wo (genera- tor) which is used to rollout agent trajectories (assuming access to transition dynamics), and (b) a discriminator D& which distinguishes between "real" expert demonstrations and "fake" agent tra- jectories. Given expert trajectories TE and agent trajectories TB, the discriminator minimizes the cross-entropy loss:
(R) == -Erg [log (TE) - Ere [log(1 Do(To))]
We then feed the discriminator output log(1 (s, a)) the inferred reward function to the gen- erator policy. The policy parameters can be updated via any regular policy optimization algorithm for the RL objective, e.g. Ho & Ermon (2016) use the TRPO algorithm (Schulman et al. 2015). By simulating agent rollouts, GAIL seeks to match the full trajectory state-action distribution of the imitation agent with the expert as opposed to BC which greedily matches the conditional distribution of individual actions given the states. In practice, GAIL and its variants (Li et al. [017]] Fu et al.) 2017) outperform BC but might need excessive interactions with the training environment for sam- pling rollouts during training. Crucially, both BC and IRL approaches tend to fail catastrophically in the presence of small perturbations and nuisances at test-time (de Haan et al. 2019).
3 THE IMPLANT FRAMEWORK
In the previous section, we showed that current IRL algorithms consider reward inference as an aux- iliary task for imitation learning. Once the agents have been trained, the reward function is discarded and the learned policy is deployed Indeed, if the RL step post reward inference (e.g., generator updates GAIL) were optimal, then the reward function provides no additional information about the expert relative to the imitation policy. However, this is far from reality, as current RL algorithms can fail to return optimal solutions due to either representational or optimization issues. For ex- ample, there might be a mismatch in the architecture of the policy network and the expert policy, and/or difficulties in optimizing non-convex objective functions. In fact, the latter challenge gets exacerbated in adversarial learning scenarios due non-stationary reward.
Building off these observations, we propose Imitation with Planning at Test-time (IMPLANT), an imitation learning algorithm that employs the learned reward function for decision-time planning. The pseudocode for IMPLANT is shown in Algorithm I We can dissect IMPLANT into two se- quential phases: training phase and planning phase.
Training phase: We can invoke any IRL algorithm, e.g., GAIL to optimize for stochastic imi- tation policy Ty by optimizing for some inferred reward function Tg. Additionally, we also train a parameterized value function V₂ this stage. Value function estimation is often subroutine for many RI. algorithms including those which are used to update the policy within the IRL. setup, such actor-eritic methods (Konda & Tsitsiklis 2000). For such algorithms, learning value function does not incur any additional computation.
Planning phase: At decision-ting -tie,, we use the imitation policy along with the learned value and reward functions for closed-loop planning. We build our planner based preei-pre control (MPC) (Camacho & Alba 2013). At any given state 8, and time 2 O, we are interested in choosing action sequences for trajectories which maximizes the following objective:
arg max R(T) ag 0r+
Σ P=t
* (r(s,,
ar.0r-1,"
where Sp ~ Po and 8:41 T(- ar) for all
In some cases, the reward function is transferred to new environment and a new policy is learned using the reward function and additional interactions with the new environment. See Section 5 for further discussion.
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Imitation with Planning at Test-time (IMPLANT)
Input: available dynamics T, expert demonstrations TE. rollout budget B, rollout policy n, horizon H, test start state 8p
Note: For brevity, omit relevant MDP parameters in the list of arguments
Function Train (Tg):
Learn policy Te and a reward function ra with any existing IRL algorithm given access to
demonstrations TE e.g., GAIL
Estimate value function Vy. for TTg
return Tp. rd. Vei
Function Plan( (8, To. Vi. Táe H, B, T):
Set 8 =5 while agent is alive do
10
// Agent planning
Sample B trajectories {r(1) ( r(B)} of max length H starting from using
11
dynamics T: sample the first action ap no. and sample subsequent actions from π
a=0 n, for E {I.
(1)
Estimate trajectory returns (T)((((0) using VP and no (see Eq. 4
12
arg max, &e (T!10) and execute the best action dp (i*)
13
Pick best action index
// Environment feedback
Observe true reward r(s, 4g ((")) and true next state s ~ T(-ss, (f"))
15
16
end
This objective has also been applied for model-based RL with learned dynamics model and black- box access to the rewards function (Nagabandi et al. 2018 Chua et al., 2011)). Unlike the RL setting however, we do know the reward function for imitation learning. The true dynamics model may be available for planning (i.e., f T) as in Ho Ermon (016)) or can estimated from expert demonstrations online interactions (Baram et al.) 2016). Hence, we can do rollouts before in regular model-based RL but need to rely on learned estimates for the reward function. In particular, we use the learned reward function ro up fixed horizon I! and terminal value function Ve thereafter to estimate the trajectory return as:
H-11
R(T) R
Σ P=t
(0(0 + " (v = (T).
(4)
Substituting Eq. in Eq. 3 obtain surrogate objective for optimization. To optimize this surrogate, we propose a variant of the random shooting optimizer (Richards) 2005) that works as follows. At the current state st, first sample set of B candidate actions independently from the imitation policy. For each candidate action, we estimate a score based on their expected returns by performing rollout(s) of fixed-length 11, The rollout policy ı from which sample all subsequent actions could be random (potentially high variance) or the imiiation policy (potentially high bias) or mixture. our experiments, we obtained consistently better performance with using my as the rollout policy π. For each trajectory, we estimate its return via Eq. and finally, pick the action with the largest return.
Consistent with the closed-loop nature of MPC, we repeat the above procedure at the next state Doing so helps correct for errors in estimation and optimization in the previous time step, albeit at the expense of additional computation. The algorithm has two critical parameters that induce similar computational trade-offs. First, we need to specify budget B for the total number of rollouts. The higher the budget, larger is our search space for the best action. Second, we need to specify a planning horizon H. For larger lengths, we need extra computation that also involves interactions with the dynamics of the environment and rely more the learned reward function than the value function for estimating returns in Eq. 4 However, since the rollouts are independent, we mitigate additional computational costs by parallelizing the rollouts. While this parallelization
4

---

Under review as a conference paper at ICLR 2021
Table 1: Average return of imitation learning algorithms on MuJoCo benchmarks.
Hopper
HalfCheetah
Walker2d
Expert
3570
891
3593
BC
127 1 85
427 + 131
258 + 262
BC-Dropout
169 +: 105
542 1: 275
1622 1 861
GAIL
3506 + 337
951 1 282
2780 + 1007
GAIL-Reward Only
319 = 123
5= 117
56 146
IMPLANT (ours)
3633 - 50
1193 + 143
3360 + 442
indeed bottlenecked by the rollout with the largest horizon, in all of our experiments, we perform rollouts of fixed length and the horizon that corresponds to the optimal performance is relatively small (10 ~ 50). Thus, the gains due to parallelization are significant.
In the next section, we present our empirical validation that also investigates the effect of planning horizon on the performance of the algorithm in greater detail.
EXPERIMENTS
Our experiments aim to evaluate the performance of IMPLANT as standalone imitation algorithm in two kinds of settings. First, we evaluate its performance in the default "no-transfer" setting, where the agent trained and tested in the same environment. Second, we emphasize the robustness of IMPLANT by evaluating its zero-shot generalization performance in environments where the test dynamics are a perturhed version of the training dynamics. We consider such perturbations: causal confusion dd Haan al. 2019), motor noise (Christiano et al. 2016), and transition noise. We will describe each of these setups subsequently alongside the results. For all transfer settings, we only assume access to the training dynamics Terain and use it as / for planning. At test-time, no additional interactions is allowed, nor do we have access the test dynamics Trast-
Setup. We evaluate our approach on MuJoCo enviroments in OpenAl Gym (Brockman et al., 2016: Hopper, HalfCheetah, and Walker2d. The expert data used for benchmarking imitation learning on these environments is publicly available" We replicate the experimental setup of Ho & Ermon (2016) by fixing a limited number of expert trajectories used for training, as well as sub-sampling expert trajectories every 20 time steps. All results are averaged over 5 runs of each algorithm with different seeds. We provide further details Appendix [
Baselines. As we observed in Algorithm I IMPLANT can employ any IRL algorithm under the hood. For our experiments, we consider GAIL (Ho & Ermon 2016) as the IRL algorithm of choice both input for IMPLANT and consequently, as the closest baseline of interest. GAIL is amongst the current state-of-the-an methods for imitation learning; see Section 2.2 for a detailed description. For every environment, report results for IMPLANT using single set of hyperparameters for the rollout budget and planning horizon. We provide further details in Appendix
In addition, also consider Behavioral Cloning (BC) bascline; Section for detailed description, Further, we also tested two variants of GAIL and BC that employ dropout (Srivastaval et al. 2014) to demonstrate the limited utility of standard regularization techniques in countering the challenges due to low data and test noise. In fact, GAIL with dropout completely failed to learn in the adversarial setting on any of the environments; for brevity, exclude it from presentation.
Last, we include a "Gwll--Rewrr Only" ablation baseline where we discard the imitation policy (generator) of GAIL during execution and instead, only use the inferred reward model (discrimina- tor) in conjunction with a random policy for decision-time planning. This directly contrasts with the GAIL baseline, which by default only uses the generator. On the other hand, IMPLANT uses both the generator and discriminator for imitation via decision-time planning.
basel ines
5

---

Under review as a conference paper at ICLR 2021
Table 2: Average return of imitation learning algorithms in causal confusion setting.
Hopper
HalfCheetah
Walker2d
Expert
3570
891
3593
BC
209 + 121
331 1: 141
119 1 206
BC- BC-Dropout
162 = 108
548 + 175
700 - 433
GAIL
579 484
699 200
613 + 465
Aewlrdd Only
515 = 302
-82
57 146
IMPLANT (ours)
1717 1262
827 375
807 + 395
4.1 IMITATION WITH LIMITED EXPERT TRAJECTORIES
With a relatively low number of expert trajectories, it has been shown by Ho & Ermon (2016) that GAIL can achieve near-expert performance in almost all these environments. We evaluate the performance of IMPLANT using the lowest number of expert trajectories tested in prior work. The results are shown in Table [ We find that IMPLANT can achieve near-optimal performance all these environments, including Walker2d where GAIL performs much worse than the expert. As expected, BC and BC and BC- BC-Dropout perform poorly in this setting. GAIL-Reward Only exhibits the poorest performance suggesting the benefits of explicitly learning a parametric policy.
4.2 ZERO-SHOT TRANSFER SETTING: CAUSAL CONFUSION
de Haan et al. 2019) observed that imitation learning approaches are susceptible causal confu- sion, Lee. their performance deteriorates significantly the presence of nuisance confounders in the state representation. To demonstrate this phenomena empirically, de Haan et al. (2019) further propose a challenging example setup in which the nuisance can be created by appending the agent's observation with its action from the previous time step. A standard imitation agent trained in this environment will learn to copy the previous action (since successive actions are highly correlated in expert demonstrations), falling prey to causal confusion. At test-time, the agent's performance drops drastically if the appended action is replaced by random noise (i.e., the confounding is removed). We refer the reader to de Haan et al. (2019) for further details and analysis.
We now benchmark the zero-shot test performance of IMPLANT under the same setup in Table 2 While all baselines, including GAIL, fail drastically due the confounding nuisance, IMPLANT significantly more robust in all environments. We can visualize the agent performance qualitatively in Figure 3 Note that we provided the IMPLANT agent access to only the confounded dynamics for decision-ting planning. The algorithm is hence zero-shot, unlike the proposed solutions Fu ctt al. (2017) and de Haan et al. (2019) which require further interactions with the non-confounded test environment for recovery.
4.3 ZERO-SHOT TRANSFER SETTING: OBSERVATION AND ACTION NOISE
Next, we consider two kinds noisy perturbations motivated by real-world applications sim2real.
First, we perturb the intended actions via motor noise (Christiano et al. 2016), e.g., due to imperfect hardware, a real robot might execute a noisy version of the action proposed by the agent. We implement this scenario by adding independent Gaussian noise to each dimension of the executed i.i.d.
action at test-time, i.e., Caction ~ (0, and we vary the noise stddev [0.1, 0.5, 1.0].
Second, we consider transition noise due to an imperfect dynamics model for simulator that may not be able to account for perturbations due to drag or friction. Hence, we specify the test-time dynamics to be perturbed noisy version of the training dynamics. Similar motor noise, we d. NN0 with [0.001, 0.002, 0.005, 0.01].
sample the transition noise as Cramsition
For case of visualization, we show the normalized performance of the different algorithms in Fig- ure See Appendix B for raw absolute results. We also include another competitive baseline GGALL-Expert-Noise relevant to this scenario that artificially adds independent noise to the demon- stration data for every gradient update during GAIL. training. For very high noise level, any

---

Under review as a conference paper at ICLR 2021
GAIL
(b) GAIL
GAIL-Reward Reward Only
(c) IMPLANT (ours)
2500
GAIL Reward Only IMPLANT (ours)
2000
Return
1500
500
1000
Time
(d) Return over time of the above agents
Figure Trajectory visualization for Hopper environment in causal confusion setup at test-time. While all agents start from the same state, only IMPLANT can effectively hop forward. All agents are trained in the confounded setting and tested in the non-confounded setting.
algorithm will naturally deteriorate in performance due to significant shift in training and testing environments. More importantly, for modest noise levels, we find that IMPLANT outperforms the baselines in almost all cases, highlighting its robustness.
4.4 EFFECT OF PLANNING HORIZON
Finally, we analyze the effect of planning horizon on IMPLANT performance in the same setup Section 4.1, Specifically, we vary the planning horizon H [0, 10, 50, 100 for a rollout budget B 10. The normalized performance curves are shown in Fig-
Retume (normafzeell
ure 3 When the planning horizon we only rely on the terminal value function for estimating returns. Conversely, for large planning horizons (e.g., H = 100), the returns are dominated by rewards ac- cumulated at every time step. We observe that picking neither very large horizon (h 2 100) nor a very small one (h = 0) re- sults in optimal performance, suggesting imperfections in both the
Hoppe
Hottzn
Figure 3: Effect of varying planning horizon H on IM- PLANT performance.

---

Under review as a conference paper at ICLR 2021
S-Gropoct
(nammaleee)
3.0
Drepppu
Ratumms (nromativec)
RI
Retums (normalieed
MdA snoin Dayy
E.
BAll Reworireey
Da1 Prwinieeey Cxpet-loie
GAL Eol-Eccet-tiiee
0.5
0.4
0.2
0.3
Sigma
Siama
Sigma
(a) Hopper: Motor Noise
(b) HalfChectah: Motor Noise
(c) Walker2d: Motor Noise
Returns (rormalted))
1.0
Cropoct
1.0
BC Drepout
Returee (normalivd)
BC Drepsu
GMA
Ravand Only
Renard Only
CALL Exper-
GAll Espurt Stee
GAL Eolettttthoo home
(eus)
0.4
0.2
60-
Sigma
Sigma
Sigma
(d) Hopper: Transition Noise
(e) HalfCheetah: Transition Noise
(f)
Walker2d: Transition Noise
Figure 2: Average return of imitation learning algorithms on motor and transition noise settings,
learned reward and value functions and the sweet-spot for the planning horizon is typically between
the extremes.
5 DISCUSSION & RELATED WORK
Traditionally, algorithms for imitation learning fall into one of two categories. They are either completely model-free during both training and execution, in behavioral cloning and its vari- ants (Pomerleau) 1991 Ross et al. 2011). Altemattiely.. they are utilize dynamics and (inferred) rewards models during training, but are model-free during execution, as in inverse reinforcement learning (Ng et al., 2000, Ratlif et al., 2006), Ziebart et al., 2008). Our work introduces a novel model-based perspective to imitation learning where the reward and tran- sition models are used both during training and execution. Borrowing the terminology from Sutton & Barto (2018), the use of models for the MDP during training and execution are also referred to as background and decision-time planning respectively.
in the sense that they
While imitation via background planning has showed immense promise for control in complex en- vironments (Abbeel & Ng] 20044] Ratliff et al. 2009 Ho & Ermon 2016 Choudhury et al. 2018)) we showed that decision vocision-time planning in IMPLANT can further improve the data efficiency and robustness of the learned policies. There have also been several alternate attempts for characterizing and enhancing the robustness of imitation policies. For example, Fuu et al. (217)) seek robustness in the sense recovering the true reward function via adversarial imitation learning and transfer the inferred reward function to external dynamics in the non-zero shot setting. A significant body of work also considers IRL approaches that can accurately capture the uncertainty in the reward function for safe deployment (Zheng et al,, 2014 Brown et al. 2018 Huang et al., 2018 Lacotte al., 2019 Brown al.) 2020). While these utility- based notions are distinct from ours, they are complementary approaches to robustness that could be combined with IMPLANT in future work.
Given the synergies between generative modeling and imitation learning as exemplified in GAIL (Ho & Ermon 2016), improvements in the former often translate into improved imitation, e.g., the use of autoencoder embeddings to improve diversity (Wang et al. 2017), better loss functions and architec- tures for stable GAN/GAIL training (Pfau & Vinyals 2016 Kuefler et al. 2017 Li et al. 2017). etc. These modifications are conceptually complementary to the key contribution IMPLAN T to incor- porate decision time planning and are likely to further boost our performance. In fact, decision-timg planning in IMPLANT can be viewed as filtering of trajectories sampled from the policy network. This is similar to recent work in using importance weighting for improving sample quality of
8

---

Under review as a conference paper at ICLR 2021
generative model (Grover & Ermon 2017 Azadi et al. 2018 Grover et al. 2019). However,
solution tailored towards sequential decision making and deterministically picks the best outcome in line with model predictive control, unlike importance weighting filters.
CONCLUSION
We presented Imitation with Planning at Test-time (IMPLANT), new algorithm for imitation learn- ing that uses decision-time planning to mitigate compounding errors of any base IRL algorithm. Unlike existing approaches, IMPLANT is truly model-based in the sense of utilizing the inferred rewards and dynamics model both during training and execution. We demonstrated that IMPLANT matches or outperforms existing benchmark imitation learning algorithms with very few expert tra- jectories. Finally, we empirically demonstrated the robustness of IMPLANT via its impressive per- formance zero-shot generalization several challenging perturbation settings.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discrim- inator rejection sampling. arXiv preprint arXiv:/810.06758. 2018.
Nir Baram, Oron Anschel, and Shie Mannor. Model-based adversarial imitation learning. arXiv preprint arXiv:1612.02213 2016.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Daniel S Brown, Yuchen Cui, and Scott Niekum. aiske active inverse reinforcement learning. In Conference Robot Learning, pp. 362-372, 2018.
Daniel Brown, Scott Niekum, and Marek Petrik. Bayesian robust optimization for imitation learn- ing, arXiv preprint arXiv:2007. 12315 2020.
Eduardo Camacho and Carlos Bordons Alba. Model predictive control. Springer Science &
Business Media, 2013.
Sanjiban Choudhury, Mohak Bhardwaj, Sankalp Arora, Ashish Kapoor, Gireeja Ranade, Sebastian Scherer, and Dehadeepta Dey. Data-driven planning via imitation learning. The International Journal of Robotics Research, 37(13-14):1632-1 2018.
Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojeiech Zaremba. Transfer from simulation to real world through learning deep inverse dynamics model. arXiv preprint arXiv:1610.03518. 2016.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, pp. 4754-4765, 2018.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, pp. 11698 11709, 2019.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: 2017.
//github.com http:/u....o.ooppppaaaaaaeeeaeie
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv:1710.11220 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.

---

Under review as a conference paper at ICLR 2021
Aditya Grover and Stefano Ermon. Boosted generative models. arXiv preprint arXiv:1702.0843: 2017.
Aditya Grover, Jiaming Song, Ashish Kapoor, Kenneth Tran, Alekh Agarwal, Eric J Horvitz, and Stefano Ermon. Bias correction of learned generative models using likelihood-free importance weighting. In Advances in Neural Information Processing Systems, pp. 11058-11070 2019.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565-4573, 2016.
Jessie Huang, Fa Wu, Doina Precup, and Yang Cai. Learning safe policies with expert guidance. In Advances in Neural Information Processing Systems, pp.
2018.
Vijay Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pp. 1008-10000000 1014, 2000.
Alex Kuefler, Jeremy Morton, Tim Wheeler, and Mykel Kochenderfer. Imitating driver behavior with generative adversarial networks. In 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 204-211. IEBE, 2017.
Jonathan Lacotte, Mohammad Ghavamzadeh, Yinlam Chow, and Marco Pavone. Risk-sensitive generative adversarial imitation learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2154-2163. PMLR, 2019.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, pp. 3812-3822, 2017.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Andrew Y Ng, Stuart J Russell, et Algorithms inverse reinforcement learning. In Icml, volume 1, pp. 2, 2000.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. arXiv preprint 1eXiv:/8/1.067// 2018.
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomputing, 71(7-9): 110-110011001 1190, 2008
David Pfau and Oriol Vinyals. Connecting generative adversarial networks and actor-critic methods. arXiv preprint arXiv:1610.01940 2016.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral computation, 3(1):88-97) 1991.
Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Nathan D Ratliff, Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning, pp. 729-736, 2006.
Nathan D Ratliff, David Silver, and Andrew Bagnell. Learning to search: Functional gradient
techniques for imitation learning. Autonomous Robots, 27(1):25-53, 2009.
Arthur George Richards. Robust constrained model predictive control. PhD thesis, Massachusetts Institute of Technology, 2005.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the
thirteenth international conference on artificial intelligence and statistics, pp. 661-668, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statisties, pp. 627-635, 2011.
10

---

Under review as a conference paper at ICLR 2021
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1958 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Ziyu Wang, Josh $ Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems, pp. 5320-5329, 2017.
Jiangchuan Zheng, Siyuan Liu, and Lionel Ni. Robust bayesian inverse reinforcement learning with sparse behavior noise. In Proceedings of the Twenty-Fighth AAAI Conference Artificial Intelligence, pp. 2012221922020 2205, 2014.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-143 Chicago, IL, USA, 2008.

---

Under review as a conference paper at ICLR 2021
Table 3: Detailed information on environments
Environment
Observation space
Action space
Hopper
Box(11,) Box(17.)
Box(3, )
HalfCheetah
Box(6. )
Walker2d
Box(17.)
Box(6,) )
Table Detailed information on expert data used in training
Environment
# of trajectories # of state-action pairs
Hopper
200
HalfCheetah
200
Walker2d
1000
A ADDITIONAL EXPERIMENTAL DETAILS AND SETUPS
A.1 ENVIRONMENTS AND EXPERT DATA
As mentioned above, we consider continuous tasks from OpenAl Gym (Brockman al. 2016) simulated with MuloCo (Todorov et al., 2012): Hopper, HalfCheetah, and Walker2d. We acquire expert data from OpenAl Baselines (Dhariwal et 2017). Table lists more detailed information about each enviromment, and Table contains information about the expert demonstrations we use for training all of the agents.
A.2 HYPERPARAMETERS AND NETWORK ARCHITECTURES
We 2-layer MLP with tanh activations and 64 hidden units for all of policy networks. For BC, we use a learning rate of 10-1 across all environments. We use a dropout rate of 0.2 for our BC-Dropout agent. The hyperparameters. of GAIL are listed in Table For IMPLANT agent, we directly utilize the value function, reward function, and policy from a trained GAIL agent. In all settings, we choose B 20 and Hl 50 for Hopper, B 2 and // 10 for HalfCheetah and Walker2d to plan.
B ADDITIONAL RESULTS
The complete results for Section can be found in Table g
Table 5: Detailed information on GAIL's hyperparameters
Parameters
Hopper
HalfCheetah
Walker2d
Discriminator network
100-100 MLP
100-100 MLP
100-100 MLP
Discriminator entropy coeff.
0.01
0.01
0.01
Batch size
1024
1024
1024
Max
0.01
0.01
0.01
CG steps/damping
10, 0.01
10, 0.1
10, 0.1
Entropy coeff.
0.0
0.0
0.0
Value fn. steps/step size
3, 3e-4
5,1e-3
5,1e-3
Generator steps
3
3
3
Discriminator steps
1
1
0.98
0.97
0.97
0.99
0.995
0.995
12

---

Under review as a conference paper at ICLR 2021
Table 6: Raw results of Figure 2 in Hopper environment
(a) Hopper with motor noise
Sigma
0.0
0.1
0.2
0.5
1.0
BC
127 + 85
114 + 64
117 + 61
179 + 105
123 82
BC-Dropout
169 + 105
156 + 85
163 = 95
158 + 89
142 94
GAIL.
3506 337
2572 1008
1360 + 687
598 +08+220 243
128
GAIL-Reward Only
319 + 123
293 + 87
268 + 92
160
49 + 54
GAIL-Expert-Nooee
3602 = 46
2716 - 921
1449 - 667
618 264
218 + 136
IMPLANT (ours)
3633 = 50
3209 + 714
1764 + 856
596 - 234
269 + 130
(b) Hopper with transition noise
Sigma
0.0
0.001
0.002
0.005
0.01
BC
127 = 85
123 = 81
116 = 65
98
114
BC-Dropout
169 = 105
175 113
165
161
151
GAIL
3506 + 337
3209 =
2672 = 1012
137
616
GAIL-Reward Only
319 + 123
804
665
349
ExoL-Expert-Noise
3576 + 36
2966 + 1089
2160 + 1311
875
323
IMPLANT (ours)
3633 + 50
3557 313
2844
1301 810
598 467
Table 7: Raw results of Figure HalfCheetah environment
(a) HalfCheetah with motor noise
Sigma
0.0
0.1
0.2
0.5
1.0
BC
427 131
377 + 139
289
118
-87 + 81
7703 + 66
BC-Dropout
542 + 275
555 + 232
409 + 194
-6 85
-655
GAIL
954 282
892 287
764 200
172
121
6677
GAIL-Reward Only
5 + 117
-11 111
-39 + 100
-214
-732 53
GAIL-EEpert-Noiss
933 216
888 + 195
740 179
127 + 124
745 745
IMPLANT (ours)
1193 + 143
1025 = 115
863 + 94
260 + 72
-652 + 65
(b) HalfCheetah with transition noise
Sigma
0.0
0.001
0.002
0.005
0.01
BC
427 + 131
433 = 126
430 + 136
395 + 131
361 + 120
BC-Dropout
542 + 275
549 252
509 + 276
505 - 249
447 223
GAIL
282
942 472
946 263
900
808
GAIL-Reward Only
5
616
815
0 112.
-11
1103 265
934 201
914 + 209
891 + 186
828 192
IMPLANT (ours)
1193 + 143
1137 + 132
1089 + 123
1021 + 115
923 + 103
13

---

Under review as a conference paper at ICLR 2021
Table 8: Raw results of Figure in Walker2d environment
(a) Walker2d with motor noise
Sigma
0.0
0.1
0.2
0.5
1.0
BC
258 + 262
402 444
384 + 368
298 + 191
172 + 128
BC-Dropout
1622 + 861
1386 + 826
1498 + 901
918 + 450
339 129
GAIL
2780 + 1007
2560 1047
1893 + 1045
576
234 + 133
GAIL-Reward Only
56 + 146
33 33+100
25 94
13
4
GxIL-Expert-Noise
2253 = 1081
2245 1098
1448 + 891
495 221
247 144
IMPLANT (ours)
3360 + 442
3251 + 682
2816 + 1019
701 301
228 + 148
(b) Walker2d with transition noise
Sigma
0.0
0.001
0,002
0.005
0.01
BC
258 +: 262
318 + 338
329 + 365
298 + 274
233 + 230
BC-Dropout
1622 861
1559 + 848
1380 + 819
1123 937
584 658
GAIL
2780 + 1007
2746 + 1015
2383 + 1136
1168 958
464
GAlL-Reward Only
+ 146
424
53 128
36 - 120
+ 129
MAIL-Expert-Noiss
2627 + 927
2244 - 1098
1832 + 1125
935 - 812
414
IMPLANT (ours)
3360 442
3299 + 558
3170 + 725
1899 = 1249
600 + 644
14


Reviews:
[{"review_id": "n8IYeFwvoPh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#######################################################################\n\nSummary:\n \nThe paper considers imitation learning problem in the presence of small perturbations and nuisances at test-time. In particular, it proposes Imitation with Planning at Test-time (IMPLANT), a new algorithm for imitation learning that incorporates decision-time planning within an inverse reinforcement learning algorithm. To counteract the imperfection due to policy optimization in RL step, IMPLANT uses the reward function estimated in the IRL step for decision-time planning. The effectiveness of the proposed method has been empirically evaluated on two kinds of setups, i.e., the default ‘no-transfer’ setting and the ‘transfer’ setting where the test dynamics is a perturbated version of the training dynamics. \n\n#######################################################################\n\nReasons for score: \n \nOverall, I vote for a weak acceptance. I like the idea of using test-time planning for tackling the test-time perturbation in imitation learning. My major concern is about the clarity of the paper and some additional environments (see cons below). Hopefully the authors can address my concern in the rebuttal period. \n \n#######################################################################\n\nPros: \n \n1. The paper takes one practical issue of imitation learning: test-time perturbation.  \n \n2. For me, the proposed IMPLANT method is novel for the issue of test-time perturbation in imitation learning. Specifically, IMPLANT builds on top of GAIL, and learns an additional value estimation during training, and utilizes the estimated reward function, value function for closed-loop planning based on model-predictive control (MPC). The design is interesting.\n \n3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. In particular, IMPLANT outperforms the BC and GAIL-based baselines in the ‘non-transfer’ setting of imitation with limited expert trajectories, and in the transfer setting with causal confusion, motor noise and transition noise. \n\n#######################################################################\n\nCons: \n  \n1. For the motivation, it would be better to provide more details about it, which seems not very clear to me. Particularly, it is unclear why the planning is introduced, and why such design can address the issue studied in the paper. Additionally, since this paper claims any IRL methods can be used in the training time, I suggest the authors to showcase and discuss the performance with other IRL methods than GAIL at training time. \n\n2. For the zero-shot transition setting in section 4.3, IMPLANT is only robust to the noise with small sigma. It would be better to explain why the IMPLANT can achieve this level of robustness and which issue limits the performance in the small perturbation. This might help the authors to further improve the robustness of IMPLANT. \n \n3. Although the proposed method has been evaluated in diverse settings, the environments are limited to three continuous control tasks. It would be more convincing if the authors can provide more cases with both continuous and discrete action space in the rebuttal period. \n \n#######################################################################\n\nQuestions during rebuttal period: \n \nPlease address and clarify the cons above. \n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kT6gbGvGWz-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "IMITATION LEARNING REVIEW\n============================\n\nThe gist of the paper is to improve IRL with the so-called IMPLANT algorithm which is supposed to improve control policies in learning controllers.\n\nPROS\n====\nIt would be nice to have a framework for minimizing model estimation errors in imitation learning. This paper presents a framework for mitigating the learned dynamics during imitation learning using an augmented cost fuction at test time. The authors called this IMPLANT> \n\nCONS\n====\n\nI do not see how this algorithm is an MPC framework: where is the action/control law computed in lines 7-16 of Algorithm 1?\nWhat did the authors mean in line 11 when they wrote, \"First action a_0^{(i)} ~ \\pi_\\theta ... \". \n\n* How was \\pi in line 11 computed?\n\n* Is there an upper bound on the length of the horizon H?\n\n* If there is, how does the policy perform for slowly-varying trajectories at lower horizons versus fast-changing trajectories at longer planning horizons?\n\n\n* I am concerned that the reward function in equation (4) is just a restatement of the optimality principle and do not see why this is being rebranded as a new algorithm.\n\n* Of course, the rollout policy would guarantee better performance. What is the predicate for choosing a *random rollout policy* as the authors mentioned on line 4? What is the point of using a mixture?\n\n* Since you mentioned that the rollouts can be parallel-wise executed, did you give any thoughts to the fact that the control law has to be tightly scheduled in the feedback before the next iteration of the MPC is run?\n\nGrammar errors \n==============\n\nThere were a few sentence structures in the paper that could use some revision e.g. \"A\" popular class of approaches\"-->approach", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gx88r9Viblb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method to enhance imitation learning by using MPC on the reward function learned by an imitation learning approach. The chosen MPC approach uses the learned policy to generate candidates for a search process that maximizes the reward. To learn the reward function, the authors propose to use GAIL. Besides showing improved performance in a typical benchmark scenario, the authors compare their method with GAIL in various scenarios where some form of transfer is required. The writing is generally clear and easy to understand.\n\nHistorically, IRL methods relied on given or learned transition models as they relied on solving for a policy in an inner loop. Recently, we have seen a handful of methods introduced which do not have that short-coming and with it, we have seen a shift towards model-free methods. It is generally understood in IRL that the learned reward function can be used with any reinforcement learning or with any planning algorithm such as MPC but while some model-based approaches have been introduced, the merits of model-based imitation learning have not been thoroughly compared to recent model-free methods to my knowledge. While this makes the paper relevant, the use of a single existing MPC approach with a learned reward function is a relatively minor contribution. My main issue, however, lies in the evaluation as the dynamics model is generally just given to the agent. While the authors acknowledge that the dynamics model could be learned, that is not the case in any of the experiments and it is of little surprise that a MPC approach with perfect knowledge of the transition dynamics is able to outperform reinforcement learning approaches that have to learn from interaction. Furthermore, the authors claim their approach to be zero-shot in the transfer-experiments in comparison to AIRL. This claim is hardly defensible when IMPLANT is handed perfect knowledge of the new dynamics while AIRL has no knowledge of the new dynamics.\n\nAnother major issue with this work stems from an apparent misunderstanding of the role of the discriminator in GAIL. While GAIL treats the discriminator as a reward function in an inner loop, it is not a real IRL method. As the policy converges to the expert’s policy, the learned discriminator can no longer tell the data apart and converges to a constant. On a practical level, the proposed MPC approach uses the learned policy to generate proposals and therefore corresponds more to an iterative step on this policy. This explains why the agent is able to achieve higher scores on benchmark tasks; however, the discriminator by itself is not generally useable as a reward function like the authors suggest. This is underlined by the results that “GAIL - reward only” fails to learn. Instead, the authors should use true IRL methods as the basis for their approach.\n\nFinally, the proposed evaluation is limited as the authors choose 3 of the easiest tasks from the mujoco benchmark. The lack of results on humanoid as the hardest task is problematic.", "extra_scores": {}, "extra_text": {}}, {"review_id": "F1Cel71_t2W", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "__Summary__\nThe paper proposes to combine imitation learning (GAIL) with model predictive control to improve on the policy, especially when the system dynamics are noisier at test time.\nHere, MPC uses the reward function and value function learned by GAIL and assumes access to a model of the test-dynamics.\nThe proposed approach, IMPLANT, is compared to GAIL and behavioral cloning on MuJoCo experiments in a standard imitation learning setting and for three slightly modified test scenarios that add either action noise, transition noise, or causal confusing which adds the last actions to the state space during training and replaces this information by noise at test time.\n\n\n__Strong points / weak points__\n+ (+) The paper is well-written and very easy to follow\n+ (+) Using MPC on a reward function that was learned by IRL can be reasonable\n- (-) Lack of contribution / novelty\n- (-) Using the GAIL reward doesn't seem sound. An actual IRL method should be used\n- (-) additional assumptions compared to competitors (access to noisy model, computational tractability of online planning) reduce practicability and are not regarded during the comparisons\n\n\n__Recommendation__\nI recommend rejecting the submission because I can't find a noteworthy contribution.\n\n\n__Supporting Arguments__\n- *Lack of contribution:* I don't want to be gatekeeping, but I really don't see any noteworthy contribution. The standard approach for apprenticeship learning is to learn a reward function by IRL and then optimize this reward function to learn a policy. There is no requirement to use the same algorithm to optimize the reward that used for inferring it. It is not surprising that reoptimizing the reward performs better in face of dynamic perturbations compared to a direct policy transfer; after all, this is one of the main motivations for learning a reward function. It is also not surprising that a model-based MPC approach based on the policy can perform better than directly using the parametric policy, even when there are no changes in the dynamics. So, I actually don't even see the research question here. There can be a small contribution due to the empirical evaluation, however, the chosen baselines perform direct policy transfer and, thus, do not seem that interesting.\n\n- *Using GAIL for IRL:* The paper repeatably refers to GAIL as an IRL approach. However, GAIL does not infer a reward function in accordance with the problem formulation of IRL. The \"reward function\" used by GAIL is only valid for small policy updates of the current policy (generator). *Maximizing* this reward function will in general not result in any reasonable policy. For optimal policy and discriminator, the reward function would even be constant. So using this reward function for MPC doesn't seem sound. The evaluation did show that optimizing the reward function via MPC can be beneficial when using a small enough horizon and when sampling from the parametric policy, but this is not that surprising since the MPC control will in that case tend to remain close to the parametric policy. \n\n- *Additional assumptions:* The paper argues that IMPLANT is \"zero-shot, unlike the proposed solutions of Fu et al. (2017) and de Haan et al. (2019) which require further interactions\" with the test environment. However, I would argue that MPC should be treated like a reinforcement learning method here; it also requires interactions with the test environment (i.e., sampling from the dynamics model). I don't see why it would not be fair to use a policy-parametric RL algorithm (e.g. AIRL + reoptimizing) instead of MPC for optimizing the reward. Actually, I think that the test setting would then still favor MPC since it requires costly online optimization which is not always feasible in practice.\n\n\n__Questions__\n1. Are there problem settings where IMPLANT is applicable, but IRL+RL (e.g. AIRL + reoptimizing) is not?\n2. The paper mentions that a random rollout policy for MPC performs worse than using the learned policy. Is this also the case for a very large number of rollouts? Is the resulting control also worse in terms of the learned reward function or only w.r.t. the true reward function? \n\n\n__Additional Feedback__\nMy main concern is that the current submission has too little contribution and, thus, I think that more research needs to be done before publishing a paper on this topic. Performing MPC after IRL seems way too little as a paper story. It's hard to suggest a direction for further research because I do not see the research question to start with. Regarding the combination of IRL and MPC, it might be interesting to investigate using MPC within the IRL loop. For example, it can be challenging to learn reward functions for multimodal demonstration based on a parametric (and usually unimodal) policy. \n\nThe current paper could be improved by using an IRL reward and by comparing the approach with IRL+reoptimizing. However, at least for me, this would not affect my recommendation because these changes do not address my main concern.\n\nI hope this review is not too discouraging; the paper does have strong points in the presentation.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers highly appreciated the replies and the additional experiments. We also had a private discussion on the paper. To summarize: the replies alleviated quite a few concerns, however the consensus was that the paper still does not meet the bar for a highly competitive conference like ICLR.

The idea of combining MPC (on a 'wrong' model)  with a learned cost function is very interesting and a promising direction. On the downside the reviewers are still not entirely convinced about the contribution and believe that the paper requires a significant re-write to incorporate the discussed points as well as an additional round of reviews.
Decision: Reject