Model Predictive Adversarial Imitation Learning for Planning from Observation
Authors: Tyler Han
Yanda Bao
Bhaumik Mehta
Gabriel Guo
Sanghun Jung
Anubhav Vishwakarma
Emily Kang
Rosario Scalise
Jason Liren Zhou
Bryan Xu
Byron Boots
Venue: ICLR 2026 Poster (2026)
Keywords: Imitation Learning
Reinforcement Learning
Model Predictive Control
Primary area: applications to robotics, autonomy, planning
TL;DR: Towards real-world IRL for robot learning, we propose planning-based Adversarial Imitation Learning which simultaneously learns a reward and improves a planning-based agent through interaction and observation-only demonstrations.

Abstract: Humans can often perform a new task after observing a few demonstrations by inferring the underlying intent. For robots, recovering the intent of the demonstrator through a learned reward function can enable more efficient, interpretable, and robust imitation through planning. A common paradigm for learning how to plan-from-demonstration involves first solving for a reward via Inverse Reinforcement Learning (IRL) and then deploying it via Model Predictive Control (MPC). In this work, we unify these two procedures by introducing planning-based Adversarial Imitation Learning, which simultaneously learns a reward and improves a planning-based agent through experience while using observation-only demonstrations. We study advantages of planning-based AIL in generalization, interpretability, robustness, and sample efficiency through experiments in simulated control tasks and real-world navigation from few or single observation-only demonstration.

Published as a conference paper at ICLR 2026
MODEL PREDICTIVE ADVERSARIAL IMITATION LEARNING FOR PLANNING FROM OBSERVATION
Tyler Han, Yanda Bao, Bhaumik Mehta, Gabe Guo, Sanghun Jung, Anubhav Vishwakarma, Emily Kang, Rosario Scalise, Jason Zhou, Bryan Xu, Byron Boots Paul G. Allen School of Computer Science & Engineering University of Washington
ABSTRACT
Humans can often perform new task after observing few demonstrations by inferring the underlying intent. For robots, recovering the intent of the demon- strator through learned reward function can enable more efficient, interpretable, and robust imitation through planning. common paradigm for learning how plan-from- www-ffrrmmmmostrattiiin involves first solving for reward via Inverse Rein- forcement Learning (IRL) and then deploying it via Model Predictive Control (MPC). In this work, unify these two procedures by introducing planning- based Adversarial Imitation Learning, which simultaneously learns a reward and improves planning- caaned agent through experience while using observation-only demonstrations. We study advantages of planning-based AIL in generalization, interpretability, robustness, and sample efficiency through experiments in simu- lated control tasks and real- world navigation from few or single observation-only demonstration.
INTRODUCTION
Inverse Reinforcement Learning (IRL) offers principled approach to imitation learning by in- ferring the underlying intent, or reward func- funda-
Planning & Control
Ce C
VΦ
tion, that explains expert behavior. mental advantage of IRL is that this reward often readily generalizable beyond the sup- port of the demonstration data, enabling the discovery of new policies through interaction and plans through celf-prediction. Especially when demonstrations are sparse, ambiguous, or suboptimal, IRL's interpretabilityy is particularly well-suited for domains where understand- ing preferences and ensuring reliable plan- ning are essential, such as routing on Google Maps (Barnes al.] (023), socially aware nav- igation (Kretzschmar et al. 2016), and au- tonomous driving (Bronstein al. 2022).
Transitions
Discriminator
Value
Expert
L ($)
Slabes
Btates
Returns
Caca(P)
Figure 1: Model Predictive Adversarial Imitation Learning (MPAIL) learns costs for a planning- based, Model Predictive Control (MPC) agent from observation-only demonstration. Interac- tions with these costs are simultaneously used to learn value function for experience-base based rea- soning beyond the horizon of the planner.
For real-time systems, learned IRL and In- verse Optimal Control (IOC) rewards are typ- ically deployed via Model Predictive Control
(MPC) (Lee et al.) 2022b
Rosbach et al.) 2019 Triest et al. 2023 Das et al. 2021 Lee et al.
2021 Kuderer et al. 2015 Lee et al. 2022a). Here, the offline IRL algorithm iteratively solves
Reinforcement Learning (RL) problem in an inner loop, guided by the current reward estimate. An outer loop then updates this reward to minimize the discrepancy between the agent's 's and the pert's 's behavior. Once training is complete, the resulting reward is integrated with MPC for real-tin time planning and control.

---

Published as a conference paper at ICLR 2026
Adversarial Imitation Learning (AIL) has made significant improvements over IRL in algorithmic complexity and sample efficiency (Ho & Ermon] 2016 Baram et al. 2017). However, the reliance applications with safety constraints (Lee) et al. 2022b Triest et al. 2023 Das et al., (02T]. Further limited by partial observability, these deployments will often prioritize planning using model for the sake of real-time performance,
an RL policy in AIL methods complicttss their
trustworthiness, and interpretability (Han et al. 2024a
Katrakazas et al. 2015 Choudhury et al.
2018
In this work, we derive planning-based AlL, yielding key benefits:
Soaniingfrom--Obervvaa (PFO). Towards interpretable yet scalable imitation learning, a predictive model precludes the need for expert action data and enables access to the agent's optimization landscape. This grants crucial insight and steerability into the agent's s decision making process even as it learns from ambiguous expert data. We further show that this improves out-of-distribuiie
generalization, robustness, and sample efficiency when compared to policy-based AlL. We also demonstrate how policy-based AlL is fundamentally limited by the absence of reward deployment.
2. Unification of IRL and MPC. Otherwise considered independent training and deployment procedures, planning based AIL allows for end-to-end interactive learning of the entire planner. Critical online settings (e.g., dynamics, preferences, control constraints) can thus brought into training while enabling experience- horizon, which demonstrate in this work. We also find this induces more effective adversarial dynamic than polic--lyy-aue policy-based generators when learning from partial observations in the real world.
reasoning beyond the planning
To our knowledge, this work presents the first end-to-end cwwnniing-frommooberrwee work, extending PfO to continuous spaces and interactive learning. By choosing Model Predictive Path Integral control (MPPI) (Williams al., 2017) as the embedded planner, further gain the- oretical perspective on planning-
(PfO) frame-
AIL and its relationship to the seminal GAIL objective (Ho) & Ermon 2016 Torabi al., 2019a). Thus, we name this learning algorithm: Model Predictive Adversarial Imitation Learning (MPAIL {impale)).
2 RELATED WORK
IRL-MPC. figh--dmeensioal continuous control applications often require an online planner for real-time control, trustworthiness, safety, or additional constraints. When using IRL to learn a re- ward, online deployments of these reward functions rely on an independent online MPC procedure. To enable learning local costmaps across perception and control for off-road navigation, [ee et al. (002b) and Triest et al. (2023) similarly propose solving the forward RL problem using MPPI but deploy the reward on a different configuration more suitable for real-time planning and control. This reward deployment framework of IRL-then-MPC is currently the dominant approach for planning in high-dimensional continuous control tasks from demonstration (Lee et al. [2022b] Rosbach et al.
2019 I'riest et al. 2023 Das et al. 2021 Lee al.
2021 Kuderer et al. 2015 Lee al. 2022a).
Masel-Based IRL and Planning-Basse Based RL. Various other works have also explored model-based
AIL (Baram et al. 2016 Bronstein al. 2022 [Snn et al.] (021). However, scope is directed at training stabilization and policy optimization rather than examining planning with the learned reward. When the reward is known, as in RL, planning-based algorithms have demonstrated consid- erable improvement in simulation benchmarks over existing state-of- -fhe--rttte--tttt- RL. algorithms through developments such as: online trajectory optimization, value bootstrapping, latent state planning, policy-like or learned sampling priors, and much more (Hansen et al.) 2024, (hardwayr et al. [021]] Lowrey al. 2019 Jawale et al., 2024). This work's implementation of MPAIL performs online trajectory optimization and value bootstrapping. This work does not implement latent state planning nor a pasecy-based prior to better isolate our investigations in interpretability and planning (Fu et al., 2018 Sun et al. 2021). Other existing integrations of planning-based RL with imitation learning also rely on access to expert actions (Li et al., 2025 Yin et al. 2022). No aforementioned works, save (Jawale et al., 2024), evaluate planning based RL AIL a real world platform. However, we find that is precisely real-world and of settings in which planning and control
most advantageous.
2

---

Published as a conference paper at ICLR 2026
3 MODEL PREDICTIVE ADVERSARIAL IMITATION LEARNING
3.1 THE POMDP SETTING AND THE MODEL. PREDICTIVE AGENT
We adopt the Partially Observable Markov Decision Process (POMDP) to best consider highly desir- able applications of IRL in which partial observability and model-based planning play crucial roles. In an unknown world state Sw E Sw. the agent makes an observation o p(o)s ). From a history of observations Op:t, the Agent perceives its state 8; p(s[ou:)). Actions a E A and states E S together allow the agent to self-predict forward time using its predictive model f Note that these definitions crucially suggest the partial observability of due to the implicit de- pendency on the observation history oor through the agent's 's perception (c.g. mapping (Jung et al.) 2)24 However, partial observations in S are desirably used for demonstrations perform IRL and AIL, as full observation history would quickly become intractable (Triest et al., 2023).
A S.
The planner itself is model predictive agent. It is capable of performing model rollouts (H) 1++H such that 8 f(*rr, and each rollout can thus be evaluated with cost C(T,). The agent's objective to create an H-step action sequence a H, or plan, that best minimizes the plan's 's corresponding trajectory cost. The true costs rewards under which the expert is acting is not known. Towards learning- ceenning-from--oe from- coomoobservation (LfO) and lower-level policies, we consider expert data in which only states available, as actions may be challenging impossible to observe (Torabi et al. 2019b).
{se,, ay Je'=t
3.2 ADVERSARIAL IMITATION LEARNING FROM OBSERVATION
IRL algorithms aim to learn a cost function that minimizes the cost of expert trajectories while maximizing the cost of trajectories induced by other policies (Torabi et al.) 2019a Ho & Ermon 2016). As the problem ill-posed and many costs can correspond to given set of demonstrations, the principle of maximum entropy is imposed to obtain a uniquely optimal cost. It can be shown that the entropy maximizing distribution a Boltzmann distribution (Ziebart al.). The state-only IRL from observation problem can be formulated by costing Kate-transitions c(s, s') rather than state-actions c(s, a) as Torabi et al. 2019a):
IRLFO (TE) arg max - (c)) +
min AH(T) E [c(s, s')]
E [c(s, s')],
cEReXS
TEl
where (c) is a convex cost regularizer, ag is the expert policy, H(-) is the entropy, and II is a family of policies.
Torabi et al. 2019a), this objective can shown to be dual to the Adversarial Imitation Learning (AIL) objective under specifie choice of cost regularizer sb,
As shown in (Ho & Ermon 2016
min max #EIl
Ex [lo (D(s, s s')) + E.g [log(1 D(s, s )] AHH(())
(2)
ssxs
where D(.) is the discriminator function. The exact form of has consequences on the policy ob- jective and differs by AlL algorithm. Now equipped with the optimization objective, continue in our derivation of planning-based AlL by choosing the form of the policy class and reward function.
3.3 CHOOSING A POLICY AND REWARD
To reiterate, we set our sights on the AIL objective (Equation (2)) which aims to simultaneously learn reward and policy from demonstration. However, the formulation remains intimately connected with policy optimization through the assumption of an RL procedure. Towards planning- optimization, we proceed by modifying the RI. formulation in Equation (T) as described in (Ho & Ermon 2016 Fu et al. 2018). Similar to (Bhardwaj et al.), we replace the entropy loss -XH((T) with Hullback-Leibele (KL) divergence constraint the previous policy म.:
min Ex [o(s, s')] + KLL(T
(3)
#ell
Note that this incorporates prior information about the policy (i.e. previous plans) while seeking the next maximum entropy policy. Specifically, shown in Section B.T the closed form solution to
3

---

Published as a conference paper at ICLR 2026
Algorithm Model Predictive Adversarial Imitation Learning
Require:
Expert state- transitions DE {(s,s')
Maximum-Entroopy Planner 7MPPI, Discriminator Dp. Value V 1: while not converged do
2: Collect transitions (s, re(()) E de by running ТMPРI (Alg. 3 in the environment
3:
Update Discriminator parameters 0 with Binary Cross Entropy (BCE) loss:
V Vo [log Do(s, s, s')) + Vo [log Da(s,s'))
(5)
4: Update Value parameters & using estimated returns:
Bgg [(G vo(s))
(6)
5: end while
Equation is xt" (als) (alsse where 7(s, = Eves T(S 8/S, s)c(s, and T(S [++1+1 S' = s) denotes the transition probability from state to s'.
We then observe that a choice of planner satisfies the RL objective as in Equation 3 Model Predictive Path Integral (MPPI) the planner, proven in Section B.I] we solve an equiv- alent problem provided the MDP is uniformly ergodic. Namely, MPPI solves for KL--constranned cost-minimizer trajectories (Bhardwaj et al.))
By choosing
min Erax (C(T) + (L(T(T
(4)
TEll
where C(T) is the discounted cost of trajectory and KL(((T)) is the discounted KL diver- gence over a trajectory.
Model rollouss must practically be limited to some timestep length H, resulting in myopic plans and limiting applications to short-horizon tasks (Bhardwaj et al. 2021). To resolve this, infinite-horizon MPPI bootstraps the final states in the rollout using terminal cost-to-go,
value, function (Bhard- waj et al. 2021; Hatch & Boots 2021 Lowrey et al. 2019) Vo S - I that estimates the expected S, se]. where Ro+1 R(s,, 8(+1). The result of MPPI's approximately global policy optimization at each timestep is what is referred to as the MPPI policy, TMPPI- Section [.I how this formulation can be equivalent the entropy- regularized RI. objective, while also in the observation-only setting. Pseudocode for the MPP procedure can be found in Algorithm 2 as well as for its adaptation as an RI. policy in Algorithm Figure 2 illustrates the policy.
return G of state 8, as G, E, [ 7
Our chosen AIL agent, infinite-horizon MPPI, can be viewed as optimizing for new policy at every state. As the online learning perspec- tive (Wagener et al. 2019), these MPPI opti- mizations can occur online rather than offline part of, for instance, slower actor-eritic up- date. By deconstructing the agent this way, we require not the policy to generalize but the re- ward.
Discriminated Plans
One- Stap Local Pollcy
TMPPI
Plana
Provided the agent, now proceed with se- lecting its objective. Recent work has shown many potential choices of valid policy ob- jectives, each with various empirical trade- offs (Orsini al.) 2021). We found the re- ward as defined in Adversarial Inverse Rein- forcement Learning (AIRL) (Fu et al.) (2018) to be most stable when combined with the value function when applied to MPPI. In the state-only setting, the policy ob- jective becomes r(s, s') = (ss log(1 - D(s,s')) and the discriminator D(s, = do(s 8'). Simply put, the reward the logit of the discriminator r(s, 8') do(s,
Figure 2: Illustration of TMPPi in MPAIL. (1) A set of action sequences (plans) are sampled and rolled out. (2) Plans are costed according to the discrim- inator, shifting the distribution towards the expert. Temperature À optionally decays over episodes, narrowing the distribution.
(3) The policy aMPPI is the result of a Gaussian fit to the optimized plans and their respective first actions.
4

---

Published as a conference paper at ICLR 2026
In short, our choice of MMPI and r(s,s') yields MPAIL. As proven in Section B.2 MPAIL is indeed an AIL algorithm in the sense that it minimizes divergence from the expert policy. The procedure (Algorithm TD itself closely resembles the original GAIL procedure. However, upon updating the value network, MPAIL. does not require a policy update thereafter, since "policies" are in theory solved online. In practice a temperature decay can be helpful for preventing early collapse, espe- cially in the case of online model learning. We leave a theoretical justification for this choice for future work. An overview of the training procedure can be found in Figure A discussion of further meaningful implementation details, like spectral normalization, can be found in Section Though, these modifications are kept minimum towards our analysis of TMPPI in AlL.
4 EXPERIMENTAL RESULTS
Advancements in IRL and AIL continue to demonstrate improvements in sample efficiency. How- ever, there remains apparent gap with applications robot learning. In addition to fundamental investigations about the MPAIL algorithm itself, our experiments target evaluations critical towards real-world robustness and generalization.
Without a policy, vanilla MPPI possesses no "memory" about actions taken in previous episodes, save for those implied through the value (s). And as discussed in Section MPPI's approximate online optimization is more practical for robustness and generalization but potentially less accu- rate for producing policies. These novelties raise critical question about planning- -aased AIL; policies solved online through planning sufficient as adversarially generative policies? We find that MPAIL indeed trains an effective imitator, provoking our follow-up questions:
Q1 What is the advantage of deploying an AIL planner over an AlL policy?
Q2 How does MPAIL help enable real-world planning capabilities from observation?
Q3 How does MPAIL compare to existing AIL algorithms?
Hyperparameter settings are kept consistent across all experiments. Exact values and other imple- mentation details such as regularization and computation are reported and discussed in Section C
SIMULATED NAVIGATION TASK
For simulated evaluation, we design a navigation task with 10- DoF vehicle. Reward is proportional to the negative squared dis- tance to (10, 10). Initial poses are within m of (0, 0). The state 12-dimensions: position, orientation, linear velocity, and angu- lar velocity. Actions include target velocity and steering angle. MPAIL plans using approximate prior model, the Kinematic Bi- cycle Model (Han et al., 2024b). This approximate model not tuned the agent dynamies. For instance, slipping and suspension dynamics occur in simulation but are unmodeled.
Navigation
Pasitton (md
x X.Position (m)
Figure
Four Expert Trajectories in Navigation Task. Cars initialized around (0,
The expert for this environment obtained through running PPO on a known reward (see Section D.2 for details) (Schulman et al. 20176). At convergence, the optimal policy occasionally circles near the goal instead of remaining on it (see F). We choose to use the circling demonstrations as expert data, because is a more challenging behavior im- itate. This is corroborated by (Orsini et al. 2021), who stress that demonstrator suboptimality and multimodality is a critical component in algorithm evaluation towards practical AlL. from human
data.
This circling behavior acts as a critical "distractor mode". training, the policy may begin to only circle around the origin. If the AIL algorithm is not able to sufficiently explore, training collapses such that the policy continuously circles the origin, unable to return to the expert distribution which requires the circling behavior to occur around the goal. For instance, find that AIRL is unable to successfully learn both behaviors likely due to the instability introduced by its logit shift (Orsini et al. 2021).
5

---

Published as a conference paper at ICLR 2026
t -=0 0
t = 50
t = 100
GAIL (Policy)
Expert States
food
MPAIL (Planner)
Figure 4: Comparison of policy- and planning-based AlL in Out-of-Distribution (OOD) states. Agents trained on the navigation task (Section 4 are placed uniformly with random orienta- tion between 40 40 m box centered on (0, 0). The policy and planner are run for 100 timesteps in the environment. Data support of the expert exists mainly between (0, 0) and (10, 10 Quantita- tive evaluation this experiment can be found in Figure comparison which includes learned
dynamics model can be found in Figure 8
4.1
ROBUSTNESS THROUGH PLANNING
Q1
When deploying learning-basedd based methods to the real-world, re- liable performance in ouittf-dist (OOD) states are of critical importance, especially in imitation learning when ex- pert data can extremely sparse. We show that planning- based AIL (MPAIL) improves generalization capabilities when OOD. In this experiment, we use the simulated nav-
Reward Cle 100)
igation environment but expand the region of uniformly dis- tributed initial positions and orientations from a 1 1x1 I square large 40 m square around the origin (Figure 4.. The policy-basee based approach is represented by GAIL, as AIRL does meaningfully converge the navigation task (Section 4). Only four expert trajectories are used in training.
GAIL
NPAIL
-10"
000 Energy (1=0)
Figure 5: OOD Navigation Eval- uation. Agent initial poses vary from In-distribution (ID) to OOD relative to the expert data and are plotted with their final reward after 100 timesteps. Metric from (Liu et al. 2020) (see Section D.2)
We find that planning- based AIL generalizes to significantly more states than based AIL when outside the support of expert data. this experiment, the planner's horizon is a maximum of 3 meters. As result, the task horizon may be up to 15 times longer than the planning horizon. Evidently, planner could not navigate to the goal if the learned optimiza- tion landscape (induced by cost Gø and value V₂) did not also generalize to OOD states. These results suggest fundamental limitation of current AlL approaches and their single policy solution. The trained reward and value are inefficiently underutilized in policy-based AlL and not utilized at all on deployment. By contrast, MPAIL in-rrouooues the reward and value online to solve for new policies
Note that the state space 12-dimensional: expert data support extremely sparse in this environment.

---

Published as a conference paper at ICLR 2026
each moment in time. These results illustrate that generalization in AlL is substantially improved through reward deployment in addition to reward learning.
4.2 REAL-SIM-REAL NAVIGATION FROM SINGLE OBSERVATION Q2
Real-world evaluation of AIL is currently challenging. RL-like interaction efficiency renders training simulation more practi- cal than in the real-wor world (Tai et al. 20188) but demonstrations must realistically still be from the real-world. Nonetheless, it imperative to evaluate AlL methods on real-world suboptimal data and hardware since results tend to diverge significantly from ideal settings, synthetic data, and
1x Expert Demo
Sim Training
Deployment
simulation (Orsini et al. 2021 Tsurumine & Matsubara 2022).
Online Planning
Our hardware experiment evaluates GAIL, IRL-MPC, and MPAIL. through Real-to- Sim-to-Real: 1) a single partially observ- able (position and body-centric velocity) trajectory is collected from the real-world subject to sensor noise, 2) the method trained using interactions from simu- lation, finally 3) the method is deployed zero-shot the real-world for evalua- tion. This experiment uses small-scale RC car platform with NVIDIA Jetson Orin NX (Srinivasa et al., 2023). For IRL-MPC, the reward and value is trained through GAIL, which subsequently re- quires hand-tume hand-tuning for deployment on MPPI (Triest et al., 2023).
-2
Cost
Expert
IRL-MPC
Position
Figure Real-Sim-Real Experiment. Bottom Left (MPAIL). Real-time (20 Hz) parallel model rollouts and costing are visualized while the robot navigates through the turn. Current optimal plan for the next second in gold. Bottom Right. Trajectories performed MPAIL, GAIL, and IRL-MPC (see Table 2 for eval- uation).
remark that the direction of travel can- not be uniquely determined by single state s due to the partially observable
body-centric velocity. Only with the state-transition (sg, ) is it possible to deduce the direc- Section D.1 Partial observability and rate-transitions
tion of travel. This property is detailed play critical roles in the recovery of a cost function for this task, presenting necessary challenge towards practical AlL (Orsini et al. 2021 and scalable Leamiing-romm-Observatnn (LfO) (Torabi al. 2019b).
We find that MPAIL is able to qualitatively re- produce the expert trajectory with average (CTE (Rounsaville al.) of 0.17 while traveling aver- age of 0.3 m/s slower. In addition, Figure 6 illustrates a key advantage of planning-based AlL. By granting access to the agent's opti- mization landscape, MPAIL significantly im- proves the interpretability of agents trained through observations of ambiguous and com- plex human data when compared to black-box policies. Note the lower costing of on-track trajectories and final plan.
Reward r(s,s')
Policy Optimizer
Deployment
Relative Tross-Track-Bnee
GAIL.
log(!) ())
PPO (Offline)
Policy
AIRL IRL-MPC
logi
PPO (Oftine)
Policy
lug(D)
PPO (Ofline)
Planner
MPAIL
p-)) log(
log( 0
MPPI (Online)
Planner
Table 1: Summary of Baselines. The discrimina- tor is denoted D (s,,s)) Value estimation is performed via GAE-A (Schulman et al. 2018) for all methods.
GAIL does not reliably converge to the expert even in training. During deployment, GAIL's policy consistently veers off path or collapses into driving in a circle. Various starting configurations were attempted without success. While literature on the evaluation of All. methods in the real-world

---

Published as a conference paper at ICLR 2026
sparse, we find that AIL policies can be extremely poor performing in the real- real-world, as corroborated by (Sun et al. 2021). A more detailed discussion is provided in Section D.I
IRL-MPC acts as middle-ground between MPAIL and GAIL; the learned reward and value exactly the same as GAIL's and thus differs by the deployment of the reward through planning. IRL-MPC's improvements over GAIL provides evidence that: (i) model-based planning can grant robustness to a model-free reward and, (ii) despite GAIL's poor performance, the learned reward was still meaningfully discriminative and suggests failure of the policy arrive at solution under the reward. On the other hand, IRL-MPC diverges from MPAIL by mainly learned reward and value. As a result, we find that online policy optimization through TMPPI induces more competitive adversarial dynamic than offline policy optimization as in actor-critic RL. In this case, the end-to-end inclusion of the planner enables training the reward and value to completion.
Meanwhile, MPAIL's success and IRL-MPC's improvement over GAIL is attributed to model-basee planning capabilities. If the robot should find itself away from the expert distribution, the online planner enables the agent to sample back onto the demonstration. In this sense, an MPC-based (or any online- optimizing) agent brings control-thewe theorretcc disturbance rejec- tion online. Policies, the other hand, are far more suscep- tible to erratic hehavior in the real-world due to open-loop ac- tion prediction. This becomes especially important when the demonstration data is severely under-defined as in this partially observable setting, which results in ambiguous reward signal in most states in the environment; recall that low discrimina- tor confidence reflected by low magnitude logit fe (cost) predictions. Real experiment cost values are in the range of (-0.022, 000180) whereas cost values in benchmarking runs with synthetic demonstrations are in the range of (-3,3). On real data, the discriminator also required more frequent up- dates to provide more reliable signals (see Table 3..
CTE (m)
Average
Max
Mcan
Speed (m/s)
Expert
1.0
GAIL
1.29
0.56
0.37
IRL-MPC
1.28
0.37
0.30
MPAIL
0.76
0.17
0.70
Table 2: Evaluation of Real Ex- periment. Relative Cross-Track Error (CTE) and speed are com- puted over the best five laps. AIRL is excluded as it does not exhibit meaningful behavior even in simu- lation.
4.3 EFFICIENCY Q3
As mentioned at the beginning of Section 4 MPAIL. does not possess a persistent "pol- icy" as it employs a zeroth-order optimization with bootstrapped value estimates (i.e. infinite- horizon MPP1) to instead resolve these poli- cies online. While we have shown that this "deconstructed policy" significantly improves generalization and robustness, concerns regard- ing interaction efficiency may arise due to the
Nevigation
Cartoole
0.4
0.4
(Nomalza))
250
430
300
50
3000
250
200
250
Iterations
Earations
3.0
lack of gradient-based value optimization as in actor-critic policy optimization (Schulman let al.] (0177). Here, we perform additional benchmarking experiments evaluate these hypotheses.
MPHL MPti (0M)
ABL Feomn
0.4
D44.
foncer
We train GAIL, AIRL, and MPAIL on the navi- gation task and the cartpole task across varying quantities of expert demonstrations and random seeds as done in (Ho & Ermon, (016) Kostrikov (et al. 2018). While MPAIL uses an approx- imate prior model for the navigation task, we choose to learn a model during training of the cartpole task demonstrate the generality of MPC and support future work additional tasks. This is represented by the label, MPAIL (OM), indicating that there is a fully Online Model. Imple- mentation details of the leamed model can be found in Section D.3
Figure 7: Benchmarking Results. Top row rewards are computed across all demonstration quantities and seeds. Bottom row rewards are the average of the final 10 episodes computed across seeds. See Figure for de-aggregated plots.
8

---

Published as a conference paper at ICLR 2026
BC*
GAIL
MPAIL
MPAIL (OM)
100 with MPAII. (OM) and Behavior Cloning (BC) included. **BC requires access to expert actions and not an LfO bascline. MPAIL (OM) indicates that a dynamics model is learned online and used for planning in MPAIL. Without prior model, MPAIL (OM) is limited to the same amount of total information as GAIL; any improvement in OOD generalization over a policy network purely a result of learning deconstructed policy for online planning. Details of MPAIL (OM) in Section 4.3 and Section D.3
Figure 8: Dut-of-Distribution (OOD) performance evaluation (as in Figure
On the navigation task, MPAIL reaches optimality in less than half the number of interactions when compared to GAIL. We also observe MPAIL to train more stably than GAIL on this task. AIRL struggles learn from the multimodal data (see Section 4. In addition alleviating concerns regarding efficiency, these results support MPAIL's characterization as a model-based algorithm as it is more sample-efficient when provided an (approximate) prior model.
On the cartpole task, expert demonstration data results in cquivalently, AIL reward signal. It is likely that, due online dynamics model learning, MPAIL (OM) requires more exploratory interactions to combat large local minima induced and rein- forced by sparse discriminator reward, model bias, and task dynamics. Similar performance between MPAIL and AIRL may also suggest that GAIL benefits from its inherent reward bias on this task (Kostrikov et al. 2018). Importantly, MPAIL attains comparable performance while maintaining the benefits of model -based planning, such as interpretability, transferabilityy and robustness.
state visitation and,
ADDITIONAL. RESULTS
In the Appendix, additional experiments are performed to holistically evaluate MPAIL for the purposes of real-world deployment and robot learning. Wall clock time comparisons, architecture, hyperparameters, model ablations, proofs, de-aggregated benchmark results, and more experiment details and discussion can be found the appendix. We briefly highlight some key auxiliary results:
Wall Clock Time. Our timing evaluations reveal that "inference" and training times of MPAIL can be faster slower than GAIL. (PPO) depending mostly on the number of MPPI iterations per step and the planning horizon H. For H = 10, MPAIL is about 10% faster than GAlL. at 2 iterations. 5 iterations, the same horizon about 10% slower. More settings are evaluated and shown in Figure
MPAIL (OM) OOD Experiment. Does OOD robustness hold when the dynamics model is also learned? Figure 8 compares the final states of the OOD evaluation in Section 4.1 with MPAIL (OM) included. Note that MPAIL (OM) has access the same amount information as the policy-based method. Nevertheless, there is clear improvement in OOD performance by MPAII. (OM) when compared to policy networks. These results crucially suggest that generalizability advantages of the deconstructed policy extends to learned dynamics models in addition to reward and value.
Even in the case of Behavior Cloning (BC), which is directly supervised with access to expert actions, generalization of the policy network appears random and unpredictable. For policy networks (BC and GAIL), there numerous agents that initialized near the expert distribution (within red highlight) but remain there. These agents were often merely initialized facing away from the goal, demonstrating that policy networks tend to learn incredibly brittle representations. Most MPAIL.

---

Published as a conference paper at ICLR 2026
(OM) agents that are seen far away from the goal will continue slowly arrive at the goal, visually evident by their orientations directed towards the goal. Contrasting with MPAIL, MPAIL (OM) agents tend follow longer, less optimal paths when strongly OOD. This is likely result of the learned dynamics being OOD and disrupting planning.
5 CONCLUSION
This work adopts an imitation learning setting familiar to humans and animals in which: (1) ex- pert actions not known, (2) few demonstrations are observed, and (3) the agent infers intent and improves through interaction. While work in Inverse Reinforcement Learning (IRL) and Adversar- ial Imitation Learning (AIL) from Observation continue to advance these goals, their applications to real-world robots lag behind. To bridge this gap, we observe that connections to model-bases planning offers potential towards (a) improved efficiency and transfer, (b) safe and steerable de- sign, and (c) robustness through online optimization. We call this problem setting Planning-from Observation (PfO). We address PfO by introducing planning-base based AlL. as a unification of IRI. and MPC. Model Predictive Adversarial Imitation Learning (MPAIL) is then introduced as an imple- mentation of planning-based AIL, where a planner is continually improved through cost and value learning.
Towards robot learning applications, we conducted three elucidating evaluations. Section reveal that reward deployment- not only reward learning in current AlL-is critical towards generalizable imitation learning. Re-introducing the learned reward online alleviates the burden on policies generalize and requires the reward, intent, generalize instead. In Section 4.2 that reward deployment must be met with online optimization for real-world robustness. Especially in partially observable AIL, it may not be sufficient utilize a planner only during deployment, but the planner should also be included in the learning process to train the reward to completion. From single partially observable demonstration in real-world navigation, we find that MPAIL is the only successful imitator when compared to policy-based AIL and IRL-MPC. MPAIL also employs representations (e.g. model, reward, value) which grant direct access the agent's optimization landscape and thus decision- process-a powerful prerequisite towards safe and interpretable robot learning. Finally, in Section 4.3 interaction efficiency benchmarks favorable to MPAIL with an approximate prior model address concerns regarding MPPI's zeroth-order policy optimization and empirically supports MPAIL's 's characterization as a model- based algorithm.
MPAIL is derived from, and naturally admits, abstractions from Model Predictive Control, model- based RL, and imitation learning. Thus, its open-source implementation aims to reflect this and offers common ground for instantiating the many possible extensions to adjacent work through these connections: off-policy value estimation for improved sample efficiency or offline learn- ing (Kostrikov et al., 2018), policy-like proposal distributions and latent dynamies for scaling MPPI to higher dimensional spaces (Hansen et al. 2024: Zhu et al., (025), model-free and model-based reward blending for alleviating model bias (Bhardwaj al. 2021 diffusion-inspiree MPPI for improved online optimization (Xue et al./ (2024), and much more (Han et al., 2026).. We envision
that this work can provide theoretically and empirically justified foundation for future work the intersection of MPC, RL, and Imitation Learning.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
Included in the supplementary material is the code for MPAIL, developed with the intention to be released standalone package. Configuration files and hyperparameters for experiments almost entirely "flat" (viewable in the code) and are as presented in T Simulation environment built with Isaac Lab (Mittal et al. 2023) and one of its extensions, Wheeled Lab (Han et al.) 2025). Instructions for installation can be found in their respective GitHub repositories. Hardware deployment code is also included for the (open-source) platform, MuSHR (Srinivasa et al. 2023); an example script detailing how to save and load an MPAIL planner onto the hardware can be found in the supplementary as well. More details explaning souree code organization, as well as experiment videos, can be found in the README.md. We welcome efforts towards reproducibility and encourage practitioners to correspond regarding difficulties.
ACKNOWLEDGMENTS
TH supported by the NSF GRFP under Grant No. DGE 2140004.
REFERENCES
Nir Baram, Oron Anschel, and Shie Mannor. Model-based Adversarial Imitation Learning, Decem- ber 2016.
02179 arXiv:1612.0211 [stat].
Nir Baram, Oron Anschel, Itai Caspi, and Shie Mannor. End-to-End Differentiable Adversarial Imitation Learning. In Proceedings of the 34th International Conference Machine Learn- ing. pp. 390-399. PMLR, July 2017. URL https: atttps:/pproceedeee baram 7a. html
Matt Barnes, Matthew Abueg, Oliver F. Lange, Matt Deeds, Jason Trader, Denali Molitor, Markus Wulfmeier, and Shawn O' Banion. Massively Scalable Inverse Reinforcement Learning in Google Maps. October 2023. hnttps://openne
net. mett/forumiddz3330 GALM
Mohak Bhardwaj, Ankur Handa, Dieter Fox, and Byron Boots, Information Theoretic Model Pre- dictive Q-Learning.
Mohak Bhardwaj. Sanjiban Choudhury, and Byron Boots. Blending MPC & Value Function Ap- proximation for Efficient Reinforcement Learning, April 2021. URL http://arxiv.orr abs/2012. 05909 arXiv:2012.05909 [cs].
Eli Bronstein, Mark Palatucci, Dominik Notz, Brandyn White, Alex Kuefler, Yiren Lu, Supratik Paul, Payam Nikdel, Paul Mougin, Hongge Chen, Justin Fu, Austin Abrams, Punit Shah, Evan Racah, Benjamin Frenkel, Shimon Whiteson, and Dragomir Anguelov. Hicrarchical Model-Based Imitation Learning for Planning in Autonomous Driving, October 2022. URL URLpttt://arxi orq/abs/ 2210. 09539 arXiv:2210.09539 [cs].
Sanjiban Choudhury, Mohak Bhardwaj. Sankalp Arora, Ashish Kapoor, Gireeja Ranade, Sebastian Scherer, and Debadeepta Dey. Data-driven planning via imitation learning. The International Journal of Robotics Research, 37(13-14):1332-672.. December 2018. ISSN 0278-3649. doi: URL httpt:tttt.... http:tttt////i.ccoo wtt.:///.....1111100088086111111
0111777027836491888111111
Neha Das, Sarah Bechtle, Todor Davchev, Dinesh Jayaraman, Akshara Rai. and Franziska Meier. Model-Based Inverse Reinforcement Learning from Visual Demonstrations. Proceedings of the 2020 Conference on Robot Learning, pp. 1930-1942. PMLR, October 2021. URL https:
//proceedings.. www.ping.................eeeee.. wwt.....eee.l../aaaaaaa...
html
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization. In Proceedings of The 33rd International Conference Ma- June chine Learning, pp. 49-58. PMLR, June 2016. URL httphhppps//prroeeeinnie//ee rmrprerre/
v48/finn1 w.r.mn6.hhii 6.hml
Justin Fu, Katie Luo, and Sergey Levine. Reinforcement Learning, August 2018. URL http: fttt:://ariss wttg/ass/////// 11248 arXiv: 1224. 1248 [cs].
Learning Robust Rewards with Adversarial Inverse

---

Published as a conference paper at ICLR 2026
Jasper Geldenbott and Karen Leung. Legible and Proactive Robot Planning for Prosocial Human- Robot Interactions. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 13397-13403, May 2024. doi: 10.110900 1eeexp ore. feee. org/ http://ddmmmenn111122200
URL https://
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A Divergence Minimization Perspective Imitation Learning Methods. In Proceedings of the Conference on Robot Learning, pp. 1259-1277. PMLR, May 2020. URL https: http::://proceeeiing wttp://ddd...........ssssss///.e press/v10//
mlr www.prressseeee
ghasemipour20er .html]
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Ad- vances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL http::/:e//ppooeeeegsgsssssseee hash/ 113448006022377735557770222-23-2
ct cc/paper..eppeee.eeppee files/paper/201e/
Tyler Han, Alex Liu, Anqi Li, Alexander Spitzer, Guanya Shi, and Byron Boots. Model Pre- dictive Control for Aggressive Driving Over Uneven Terrain. In Robotics: Science and Sys- tems XX. Robotics: Science and Systems Foundation, July 2024a. ISBN 9798990284807. robot URL http:://www wtwp::oooooooooocceeeeeeeeeeeeee www.cccccoeennss.........
doi: (0.15607/RSS.2224.XX. rss20/p022. pdf
Tyler Han. Sidharth Talia, Rohan Panicker, Preet Shah, Neel Jawale. and Byron Boots. Dynamics Models in the Aggressive Off-Road Driving Regime, May 2024b. Utttatt:://aaat http:://aaxxv.rooo/
abs abs/2403 2405. 6487 arXiv:2405. 16487 [cs].
Tyler Han, Preet Shah, Sidharth Rajagopal, Yanda Bao, Sanghun Jung, Sidharth Talia, Gabriel Guo, Bryan Xu, Bhaumik Mehta, Emma Romig, Rosario Scalise, and Byron Boots. Wheeled Lab: Modern Sim2Real for Low-cost, Open-source Wheeled Robotics, August 2025. URL http:
f/arxiv.org/ abs//aa/////// www.///55000 220...252..00 250220732200 axXiv:2502.03880 [cs].
Tyler Han, Siyang Shen, Rohan Baijal, Harine Ravichandiran, Bat Nemekhbold, Kevin Huang, Sanghun Jung, and Byron Boots. Planning from Observation and Interaction, February 2026.
URL hhttp:///axii....t///
abs /2022 24121 arXiv:2602.24121 [cs].
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal Difference Learning for Model Predictive Control, July 2022. URL http:////aaa....rrraaa//////0/ 04955, arXiv:2203.0499 [cs].
Nicklas Hansen, Hao Su, and Xiaolong Wang. els for Continuous Control, March 2024. URL http:://axxy wttp://wwwwgg//b//33331..// 16828 arXiv:2310. 16828 [es].
TD-MPC2: Scalable, Robust World Mod-
Nathan Hatch and Byron Boots. The Value of Planning for Infinite-Horizon Model Predictive Con- trol. In 2021 IEEE International Conference Robotics and Automation pp. 7372-7378, May 2021. doi: 10. 1110///CRRAA00662200000011 brc/document//56111
URL
ieee.
Jonathan Ho and Stefano Ermon. Generative Adversarial Imitation Learning. In Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.,
2016. URL Inttps:/ nww..pss...e.ur.......m ec htapaasseecpppeer..eeee//eeeeee http://cc.ebbbtlloooceeeee222213 Abteaaaa..................aa...e files//aper/2016/
Abstract. html
Neel Jawale, Byron Boots, Balakumar Sundaralingam. and Mohak Bhardwaj. Dynamic Non-
Prehensile Object Transport via Model-Predictiv Reinforcement Learning, November 2024.
URL http:://t/ai....t/// org. abs/ 2412. 00086 arXiv:2412.00086 [cs].
Sanghun Jung, JoonHo Lee, Xiangyun Meng, Byron Boots, and Alexander Lambert. V-STRONG:
Visual Self-Supervised Traversability Learning for Off-road Navigation, March 2024. URL
http://arxizz wwweiiiorrg//// absg//ab///e 2312 16016 arXiv:23 12. 16016 [cs].
Christos Katrakazas, Mohammed Quddus, Wen- Hua Chen, and Lipika Deka. Real-time motion planning methods for autonomous on-road driving: State-of-thh-aar and future research directions. Transportation Research Part C: Emerging Technologies, 60:416-442. November 2015. ISSN
12

---

Published as a conference paper at ICLR 2026
0968-090X. doi: 10.1016/.1t:22015.99.011 URL wtt.tttss./ww...ciinndee........ occaaee//arrrcee///////0605663.
Iyy Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Disrrininator-Actor-CCiii Addressing Sample Inefficiency and Reward Bias in Adver- sarial Imitation Learning, Octoher 2018. URL http::/aa.i......// www./rgg//////////////.o//eeeeee 8098 02925
arXiv:1809.0222 [cs].
Henrik Kretzschmar, Markus Spies, Christoph Sprunk, and Wolfram Burgard. Socially com- pliant mobile robot navigation via inverse reinforcement learning. The International Journal of Robotics Research, 35(1 1): 128912291111 1307, September 2016. ISSN 0278-3649. doi: 10.1177/
0278364915619772. URL C https:://///.t
Markus Kuderer, Shilpa Gulati, and Wolfram Burgard. Learning driving styles for autonomous vehicles from demonstration. In 2015 IEEE International Conference on Robotics and Automa- tion (ICRA), pp. 2641-2644 May 2015. doi: 10.1009/CRA.221557195555. URL https:
//ieeexplore leee. org/ /tt///mttt/ttt55555/0
Keuntaek Lee, Bogdan Vlahov, Jason Gibson, James M. Rehg. and Evangelos A. Theodorou. Approximate Inverse Reinforcement Learning from Vision-based Imitation Learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 10793-10799, May 2021. doi: 10. (0.1100/44444450002010..0 252.666916.. URL httpss//eeeeeexloeeeeee iew.. org/ Document/99609100
Keuntaek Lee, David Isele, Evangelos Theodorou, and Sangjae Bae. Risk-sensitive MPCs with Deep Distributional Inverse RI. for Autonomous Driving. In 2022 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pp. 7635 October 2022a. doi: 1109/R00446112222..98122222 URL atttss://eeexp..e
ieee org/ document/
9981223/
Keuntack Lee, David Isele, Evangelos A. Theodorou, and Sangjae Bae. Spatiotempo- ral Costmap Inference for MPC Via Deep Inverse Reinforcement Learning.
IEEE
Robotics and Automation Letters, 7(2):319432211.
April 2022b. ISSN 2377-
3766. doi: 10.1009/1RA.22222114665
URL
ieee.
org /bgg/abstio http//aatrcct/cccceeeee//3333333 http:///00S.aaa.aooo..eeeeeaaaaa
V1JKcxuQmUk77q-11B2 20g.222000...a000001661111112a.. Satpll_voozzzzzi2 7xHassJJ
1 1zmB dn Conference Name: IEEE Robotics and Automation Letters.
Shangzhe Li, Zhiao Huang, and Hao Su. Reward-free World Models for Online Imitation Learning. June 2025. URL.
Yunzhu Li. Jiaming Song, and Stefano Ermon. InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations, November 2017. URL http://arxiv.oog//bbb///00 [cs].
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, July 2019, (RRt:ht::///aeet http://xxxi....aaaaaa////a/aaae/ www./bs/////oooooooo/tt.../a...a wwtp.os///ab....o.o////// C2971 Mxiv:155090097) [cs].
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based Out-of-distribution Detec- tion. In Advances in Neural Information Processing Systems, volume 33, pp. ran Associates, Inc., 2020. URL ttLL:ttttss:::roeeddee htt.://ddeeeoeeereeeieooo/oo///e
Cur- wttpp/ppp.ccpppppp///////// haash//t499333300333e33a.aa111.. 2tt.:///000CC0003333331129000000 Abteeoococcbttttt....o..... html
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control, January [cs].
2019. TIRLatta /ttl:://aaaxx /ttp///aavvvvvvv wttpaaiii....aaaaaaaaaaaaaaaaaaa wwtp://pp//aaaaaaaoooooooooaaaat 1tt.a/////1..111111111.1.1111111
arXiv:1811.00
Tianjiao Luo, Tim Pearce, Huayu Chen, Jianfei Chen, and Jun Zhu. C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory, October 2024. URL /ttt::://xxv...
org/abs//2402. 6349 arXiv:2402.1633 16349 [cs].
13

---

Published as a conference paper at ICLR 2026
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yumrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments. IEEE Robotics and Automation Letters, 866):3740-3747, June 2023. ISSN 2377-3766, 2377-3774. doi: 10.1
utt.http://arxiv..
abs/23011 04195
arXiv:2301.0411 [cs].
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Normalitioon for Generative Adversarial Networks, February 2018. URL http:///aaxxxvvoooaaa////ssa////
05957
1rXiv:1802.05957 [cs].
Andrew S. Morgan, Daljeet Nandha, Georgia Chalvatzaki, Carlo D D'Eramo, Aaron M. Dollar, and Jan Peters. Model Predictive Setor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning. In 2021 IEEE International Conference on Robotics and Au- tomation (ICRA), pp. 6672-6678, May 2021. doi: 10.110000CRA44550222000000000 ieee. org/ htttps///ieeee: orgpa//ee......tooooooooooo//o/a
URL
Manu Orsini, Anton Raichuk, Leonard Hussenot, Damien Vincent, Robert Dadashi, Ser- tan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychow- icz. What Matters for Adversarial Imitation Learning? In Advances in Neural In- formation Processing Systems, volume 34, pp. 14656-14668. Curran Associates, Inc., 2021. URL /tttps: he5.:/66c11448554669999666666660 www.seeee-bbstttcee Abstract. html
nwwrriurrs.... cc /tspaarrpppppeppppesreieeee//e/e wwwess/pppppp/22111111
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library, December 2019. URL http: http://arxvv...o org/ abs/ 1912. 01703. arXiv:1912.01700 [es].
Sascha Rosbach, Vinit James, Simon Großjohann, Silviu Homoceanu, and Stefan Roth. Driving with Style: Inverse Reinforcement Learning in Gceneral-Purpot Planning for Automated Driv- ing. In 2019 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 2658-2665, November 2019. doi: 10. 1011009/RG448887222118888333 8962000
URL hetps
/ //ieeexplore leee. org/
wttp:t//////2220088822 2tt.:// 8968205
Joseph D Rounsaville, Joseph Dvorak, and Timothy Stombaugh. Methods for Calculating Relative Cross-Track Error for ASABE/ISO Standard 12188-2 from Discrete Measurements.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. In Proceedings of the 5th Conference Robot Learning, pp. 91-100. PMLR, January 2022. URL v164 #164/rudin22em html
mlr, press/
Jacob Sacks and Byron Boots. Learning Sampling Distributions for Model Predictive Control. In Proceedings of The 6th Conference Robot Learning, pp. 1733-1742, PMLR, March 2023.
URL htt.httts://ppocceeiiii
mlr hta..rppeees//20/sacc222222
html
Jacob Sacks, Rwik Rana, Kevin Huang, Alex Spitzer, Guanya Shi, and Byron Boots. Deep Model Predictive Optimization. In 2024 JEEE International Conference on Robotics and Automation (ICRA), pp. 16945-16953, May 2024. doi: 10.11099/C4557177.0000 24..2224..211111149 1061149.. URL https:
//ieeexplore i ieee wttp:o/////ocoooooooooeeee
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust Re- gion Policy Optimization, April 2017a. URL arXiv:1502.05477 [cs].
httpg/abs115022 05477
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017b. URL attt:://arxt wttp://////0..00e6/// arXiv: [es].
14

---

Published as a conference paper at ICLR 2026
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation, October 2018. URL
http://arx:: org/ hwrg/abs//e abs/1505. 02438 arXiv:1506.0245. [es].
Siddhartha S. Srinivasa, Patrick Lancaster, Johan Michalove, Matt Schmittle, Colin Summers, Matthew Rockett, Rosario Scalise, Joshua R. Smith, Sanjiban Choudhury, Christoforos Mavro- giannis, and Fereshteh Sadeghi. MuSHR: A Low-Cost, Open-Source Robotic Racecar for Ed- ucation and Research, December 2023. URL http://arxiv.ozg/
08031
arXiv:1908.0880 [cs].
Jiankai Sun, Lantao Yu, Pingian Dong, Bo Lu, and Bolei Zhou. Adversarial Inverse Reinforcement Learning With Self-Attention Dynamics Model. IEEE Robotics and Automation Letters, 6(2):
1880-1886, April 2021. ISSN 2377-3766. doi: 21.09//....22..366111... URL https:
/ //ieeexplore. i wttp////emee/////00600
Lei Tai, Jingwei Zhang, Ming Liu, and Wolfram Burgard. Socially Compliant Navigation Through Raw Depth Inputs with Generative Adversarial Imitation Learning- In 2018 IEEE International
Conference on Robotics and Automation (ICRA), pp. 1111-1 1il--117, May 2018 8. doi: 10. 10.110.. 8016.846 [RLLttts:///eeeepppple
document htw.eeummnn/446995500 8460988
Faraz Torabi, Garrett Wamell, and Peter Stone. Generative Adversarial Imitation from Observation, June 2019a. htthttp::// http://aawwvv..../aba//a//
06158 1xXiv1887.00615. [cs].
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent Advances Imitation Learning from Ob- servation, June 2019b. URL. http:://aaxx...rrr/bb////// [cs].
13566
arXiv: 1955.13566
Samuel Triest, Mateo Guaman Castro, Parv Maheshwari, Matthew Sivaprakasam, Wenshan Wang, and Sebastian Scherer. Learning Risk-Aware Costmaps Inverse Reinforcement Learning for Off-Road Navigation. In 2023 IEEE International Conference on Robotics and Automa- tion (ICRA), pp. 924 930, May 2023. doi: 10.1109/ICRA4889 000.889..222.1000 10161268. //leeexplore i eee. org /ttd//dccmtttt111eeee.111
URL https:
Yoshihisa Tsurumine and Takamitsu Matsubara. Goal-aware generative adversarial imitation learning from imperfect demonstration for robotic cloth manipulation. Robotics and Au- tonomous Systems, 104:64,,,,, December 2022. ISSN 0921-8890. doi: r0.b016/j.
104264. / 50921889022001544
Bogdan Vlahov, Jason Gibson, David D. Fan, Patrick Spieler, Ali-akbar Agha-mohammadi, and Evangelos A. Theodorou. Low Frequency Sampling in Model Predictive Path Integral Control. IEEE Robotics and Automation Letters, (0))4533-4550 May 2024. ISSN 2377-3766, 2377- URL
3774. doi: 10.1 R..1009/ arXiv:2404.03094 [cs].
wwwp:/o/gg/b/////// 03094
Bogdan Vlahov, Jason Gibson, Manan Gandhi, and Evangelos A. Theodorou. MPPI-Generic: A CUDA Library for Stochastic Trajectory Optimization, March 2025. URL /tttp://axxiv.
org/abs 2409. 07563 arXiv:2409.07563 [cs].
Nolan Wagener, Ching-An Cheng, Jacob Sacks, and Byron Boots. An Online Leamingg Approach to Model Predictive Control, October 2019, htthttp::/ /tttpp//arxiv wttpp/.oor//ab//1902.000 arXiv:1902.08999 [es].
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M. Rehg, Byron Boots, and Evangelos A. Theodorou. Information theoretic MPC for model-based reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1714-1721, May 2017. doi: 10.1109/CCA2201..7999200 URL htttttss/ieexxplorreeeess document/ 7989202
Haoru Xue, Chaoyi Pan, Zeji Yi, Guannan Qu, and Guanya Shi. Full-Order Sampling- -ased MPC for Leveue-Level Locomotion Control via Diffusion- -tffusion-Style Annealing, September 2024. URL
http://arse wwtp::ww.wwrgggoooooooo 15610 a arXiv:2409.15 15610 [es].
15

---

Published as a conference paper at ICLR 2026
Zhao-Heng Hengg Yin, Weirui Ye, Qifeng Chen, and Yang Gao. Planning for Sample Efficient Imitation Learning. 2022.
Chuning Zhu, Xinqi Wang, Tyler Han, Simon S. Du, and Abhishek Gupta. Distributional Successor Features Enable Zero-Shot Policy Optimization, January 2025. URL http://axxiv.org/ abs/ / 2403. 06328 arXiv:2403.03323 [cs].
Brian D Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum Entropy Inverse Reinforcement Learning.
16

---

Weight using exponential negative cost Published as a conference paper at ICLR 2026
APPENDIX
A INFINITE HORIZON MODEL PREDICTIVE PATH INTEGRAL
this section we present the full algorithm in detail, including MPPI as described in (Williams) al. 2017). Modifications "conventional" MPPI for MPAIL are highlighted in blue. Where applicable, (x); indicates the ith entry of x (in its first dimension, if is a tensor).
Algorithm 2 MPPI
Require:
Number of trajectories to sample N; Planning horizon Hi;
Number of optimization iterations
Fixed action sampling variance Σ;
Previous optimal plan a = {(a--1)e 1=
Current state 8t;
Dynamics model (s,
Costs (s s')
Value (s)
1: Procedure MP 1)-1)
2: (at) + (a,-1)i+1
Roll previous plan one timestep forward Set sampling mean to for last timestep
3: (ar) +0
4: for jo -
J do
5:
for 4- 0 to N do
Model rollouts and costing (parallelized)
6:
s/ f- 8; for + to H - do
7:
8:
ar (N((a)e) E) 8 (
b Sample action predicted state Predict next state Compute state-transition costs
9:
c co(s/, s1+1)
11:
end for
C(T.) 1 2 (x( H-1 & end for
12:
Total trajectory cost
13:
14:
3 - [C(n)(((TT))
15:
Z k= exp C(Tr)
16:
for +
to do
17:
w(TT) 1/2 exp ((m)
18:
end for
for
to H 1 do Optimal plan from reighted-average actions (at)e Σ w(T))a
21: end for
end for
23: for it 0 to
|A| do & Compute optimized standard deviations for policy Σ N-1 w(T))((a)))) (4
24: (a.);
end for
26: return a/" T
27: End Procedure
B PROOFS
In Section 3 introduce the replacement of the entropy loss in Equation 2 with KL divergence loss. This replacement allows the MPPI planner, in place of a policy, to solve the required forward RL problem. Integrated with the AIL objective in Equation (2), we further show that this allows MPAIL to correctly recover the expert state occupancy distribution ( s'). In this section, we prove both of these claims.
17

---

Published as a conference paper at ICLR 2026
Algorithm 3
Require:
Reward ri == CBi
Value Va:
(PPI( a) MPPI( N, H,,, Ju co. Vo) (Algorithm 2
T length of episode
1: a 4-0
Initialize optimal plan
2: B ()
3: for
to do
4:
St ~ [-x_(())-1,
Step and perceive environment
5:
aj. 0r - - a)
6:
Train then
7:
O₁ (N((aa) 1o,)
8:
else if Deploy then
9:
Og <- )o
end
11: Ti 4- (t(51,8t11)
Reward from discriminator
12:
BU Oe: Tt, 8t41)
13:
end for
14:
return
B.I MPPI AS A POLICY
In this section we justify claims regarding MPPI in the forward RL problem. For completeness, we also verify that known results remain consistent with our state-only restriction.
Proposition B.1.1. The closed form solution of Equation
min E (c(s + B KIL [),
(3)
is
T" (als) x [(aas)e ((,0) where a(s,a) = Σ
T(S++1 = s'SS, s)c(s, =))ss=))((,,s))
#ES
Proof.
We begin by noting that
Ex [= =] s] Σ m(als){(s,a)
where the weighted cost a(5, a) is defined
7(8,a) = Σ T(S+11 = = s)c(8,8)
(9)
e'ES
Then for fixed state E S, noting that the policy is normalized over actions DaEAA((R3) 1, we may form the Lagrangian with respect the objective in Equation (3) as:
(a)s) 8)
Σ m(uls)c(s,)) + B n (als) log a
Σ m(als) -1
(10)
T(als)
aEA
aEA
Taking the partial derivative with respect to π (als) and setting to 0 we have
ac (x(s))
(a)s)
= 7(8, a) + 3 log
(a)s) + 1
(11)
Finally,
9 log [alss)
= [(s, - 11 A ((ass)
(12)
T(als) x [(a|s)e -
(13)
m(lls) x <(als)e #((a,))
n
18

---

Published as a conference paper at ICLR 2026
Remark B.1.2. Given uniform policy prior, the KL Objective in Equation
min Re [c(8, 8 + 3 [T [7)
(3)
xell
is equivalent to the Entropy Objective,
(c(, - (T)) TEn
min
(14)
In order to prove this, it suffices to note that minimizing KL is equivalent to maximizing entropy:
Proof.
(als) log [(ass) [[ass)
KL [T|(T]]
Σ
d
(8)
Σ x
(15)
8E8
DEA
Σ d= (8) -HI(T(--()) Σ T(als) log
(16)
Σ d" (s)
([gls))og
(17)
aEA
--H(T)-
log k.
(18)
where k, rals)) for any a E A. Note that the sum on the left collapses by definition and the inner sum on the right collapses since the probability of taking an action in any given state is 1. Finally, since all the k, are constant, the second term on the right hand side is constant. Since both objectives differ by constant, minimizing the KL is equivalent to maximizing the Entropy given uniform policy prior.
Proposition B.1.3. Provided the MDP is uniformly ergodic, the MPPI objective in Equation
min Erar ( (KL
nell
equivalent to the RL. objective in Equation
min M.nEELehh (e,,, + B KIL T)) xell
Proof. Before continuing, verify that infinite horizon MPPI indeed predicts an infinite horizon estimate of the return. For simplicity, we momentarily revert to a reward only formulation, replacing the cost cs s') with the reward s') and the control discount 7) with 7. We proceed by expanding the return,
H-1
Eyez [R(T)] = Eynnl in H
Vo(sH)
+
7' R(si, St41)l
(19)
H
E
R(ss, 51 =sl +
Σ
((11))
(20)
1=#
1=1
H-1
E
* R 81-1)
Σ
r R(ss, 844))
(21)
Eron
y R(st,
(22)
where we have made use of the definition of a value function V,, (Equations 18 to 19) and the tower property of expectation (Equations 19 to 20).
19

---

Published as a conference paper at ICLR 2026
Let s') d's.s') + (KLL(T((15) 8) (+-1s)). For either objective be valid the cost and KL Divergence would have to be bounded. Thus, we may safely assume that f is uniformly bounded ll I K. Let 6, Bs, (f St41)] sa s')] be the enror between estimates of the objective.
Since the MDP is uniformly ergodic, may bound the rate of convergence of the state distribution at a time i, d' to the stationary distribution d'
YE E (0, 1), M E Nss -d"
d'v S MX
(23)
where || 1/v is the total variation metric.
Continuing by bounding error ór,
ldel S 11/1bb/ldd d=I S KMX
(24)
We then have that E n/ KM E.o = KM
We may now begin working with the MPPI Objective in Equation
E E (LTT))
(25)
n Ee [f(,,
(26)
Σ n/ ( [f s')] 5r]
(27)
B [[ 8')] + Σ n
(28)
1=
Note that the MPPI objective and Entropy Regularized RL objective differ by scaling and bounded additive constant, independent of æ. Thus, minimizing both objectives are equivalent.
B.2 MPAIL AS ADVERSARIAL IMITATION LEARNING ALGORITHM
In this section, integrate findings from Section B.J with the AlL objective to theoretically validate MPAIL as an AIL algorithm. Specifically, we observe that at optimality, we recover the log expert- policy transition density ratio, which in turns yields maximum entropy policy state-transitions. We then discuss the identifiability limits imposed by observing only (s, rather than (s, a, 8'). Throughout this section make use of the tratsstranition occupancy measure, defined as Px
x S + IR where Px (s, E 7 "T((++1 = s', St = s TT) as in (orabi et al. 2019a).
Proposition B.2.1. The optimal reward is
PE(S, Px (8,
18 8') = log
(29)
Proof. Note that the optimal discriminator is achieved when (s,s'')
(etss,s'++00e as used
pgls.s")
in (Ghasemipour et al. 2020) and shown in (Goodfellow et al. 2014) Section 4 Proposition 1.
fá (s) = (s, 8')) logllll111111 DD
(30)
Pa (s,
= log
log 8') + P P=(88 8²)
(31)
(e) + (x,
PE(s,s')
= log
Px (s,
20

---

Published as a conference paper at ICLR 2026
This shows that, by setting r(s, s')s,,)), the recovered reward function is the log-ratio of state-trand traessrranition occupancy measure from the expert to the policy.
Lemma B.2.2. MPAIL minimizes regularized KL divergence between the policy' 's transitoannition occupancy measure and the expert's.
Proof. Recall from Proposition B.1.1 that while solving for the RL objective, MPPI finds policy of the form
<als)) x (T) s)e )
(32)
Applying Proposition B.2.1, we plug c(s, = fō (s, (s, into Equation to obtain
( E T(s'|s)[logp Px log pE)]
T" ax( exp
(33)
S'ES
Note that when the policy distribution matches the expert distribution pe the exponential term collapses. Thus, when the occupancy measures match, the policy updates cease to have effect and the optimization attains fixed point.
In fact, we may note that for any fixed state S, the cost accumulated by the policy is
Pr (x((s,))) = (L( )llEE(s,)) (
[(((ss)) = p'(,,,'))
(34)
E'ES
d'ES
Finally, the MPPI objective can be written as
min KL(p=lIpE) (L(# [|)
(35)
Hell
showing that the MPAIL procedure minimizes the entropy regularized KL Divergence between state- transition occupancy measures. In this sense, we have shown that MPAIL can be indeed classified as an AlL algorithm which seeks match the expert's ''s occupancy measure through an MPPI Policy.
Remark B.2.3. On Identifiability. question naturally arises about the limitations that being state- only imposes. If state transitions are deterministic and invertible, observing (s, s') is the same as observing the unique action a that caused it. Then r(8, r(s, a) and by the 1-1 correspon- dence of policies with state-action occupancy measures (Ho & Ermon 2016), the recovered policy becomes unique.
In general, this assumption has varying degrees of accuracy. When transitions many to one or stochastic, multiple actions can produce the same transition (s, Then Ps (s, = (s()) EBEA (aass)T( a) becomes mixture tive policies. For instance, if T(s'ls, a1) T(s'ls, a2) for actions d1, d2, the expert could perform either action and state-transition based reward would not distinguish between them. Nonetheless, IRL is already an ill-posed problem due to the many one relationships between policies, rewards, and demonstrations. Though the ambiguity is exacerbated by lack of demonstrated actions, it still inherent to the problem.
actions which induces range of respec-
C IMPLEMENTATION
In this section provide further details about the algorithm implementation. Some features in- corporated here are deemed well-known (i.e. spectral normalization) or not rigorously studied for statistical significance but included for completeness and transparency.
21

---

Published as a conference paper at ICLR 2026
C.I REGULARIZATION
Spectral Normalization. As often found in GAN and AIL surveys (Orsini et al. 2021: Miyato et al. 2018), we corroborate that applying spectral normalization to the discriminator architecture appeared to have improved MPAIL training stability and performance. Application of spectral nor- malization to the value network did not appear make noticeable difference.
L2 Weight Regularization. Some experimentation was done with L2 weight regularization, but it was ultimately not used for any simulation results. Instead, usage of the weight regularization for the real experiment (Section 4.2 appeared to help stabilize training and allow for more reliable model selection and deployment.
C.2 HYPERPARAMETERS
Fundamentally derived from AIL and MPPI, can etymologically partition hyperparame- ters into those induced by AIL (orange) and by MPPI (blue). Remaining non-highlighted pa- rameters for this work are introduced and dis- cussed below.
Hyperparameter
Value
Disc. optimizer (8)
Adam (9) = 0.5, Bg 0.999)
Disc. learning rate
le-4
Disc. hidden width
32
Disc. hidden layers
2
Disc. L2 coefficient
(sim), 0.001 (real)
Value optimizer (o)
Adum (9. 0.9. 8g 0.909)
Temperature Decay. As noted in Section we found that an initial temperature with gradual decay (down to minimum) was help- ful preventing early and unrecoverable col- lapse. The intuition for this decision is simi- lar to that of decaying policy noise injection in many popular RL frameworks (Lillicrap et al. 2019 Hansen et al. 2024), since the tempera- ture is directly related the variance of the op- timized gaussian distribution. This component remains under investigation as its usage is not always necessary for meaningful convergence, but perhaps practically useful as it allevi- ates temperature tuning labor.
B
Value learning rate Value hidden width
le-3
32
Value hidden layers
2
Value loss clip
0.2
Discount (?)
0.99
Generalized return ())
0,95
Value max grad norm
1.0
Mini batches
3
Epochs
3
Trajectories (N)
512
Planning horizon (H)
10
Iterations (J)
5
Sampling variance (5)
diag( (0.3 ....I)
Initial lemperature (Ag)
1.0
Markup/Discount (1)
1.01
Temp. decay rate
0.01
Minimum temp.
le-5
Value:Disc update ratio
3:1 (sim), 1:1 (real)
Diieritt-Disr Update Ratio. Com- mon to existing AIL (and GAN) implementa- tions, MPAIL benefits from balancing gen- erator and discriminator updates. Note that, like GANs, AIL tends to oscillate aggressively throughout training (Luo et al. 2024). As MPAIL does not enforce constrained policy update each epoch (as TRPO does (Schulman al. 2017ap), the policy is exposed more directly to the discriminator's 's oscillations which can further hinder on on-policy value estimation. A further converged value function is also theoretically more stationary from the perspective of mPppi as infinite-horizon MPPI.
Table 3: MPAIL Hyperparameters Used across
all experiments unless specified otherwise.
Markup. A notable quirk discovered during implementation is the relationship between costs and rewards. While the two concepts are generally regarded as dual (with negation), it is worthwhile noting that discounting is not closed under negation. Meaning, it is not correct apply the same discount factor to the costs as they are done to the summation of rewards in the return G Consider the reward with a discount applied r = 7 r(s,s') and the reward of cost with discount applied T2 = -y c(s, s¹). Observe that for < 1, decreases while increases. Thus, when using costs, the H-step factor in the MPPI horizon, 7), should not decrease over t'. In fact, when applying v) < 1, we found that MPAIL does not meaningfully converge ever on the navigation task. (Geldenbott & Leung 2024) names the usage of > as a markup. In our case, we apply a similar empirical factor such that == 1/7 > 1. While we suspect more rigorous relationship between 7) and y we leave its derivation for future work. However, we remark that a reward-only variant MPPI which precludes these relationships is equally possible as done in (Hansen et al. (022). Costs are maintained in this work due to wider familiarity in practice (Han et al.) 2024a] Williams et al. 2017
Morgan al. 2021 Finn et al. 2016).
22

---

Published as a conference paper at ICLR 2026
KEAIIPulioy inilialised. Total outber pasamas 3779
Dynanies:
Sampling:
Cosl: 3578
Temperaturo:
HPP: Ha20
HPP: h=10
APATiPallicy
(dynanics): #LommaiidBicyceeeooo
MIT,, m5-5
(cost))
4.4
ARD
[es_oust:: GATFOCoNE
(ceward) : Sequential:
4.2
(31 Minner (in. features=25, featurese32,
4.0
(1) LeakyRal0 (wg.a/ooooooiooooooooooooo.
(21 Linear Features=32, out fettureet,,,
3.8
biasePrue)
LeakyRalD wwwgstiin_soooee..o.ooo
38
(4) Mineer (in, featureee32, out featurree,,
blaseTcue)
1.4
[LaooutL: CoatToGol
(valuel: Sequential
(3) Ginear (in, featurne-12, gct 3eatureee3,,
Figure 9: Comparing "Inference" Times for Navigation Task. Time taken to com- plete one episode of 100 timesteps with 64 parallel environments across varying hori- zon lengths and MPPI optimization itera- tions. PPO (in policy-based AIL) is used implemented in the RSL library (Rudin et al. 2022]. All training runs in this work are performed on an NVIDIA RTX 4090 GPU. Isaac Lab chosen as our benchmarking and simulation environment due to its paral- lelization and robot learning extensions (Mit-
bias=True]
(11 Reti(1)
(2): Linear Featurree,, out featuuree,,
bias-True)
(4) Lineer (in, featureee3,, feattureer,,, bias-True;
ing) BellaSenp' ing
2019) Model Architecture from Train Log. MPAIL read- ily admits other well-studied components of the model-based planning framework (e.g. sampling, dynamics) (Vlahov et al., 2025 2024). This work focuses on costing from demonstration.
Figure 10: PyTorrh (Paszke et al.
tal et al. 2023
Han et al. 2025).
C.3 COMPUTATION
MPAIL is crucially implemented be parallelized across environments in addition to trajectory op- timization. In other words, in a single environment step, each parallel enviromment independently performs parallelized sampling, rollouts, and costing entirely on GPU without CPU multithread- ing. MPPI also allows for customize-able computational budget, similar to (Hansen et al.) 2024) (see Figure D). For the navigation and cartpole tasks, we find that online trajectory optimization implemented this way induces little impact training times. In exchange, MPPI can be more space intensive due to model rollouts having space complexity of O(H N S|) per agent. On the navigation task benchmark settings, this is additional 245 kB per agent 15.7 MB in total for 64 environ- ments. Figure / shows benchmarks on training times that demonstrate comparable times to PPO's policy inference. Overall, training runs for Section 4.3 between MPAIL and GAIL on the navigation task are comparable at about 45 minutes each for 500 iterations.
D EXPERIMENTAL DETAILS
D.1 REAL-SIM- SAAL-SIM-EEA REM-REAL NAVIGATION
Setup Details. Before continuing with the discussion of our results, we provide further details about the setup of the experiment. The platform itself is an open-source MuSHR platform detailed in (Srinivasa et al. 2023). Notably, the compute has been replaced with an NVIDIA Jetson Orin NX as mentioned in Section 4.2 Poses (position, orientation; [x: motion-captur system at a rate of 20 Hz. Velocities are body-centric as estimated by onboard wheel encoders (v by Uy U. by, such that b points forward, by points left, and by completes
r p v)) are provided by
the right-hand frame: basis vectors are rigidly attached to the vehicle (Han et al. (022a)).. Note that
the vehicle is operated without slipping nor reversing such that By V2 0 and U. 0 (Han
23

---

Published as a conference paper at ICLR 2026
et al. 2024b). The recorded states used for the expert demonstration data is 240 timesteps long. Altogether, the data can be written as sg E {( Yos 2,, U2,t: Vy, U2,
A remark: GAIL for this task is necessarily implemented with "asymmetry" between actor and reward. Since, the discriminator must receive as input expert observations sg while the agent is provided (r, P. y) in addition to observations in sg- In theory, there should be conflict with the IRLfO (Equation (T) formulation as this remains a valid reward but on a subset of the state.
Additional Discussion of Results.
To understand the role of partial observability in the experiment design, consider a simplified hand- designed cost using the partially observable expert data c(s, s' (8 8g)) A reference vector can be computed through the difference of positions between velocities: c(s, s' [ss,, s'g) == (s, s')- U2 e l(<'g IE) ei + (y's - y yg) )22]ll2 where e, are global basis vectors for global frame I and (v s') is the robot velocity in I. Of course, this example assumes the ability to correctly choose the corresponding (#F, (/.))))))) pair for input (s, 8') out of the entirety of the expert dataset d It should be clear that partial observability and tate-transitions play critical roles in the recovery of this non-trivial cost function. This experiment presents necessary challenge towards practical AIL (Orsini et al. 2021 and scalable Learning-from cooo--bbrrvvtion (LfO) TToabi et al. 2019b).
and then scaled by the demonstrated
IRL-MPC was evaluated across three ablations: (a) reward-only, (b) value-only, and (c) reward-and- value. The results in were distinctly worse than (a) and frequently devolved into turning in circles much like GAIL.
the performance of (a) reward-only. The other implementations
In both cases of GAIL and MPAIL, we find that the agents occasionally travel counter-clockwise (where the expert travels clockwise) during training, suggesting that (sg. s'yz) appears close to (s'p. through the discriminator. As the data is collected through real hardware, it is suspected that state estimation noise introduces blurring between states that are separated by only 50 ms. GAIL otherwise known to perform poorly the existence multi-modal data (Li et al., 2017). This is further corroborated by its unstable performance on the navigation benchmark. And, to the best of our knowledge, similar works which use real demonstration data but train in real include (Tsurumine & Matsubaral 2022 Sun et al., 2021). Even while training in real, GAIL's performance drops signficantly (90% - 20% when presented with imperfect demonstrations for even straightforward tasks like reaching (Tsu- rumine & Matsubara| [022]] Sunn et al.) 2021). These observations might suggest why the GAIL discriminator unable to learn meaningfully in simulation and produces poor policy.
-ia--iim-eeal applications of AIL appear sparse if existent at all. Adjacentt
D.2 SIMULATED NAVIGATION TASK DETAILS
Reward and Data. The exact form of the reward used for training PPO and for metrics is given by
r(s) == / 10a + 10a v(x 10)a + (y 10)2.
Figure visualizes visualizss four demonstrations from the converged PPO "expert" policy. Additional demon- strations are generated by playing more environments from this policy for one episode such that each demonstration is distinct. Each episode is 100 timesteps long, where each timestep is 0. seconds.
OOD Experiment. OOD Energy Figure computed as described by Liu al. 2020). Namely, with respect to the expert data d= we fit reference distribution using PE N (Næ, Σg). Then, the OOD energy is given by E(s; = (o)) Some limitations of this procedure can be ob- served given that ID points for GAIL do not receive as much reward as one might expect. However, this remains reasonable considering that the GAIL policy may forget ID behavior, which can be seen in Figure 4 by agents clearly ID remaining static throughout the episode. Future work might better explore quantifying OOD towards measuring AIL generalization through direct usage of the discriminator.
D.3 PREDICTIVE MODEL LEARNING TOWARDS GENERALIZABLE MPAIL
For tasks beyond navigation (see also Section D.5 for the Ant environment), planning rollouts were generated from deterministic dynamics model (s a) learned entirely online. The dynamics
24

---

Published as a conference paper at ICLR 2026
model was trained to minimize the mean squared error between the predicted and observed 81+1, given 8t and ag. The loss being optimized can be written as:
844 a.), L
(5411 8(+1) -
8(41)
(36)
HA 8
with model parameters vh transition buffer B, and a mini-batch of size H sampled from B. used, the update for the model occurs after line in Algorithm
Training augmentations. Several train- ing augmentations were made to improve model aceuracy and stability. A transi- tion replay buffer, which stored transitions from multiple episodes, was used to train the dynamics model for multiple epochs during each MPAIL training iteration. Af- ter each episode, the buffer was updated by randomly replacing old transitions with those from the latest episode. This off- policy buffer helped stabilize training and prevent overfitting when training using multiple epochs. Furthermore, applying a step-based learn- ing rate decay improved convergence speed. Dynamics model speccffc hyperparameters are listed in Table 4
Dynamies Model Hyperparameter
Value
Optimizer
Adam(
0.9, 82 0.999)
Learning rate
1e-3
LR decay
1.R decay frequency (ep.)
25 (Ant), 15 (Cartpole) 1e-6
Min.
Hidden width
256 (Ant), 64 (Curipole)
Hidden layers
Table 4: Dynamics Learning Hyperparameters.
D.4 ABLATIONS
Figure 12 shows the results of ablation study, investigating the effect of including costs or values in the MPAIL. formulation. We find that including both is necessary for reasonable behavior across varying horizon lengths. We observe that value-only planning can quickly improve but is highly unstable and is unreliable as a generator. As expected, cost-only planning performs progressively better at longer planning horizons. However, should the agent find itself off-distribution, it is not able to return to the distribution until it randomly samples back in, which may potentially never occur. For instance, many agents which are initialized facing the opposite direction drive randomly without ever returning to the distribution. Without a value function guiding the agent, the discriminator (i.e. cost) does not provide a significant reward signal for returning to the distribution. This can be observed in the H 10 plot where the performance of cost-only planning quickly drops as the discriminator is further refined on the expert data, decreasing the likelihood of randomly sampling into distribution. In this sense, the combination of cost and value operates as intended: costing is necessary for defining and staying inside the expert distribution, while value necessary for generalizing the reward beyond the support of the expert elsewhere in the environment.
D.5 TOWARDS HIGH-DIMENSIONAS TASKS FROM DEMONSTRATION WITH CAMPLED-BASED MPC
provides an experiment of MPAIL Figure 13
training an agent in the Isaac Lab implemen- tation of the Ant-v2 environment as step to- wards high-dimensional applications. As ex- pected, MPPI's (vanilla) sampling procedure struggles be competitive with paseey-based spaces. However, MPAIL demonstrates signs of life in enabling MPPI to optimize a state space otherwise considered extremely challenging for sample-based planning. Note that Isaac Lab's Ant implementation prescribes a 60- dimensional observation and 8-dimensional action space, rather than Mujoco's 26- 16-dimensional ob- servation. Thus, the space is 120-dimensional for costing cp(s, s') and 80-dimensional for MPPI with planning horizon of 10 timesteps. Despite this, learned cost is capable of guiding real-
Reward (Normaltodd)
1.00
0.88
0.75
0.6
2.4
MMM 2M0
0.25
0.00
-0.25
optimization in higher-dimension
3.0
32
64
128
620
Number Demos
Timestaps
Figure 13: Isaac Lab Ant-v2 Experiment with MPAIL (OM) and GAIL.
25

---

Published as a conference paper at ICLR 2026
Benchmark results from Figure de de tteaagggeettdd across number of demonstrations MPAIL results for the Cartpole task is differentiated MPAII. (OM) to clarify that its Model is learned fully Online.
500
200
400
300
Ieeationns
Ieeationn
100
200
demos
100
0
Expert
500
WWV
200
400
300
Random
100 iterations
Iterations
200
100
AIRL
500
GAIL
200
400
300
100 Iterations
Iterations
(OM)
200
100
MPAIL
500
200
400
300
Ieratiins
morations
100
200
100
1.00
0.50
0.25
0.00
1.00
Doward Sanrrlira
0.75
0.50
0.25
0.00
Figure 11:
Dawers (ommalized
26

---

Published as a conference paper at ICLR 2026
H
H 10
# 20
1.00
1.00
1.00
Reward (Normalzzed)
0.75
0.75
0.75
0.25
0.25
0.25
0.00
0.00
0.00
-0.25
-0.25
-0.25
Both
Value-only
Cost-only
-0.50
100
Timesteps
Timesteps
Timesteps
Figure 12: Ablating single-step costs ca and value Va across different Horizon (H) lengths. "Cost-only" experiments are performed by not evaluating V6 the final state in each model rollout. "Valuc-only" experiments are performed by not evaluating on the H-step transtrramstions.. See Algorithm line 12 for exact usage.
time vanilla MPPI optimization execute walking behaviors in the ant task, albeit slower, from few demonstrations.
"Remembering" locally optimal policies through learned policy-like proposal distribution may help planning capabilities generalize to higher-dimensional spaces. Additionally, modeling dynam- ics in latent-space and using model ensembles have been shown to significantly improve perfor- mance in model-basd madel-based reinforcement learning (Hansen et al. 2022) and are promising directions for future work for eighddimensional tasks. Finally, Figure 10 illustaate key takeaway of this frame- work: learning costs through MPAIL remains orthogonal to other works which seek to improve sample-based MPC through sampling (Xue al. [024] Vlahov al., [024] [aakks & Boots (202)), optimization (Vlahov et al. [2024, Sacks et al. 2024). and dynamics (Hansen et al. 2024). We be- lieve that integration of developments
MPC along with application-specifee cost regularization
(Finn al. 2016) may be critical for exploring the full potential of planning from observation.
27


Reviews:
[{"review_id": "CZVKudP2Gn", "reviewer": "Reviewer_9pVj", "summary": "This paper introduces Model Predictive Adversarial Imitation Learning (MPAIL), a novel framework for teaching agents to perform tasks by observing demonstrations, a problem setting termed Planning-from-Observation.\n\nThe core idea is to unify the typically separate processes of learning a reward function (inverse RL) and using it for online planning (MPC). MPAIL achieves this by using a planner, specifically Model Predictive Path Integral control (MPPI), as the \"generator\" within an Adversarial Imitation Learning loop. This allows the agent to simultaneously learn a reward (cost) function and a value function while interactively improving its planning capabilities, all from observation-only demonstrations without expert actions.\n\nThrough experiments in simulated control tasks and a real-world robot navigation task using a single demonstration, the authors demonstrate that MPAIL offers significant advantages over traditional policy-based AIL methods. These benefits include improved generalization to out-of-distribution states, greater robustness, better sample efficiency, and enhanced interpretability by providing insight into the agent's decision-making process.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of unifying the reward learning (IRL) and online planning (MPC) into a single, end-to-end training process is new.\n\n- It provides good experimental evidence that using a planner as the generator improves generalization and robustness in out-of-distribution states compared to standard policy-based AIL methods.\n\n- The paper demonstrates successful real-world application through a \"Real-Sim-Real\" experiment, where the agent learns to navigate from just a single, noisy, and partially observable demonstration.\n\n- Benchmarking results show that MPAIL is sample-efficient, achieving good performance with fewer environmental interactions than established AIL algorithms like GAIL.", "weaknesses": "- The method's scalability to high-dimensional action spaces is highly questionable. The paper relies on Model Predictive Path Integral (MPPI), a sample-based planner that suffers from the curse of dimensionality. While it works for low-dimensional actions like vehicle control (2D action), its performance degrades sharply as the action space grows. The paper's own experiment on the Ant environment (Figure 12), which has an 8-dimensional action space, demonstrates this weakness clearly: MPAIL shows \"signs of life\" but is drastically outperformed by the policy-based GAIL, failing to achieve competitive performance. This suggests the proposed method is not a viable solution for more complex, high-dimensional robotics tasks like manipulation without non-trivial extensions.\n\n- The experimental validation is conducted on relatively simple tasks. The primary successes are shown on a navigation task and cartpole balancing. These tasks do not sufficiently challenge the planning component or prove the method's applicability to complex behaviors in robotics.\n\n- A natural baseline is to replace the AIL in the proposed method with online behavior cloning. This baseline is simpler, computationally cheaper than adversarial training, and could potentially perform well on the presented tasks. By not comparing against it, the paper fails to justify the added complexity and potential instability of its adversarial training loop.\n\n- The planner's performance is entirely dependent on the accuracy of the dynamics model. For the main navigation task, the authors use a pre-defined, approximate model. For other tasks, they learn a model online. In either case, model error (sim-to-real gap or learning inaccuracies) can lead to catastrophic planning failures. The paper does not sufficiently analyze the method's sensitivity to model error or propose robust mechanisms to handle it, which is a primary challenge for any model-based approach in the real world.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0OMLrT6z05", "reviewer": "Reviewer_oifv", "summary": "This paper presents a new pipeline for learning from demonstration that conducts adversarial imitation learning while construct the policy through a model-based planning approach. The authors target the crucial setting of learning in partially observable environments that requires real-time planning and control and propose to roll out the optimal policy using a model predictive planning algorithm. The authors formulate a new previous-policy-regularized policy optimization objective that matches the idea of the proposed planning procedure.", "questions": "L778: why the planning set a 0 action for the last timestep? Does this affect the initial plan at the next iteration?\n\nIt seems that this paper mainly targets continuous action space. Can it be applied to discrete action space? If so, how can it be adapted to handle discrete actions?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "This paper presents a novel planning based policy optimization in AIL that matches well with the real deployment of the learned policy.\n\nThis paper presents several theoretical results that connect the KL-constrained policy optimization using MPPI with the adversarial imitation learning objective.\n\nThis paper demonstrates the application through deployment in a real world environment.", "weaknesses": "This paper targets the problem of learning from observational demonstrations in a POMDP setting. However, it seems that all the technical developments are based on the full state information.\n\nThe major contribution comes from the usage of model-based planning algorithm MPPI for AIL, but the motivation of using MPPI instead of other methods require more discussion. Can other model-based planning methods be used? What is the advantage of MPPI and why other methods here are inferior?\n\nThe paper is not well written overall. The presentation leaves many points unclear. Many phrases and notations used are unnatural and hard to understand. For example,\n- L116: the proposed method relies on a predictive model and a cost function for model-based planning. However, it is unclear whether these models are trained from data or known a priori. If they are learned from data, how are these models trained? This is only mentioned until the experiments section and it would be better to explain this in the method section.\n- L176: It is unclear how the action $a$ in $\\bar{c}(s,a)$ is used in the definition and how the transition probability $\\mathcal{T}$ is defined. Does $\\mathcal{T}$ depends this particular $a$ or it depends on a specific policy like $\\mathcal{T}^{\\pi}$?\n- L202: “By decomposing the agent this way, we require not the policy, or solution, to generalize but the reward, or problem”. What do solution and problem here specifically refer to?\n- Typo: L189: value function?\n\nI would strongly recommend the authors check the writing to improve the presentation, and I look forward to the authors’ feedback for further evaluations.\n\nThe paper would benefit from comparisons with more imitation learning algorithms as baselines other than GAIL/AIRL. From the figures, the performance of MPAIL converges much slower than GAIL in Cartpole.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ijlNHh0D7F", "reviewer": "Reviewer_cqwJ", "summary": "This paper presents a novel framework that unifies Inverse Reinforcement Learning (IRL) and Model Predictive Control (MPC) into a single, end-to-end imitation learning approach. Traditional imitation learning methods, such as Adversarial Imitation Learning (AIL), rely on policy optimization and require access to expert actions, limiting their interpretability and robustness in real-world settings. In contrast, MPAIL performs planning-based adversarial imitation directly from observation-only demonstrations - state trajectories without corresponding actions - by embedding a model predictive planner (specifically, Model Predictive Path Integral control, MPPI) within the adversarial learning loop. This integration allows the system to simultaneously learn a reward (or cost) function and optimize the planner online, effectively merging the training and deployment phases of IRL-MPC pipelines. The proposed framework operates under the Planning-from-Observation (PfO) paradigm, enabling interpretable, safe, and generalizable imitation without explicit action data. The key novelty lies in replacing the policy-based “generator” in adversarial imitation learning with an online MPC planner that continually solves short-horizon trajectory optimization problems guided by the learned reward and a value function that bootstraps long-term outcomes. The experiments show that MPAIL achieves better generalization, robustness, and interpretability than policy-based imitation methods like GAIL or AIRL, particularly in out-of-distribution and real-world settings. By integrating planning directly into the adversarial learning loop, it enables stable, sample-efficient imitation even from sparse or partially observable demonstrations.", "questions": "Please see weaknesses in the section above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and methodically builds from theoretical formulation to algorithm design and experiments, making a complex contribution accessible and reproducible.\n2. The approach successfully handles observation-only demonstrations and partial observability, showing real-world viability for robot learning from minimal, ambiguous expert data.\n3. The experiments span both simulated and real-world settings, demonstrating strong improvements in generalization, robustness, and sample efficiency compared to established baselines like GAIL, AIRL, and IRL-MPC.", "weaknesses": "1. Most experiments focus on navigation and simple control tasks, so it’s unclear how well MPAIL would perform on more complex, high-dimensional problems like manipulation or multi-agent settings.\n2. Although the few-shot setup is intentional, relying on very few demonstrations may make the results sensitive to the choice or quality of those examples, which isn’t extensively analyzed.\n3. The paper could better isolate which components (e.g., MPPI planner, value bootstrapping, or reward formulation) contribute most to performance improvements.\n4. The real-world experiment is limited to a single small robot platform and one trajectory, so it’s hard to assess generality or scalability to other hardware and environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KijsI62Q0J", "reviewer": "Reviewer_FaBU", "summary": "This paper proposes Model Predictive Adversarial Imitation Learning (MPAIL), a principled framework that integrates Model Predictive Control (MPC) and Generative Adversarial Imitation Learning (GAIL) to address imitation learning (IL) from observations.\nUnlike conventional adversarial IL methods that rely on a single-step policy update, MPAIL introduces a planning-based adversarial loop that optimizes both the policy and the discriminator through receding-horizon trajectory optimization. The key insight is that short-horizon predictive planning improves stability and sample efficiency by leveraging model-based rollouts for adversarial training. Specifically, MPAIL replaces GAIL’s policy gradient updates with MPC-based planning that iteratively minimizes the discriminator reward while respecting dynamics constraints. This results in a control sequence that approximates expert behavior through direct trajectory matching rather than relying solely on gradient backpropagation.", "questions": "1.\tHow is the dynamics model obtained for environments without ground-truth simulators?\nIf learned, how is model uncertainty incorporated into MPC (e.g., via ensembles or stochastic constraints)?\n\n2.\tHow sensitive is performance to planning horizon H?\nDoes longer horizon always improve imitation fidelity, or does it destabilize discriminator updates due to compounding model errors?\n\n3.\tDoes the MPC loop reduce adversarial oscillations by smoothing gradient updates?\nHave you analyzed the frequency of discriminator-policy mode collapse?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper’s core contribution of embedding adversarial imitation learning within a model predictive control loop is novel and elegant. This approach reframes IL as a constrained planning problem, where the policy implicitly solves an adversarial objective using model-based prediction instead of policy gradients. This is conceptually distinct from prior model-based IL works (e.g., PILCO, Dyna-IL) that use learned dynamics for data generation but retain standard policy optimization. Here, the MPC layer becomes the optimizer itself, aligning adversarial rewards directly with feasible trajectories. This rethinking of the optimization hierarchy is both theoretically meaningful and practically beneficial. It unifies imitation learning, trajectory optimization, and adversarial training into a single, closed-loop control process. \n\n2.\tThe paper goes beyond empirical demonstration by providing a stability analysis grounded in control theory. The authors derive sufficient conditions for local convergence to expert trajectories based on Lipschitz continuity of the discriminator and bounded model approximation errors. Although the proof is local, it provides valuable intuition about how predictive horizons stabilize adversarial learning, reducing the oscillatory updates common in GAIL-style min–max optimization. The connection between receding-horizon planning and bounded regret minimization in adversarial settings is particularly insightful, offering a bridge between MPC theory and generative adversarial dynamics. \n\n3.\tMPAIL consistently outperforms baselines, achieving higher reward imitation and smoother convergence. Notably, the ablation showing improved robustness to observation noise and model misspecification provides strong evidence of the approach’s practical value.", "weaknesses": "1.\tA central limitation of MPAIL is its assumption of an available or learnable model for predictive control. While this is standard in MPC, it restricts applicability to domains where dynamics are known or can be learned accurately. The authors claim that “MPAIL is model-free when using learned dynamics,” but this statement is misleading—using an imperfect learned model can degrade adversarial convergence. Empirical evaluation with explicit model-error quantification (e.g., using an ensemble dynamics model) is missing.\n\n2.\tThe theoretical guarantees focus on local stability near expert trajectories but do not address global convergence or nonconvex adversarial interactions. The adversarial MPC game may still exhibit multiple equilibria or oscillations if the discriminator is poorly calibrated. While the empirical stability is strong, the paper’s theoretical claims should be tempered.\n\n3.\tMPAIL performs optimization through MPC at every iteration, which introduces substantial computational overhead compared to standard policy-gradient IL. The authors briefly mention using MPPI (Model Predictive Path Integral Control) for rollout sampling, but do not provide timing benchmarks or hardware requirements.\n\n**Other Important Issues**:\n\n1. The authors used the incorrect template (ICLR 2025) to prepare the submission, and \n2. more importantly, the corresponding author's email is displayed on the first page, which potentially violates the double-blind review process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5jncFSv80X", "reviewer": "Reviewer_2sCb", "summary": "The paper presents Model Predictive Adversarial Imitation Learning (MPAIL), which combines model predictive control (MPC) with adversarial imitation learning to enable planning from observation-only demonstrations. Instead of learning a policy directly, MPAIL trains a discriminator-based reward and deploys it through a physics-based planner (MPPI) with a value function for long-horizon reasoning. Experiments show that this approach improves out-of-distribution generalization and real-world robustness, including a single-demonstration RC car navigation task, though the evaluation is limited to one type of real task and a few baselines.", "questions": "Questions*\nHow is partial observability handled in practice? What state representation does the planner use when the observation is limited?\n\nHow is the Kinematic Bicycle model integrated with partial information? \n\nCould the authors test OOD generalization in the learned-model setting (e.g., Cartpole) to separate the benefits of physics priors from the algorithm itself?\n\nPlease include a short Baselines section summarizing the setup, inputs, and weaknesses of GAIL, AIRL, IRL-MPC, and MPAIL.\n\nDoes GAIL’s circling failure continue at convergence?\n\nHow does the method compare to a policy learning methods that assumes the same world model at convergence?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper crisply frames planning‑based AIL and operationalizes it via MPAIL: MPPI + AIRL‑style reward. The KL‑regularized perspective that leads to MPPI as the optimizer is clean and well explained.\n\nThe state‑transition formulation directly targets Learning‑from‑Observation under POMDPs (no actions), which is the harder and more realistic setting.\n\nMethodology is reproducible, with transparent algorithms, hyperparameters, and implementation details.\n\nEmpirical questions in the experiments are well-posed and probe the main claims of the paper.", "weaknesses": "Partial observability details are underspecified. While the POMDP framing is explicit, how the planner’s state is constructed from limited observations in each experiment is not fully clear. For instance, the navigation experiments plan with a Kinematic Bicycle model, yet in the single‑demo real experiment the expert data are “position and body‑centric velocity” only; it remains unclear how heading/other latent state are estimated for MPPI at train/test time and how the discriminator’s input (state vs transition) aligns with the planner’s state. A concise subsection clarifying the observation to state pipeline (filters, history windows, or learned encoders), and the exact planner state per task, would strengthen soundness.\n\nAttribution of robustness to the physics prior vs learned reward. The OOD gains in navigation may partly stem from MPPI with a physics model (even if approximate). The paper takes a step toward a learned‑model regime on Cartpole, but does not repeat the OOD evaluation there. Without such a test, it’s hard to tell how much robustness persists when the dynamics must be learned and are mis‑specified. An OOD study in the learned‑model setting (e.g., Cartpole) would solidify the generalization claim.\n\nMissing “performance at convergence” perspective. When the model of the world is assumed, sampling becomes cheap. Therefore, I would like to see the comparison of the methods at convergence.\n\nA dedicated “Baselines” section summarizing inputs, optimization style, and deployment (policy vs planner, known vs learned dynamics) would reduce confusion about advantages/disadvantages among IRL‑MPC, AIRL, GAIL, MPAIL and the role of multimodality in failures.\n\nReal‑world scope is narrow. The single‑task navigation demo is valuable, but claims about safety/robustness would be more convincing with additional real tasks\n\nClarity nits.\nThe abstract/intro list many keywords; grounding them in the concrete setting focused on the contribution would sharpen the message.\nMinor: the manuscript seems to use an older template.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)