Predictive Preference Learning from Human Interventions
Authors: Haoyuan Cai
Zhenghao Peng
Bolei Zhou
Venue: NeurIPS 2025 spotlight ()
Keywords: Imitation Learning
Human-in-the-loop Reinforcement Learning
Shared Autonomy
Learning from Human Preference
Primary area: reinforcement_learning

Abstract: Learning from human involvement aims to incorporate the human subject to monitor and correct agent behavior errors. Although most interactive imitation learning methods focus on correcting the agent’s action at the current state, they do not adjust its actions in future states, which may be potentially more hazardous. To address this, we introduce Predictive Preference Learning from Human Interventions (PPL), which leverages the implicit preference signals contained in human interventions to inform predictions of future rollouts. The key idea of PPL is to bootstrap each human intervention into L future time steps, called the preference horizon, with the assumption that the agent follows the same action and the human makes the same intervention in the preference horizon. By applying preference optimization on these future states, expert corrections are propagated into the safety-critical regions where the agent is expected to explore, significantly improving learning efficiency and reducing human demonstrations needed. We evaluate our approach with experiments on both autonomous driving and robotic manipulation benchmarks and demonstrate its efficiency and generality. Our theoretical analysis further shows that selecting an appropriate preference horizon L balances coverage of risky states with label correctness, thereby bounding the algorithmic optimality gap. Demo and code are available at: https://metadriverse.github.io/ppl.

Predictive Preference Learning from Human Interventions
Haoyuan Cai, Zhenghao Peng, Bolei Zhou Department of Computer Science, University of California, Los Angeles
Abstract
Learning from human involvement aims to incorporate the human subject to moni- tor and correct agent behavior errors. Although most interactive imitation learning methods focus correcting the agent's 's action at the current state, they do not adjust actions in future states, which may be potentially more hazardous. To address this, we introduce Predictive Preference Learning from Human Interventions (PPL), which leverages the implicit preference signals contained in human interventions inform predictions of future rollouts. The key idea of PPL is to bootstrap each human intervention into T. future time steps, called the preference horizon, with the assumption that the agent follows the same action and the human makes the same intervention in the preference horizon. By applying preference optimization on the safety-critical regions
these future states, expert corrections are propagated where the agent is expected to explore, significantly improving learning efficiency and reducing human demonstrations needed. We evaluate our approach with ex- periments on both autonomous driving and robotic manipulation benchmarks and demonstrate its efficiency and generality. Our theoretical analysis further shows that selecting an appropriate preference horizon L balances coverage of risky states with label correctness, thereby bounding the algorithmic optimality gap. Demo and code are available at: https: pettps://metadee ienttadiiveese gi github. io/ppl
1 Introduction
Effectively leveraging human demonstrations to teach and align autonomous agents remains a central challenge in both Reinforcement Learning (RL) [46] and Imitation Learning (IL) [17]. In the literature of RI. and more recent RI from Human Feedback (RLHF), the agent explores the environment through trial and error under human feedback guidance, and the learning process hinges carefully crafted reward function that reflects human preferences. However, RL algorithms often require large number of environment interactions learn stable policies, and their exploration can lead agents to dangerous or task-irrelevant states [40 27 In contrast, IL methods train agents emulate human behavior using offline demonstrations from experts. Nevertheless, IL agents are susceptible to distributional shift because the offline dataset may lack corrective samples in safety-critical or ontt-ofdiitrrbutiinn states
35
incorporates human participants to intervene in the training process and provide online demonstrations. Such methods have improved alignment and learning efficiency in a wide variety tasks, including robot manipulation 7 g].
Interactive Imitation Learning (IIL) 2
34 1
28
28 and even the strategy game StarCraft [39]. One line of research on confidence-based IIL designs various task-specific criteria to request human help, including uncertainty estimation 223 and confidence completing the task 4 38). In contrast, an increasing body of work focuses on learning from active human involvement, where human subjects actively intervene and provide demonstrations during training when the agent makes mistakes [1]
autonomous driving [27
28 Compared to confidence oofidence-bbsedd IIL, active human involvement can ensure training safety 27
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1) Trajectory Prediction Facilitating Human Intervention
Can turn right here?
You will crash soon!
Let me help you.
2) Learning Human Preferences in Forecasted Rollouts
Human rejects my right-turn.
I understand human preferences in these states.
Figure Our Predictive Preference Learning from Human Interventions. (Top) Our approach forecasts the agent's upcoming trajectory (the red dotted path) and visualizes it for the human expert, who will intervene the forecasted path indicates an upcoming failure. (Bottom) A single intervention is then interpreted as hypothesized preference signals across the predicted states. These signals reflect the agent's 's imputed imagination of what the expert would prefer, guiding the policy to avoid the risky maneuver in similar future contexts. This integration of proactive forecasting and preference learning accelerates policy improvement and reduces the total number expert interventions required.
and does not require carefully designing human intervention criteria for each task 12 However, these methods require the human expert to monitor the entire training process, predict the agent's future trajectories, and intervene immediately in safety-critical states (28), imposing significant cognitive burden on the human participant. In addition, these methods often correct the agent's behavior only at the current intervened state, penalizing undesired actions step by step. For instance, in HG-DAgger [1]. the agent optimized to mimic human actions solely at the states where interventions occur. In practice, it is intuitive that the agent may repeat similar mistakes in the consecutive future steps + 1,..... ......t + repeatedly provide corrective demonstrations in these regions, compromising training efficiency (17]
following an error at step f. As a result, the expert must
In this work, we propose a novel Interactive Imitation Learning algorithm, Predictive Preference Learning from Human Interventions (PPL., to learn from active human involvement. As shown in Fig. [ our approach has two key designs: First, we employ an efficient rollout-based trajectory prediction model to forecast the agent's future states. These predicted rollouts are visualized in real time for the user, helping human supervisors proactively determine when an intervention is necessary, Second, our algorithm leverages preference learning the predicted future trajectories to further improve the sample efficiency and reduce the expert demonstrations needed. Such designs bring three strengths: (1) They mitigate the distributional shift problem in IIL and improve training efficiency. By incorporating anticipated future states into the training process, our method constructs a richer dataset, especially in safety-critical situations. This expanded dataset offers more information than expert corrective demonstrations in human-intervened states only. (2) The preference learning reduces the agent's ' visits to dangerous states, thus suppressing human interventions in safety-critical situations. (3) By visualizing the agent's 's predicted future trajectories in the user interface, we significantly reduce the cognitive burden the human supervisor constantly anticipate the agent's behavior.
Our contributions can be summarized as follows:
1. We introduce novel Interactive Imitation Learning (IIL) algorithm that leverages trajectory prediction to inform human intervention and employs preference learning deter the agent from returning to dangerous states.
2. We evaluate our algorithm on the MetaDrive [6 and Robosuite both neural experts and real human participants, showing that PPL requires fewer expert monitoring efforts and demonstrations achieve near-optimal policies.
49 benchmarks, using
3. We present theoretical analysis that derives upper bound on the performance gap of our approach. This bound highlights that the efficacy of our method lies reducing distributional shifts while preserving the quality of preference data.

---

2
Related Work
Learning from Human Involvement. Many works incorporate human involvement in the training loop to provide corrective actions in dangerous or repetitive states. For example, Human- Humen-Gated DAgger (HG-DAgger) [15,, GnsembleeDAgger (33, Thrifty-DAgger [2], Sirius [19]. and Inter- vention Weighted Regression (IWR) (22 perform imitation learning on human intervention data. These methods do not leverage data collected by agents or suppress undesired actions likely to be intervened by humans, leading to the agent's 's susceptibility to entering hazardous states and thus harming sample efficiency. EGPO (27] PVP 28 and AIM 2 design proxy cost or value functions to suppress the frequency of human involvement. However, these approaches still require human supervisors to continuously monitor the agent's 's behavior throughout training and anticipate potential failures that may necessitate intervention. This continuous oversight imposes a significant cognitive load on the human expert and can limit scalability. Furthermore, these methods do not exploit the agent's predicted future trajectories that the expert might identify as potentially leading to undesirable outcomes, which necessitates repeated corrective demonstrations in such situations.
Preference- RL. A large body of work focuses on learning human preferences by ranking One prominent paradigm, reinforce-
pairs of trajectories generated by the agent 66 9 34 ment learning from human feedback (RLHF), first trains reward model offline human preference data and then uses that model to guide policy optimization 6 25 43) RLHF has achieved impres- sive results in domains ranging from Atari games [6] large language models (25]. Alternatively, methods such as Direct Preference Optimization (DPO) 31.. Contrastive Preference Optimization
44 37
26
(CPO) 45), and related variants I 244
bypass explicit reward-model training and instead directly
optimize the policy to satisfy preference labels via a classification loss.
However, applying RLHF and DPO to real-time control problems faces challenges due the need for extensive human labeling of preference data [I]. These labels are inherently subjective and prone to noise [37] Moreover, acquiring high-quality preference dataset and achieving near-optimal policies often requires a substantial number of environment samples, thereby imposing a considerable burden human experts [10]. In contrast, framework elicits preferences in online, interactive manner: experts review the agent's 's predicted future trajectory at each decision point and intervene when failure is anticipated; these interventions are then converted into contrastive preference labels. This real-time preference collection enables the policy to adapt continuously to the evolving state distribution and receive targeted feedback precisely where it is most needed. In summary, our approach PPL bridges preference-based RL and imitation learning by demonstrating that DPO-style alignment techniques can be effectively adapted control problems within an interactive imitation learning framework.
3 Problem Formulation
In this section, we introduce our settings of interactive imitation learning environments. We use the Markov decision process (MDP) M (S, A, P,r, r, 7, do) to model the environment, which contains a state space S, an action space A, a state transition function P S x A + S, a reward function r S x A [Rmin, Rmax], a discount factor y E (0, 1), and initial state distribution du S - [0, 1]. We denote a(a 8) S x A - [0, 1] as a stochastic policy. Reinforcement learning (RL) aims to learn novice policy n., (a) () that maximizes the expected cumulative return
(TT,) = I ΣΣ [t)], wherein T = (so: ag: $1, ...). is the trajectory sampled from T-P__ t=
trajectory distribution Pr,, induced by Tn, do and P. We also define the discounted state distribution under policy n. as dr,, (s) (1-7)
E 7'[[s *]]. In this work, we consider the reward-free T~ PTe. 1=0
setting where the agent has no access to the task reward function r(s,
In imitation learning (IL), we assume that the human expert behavior on follows a human policy (a s). The agent aims to learn nn from human expert trajectories Th ~ PA,,, and it needs to optimize T. to close the gap between Ta ~ Pr,, and тІ- Prior works on imitation learning have shown that using an offline expert demonstration dataset may lead to poor performance due to oft-o--distritution states 36 41. Therefore, interactive imitation learning (III.) methods incorporate human expert into the training loop to provide online corrective demonstrations, making
3

---

the state distribution of expert data more similar to that of the novice policy [28]3]] the human expert monitors the agent and can intervene and take control if the agent's action a, at the current state & violates the human's 's desired behavior or leads to a dangerous situation. We use the deterministic intervention policy T(s, ax): S x A - {0. 1) to model the human's 's intervention behavior, where the agent's & action follows the novice policy On nnp( s), and the human subject takes control when (s, ax)
35]. During training.
With the notations above, the agent's actual trajectories during training are derived from the following shared behavior policy
To(a 8) (a s) T(s, a)) + (a s))(s)
(1)
wherein S a(s,a')T,, (a' s) is the probability of the agent taking action an that will be rejected and intervened by the human expert.
Preference Alignment. Recent works on preference-basee RL have also leveraged offline preference datasets to learn human-aligned policies [3T] 45, [ Given an offline preference dataset Prre where each preference data (s, a== state 8, we can learn an agent policy Tu that aligns with the human preference model. The Contrastive Preference Optimization method [5 uses the following objective to train an agent policy Ty from the preference dataset D pref:
Pret means that the human expert prefers the action a+ over
E (Te) = (e.a+ a DnDDe
[log a (Blog > - B log Fa(a [
where (-) is the Sigmoid function, and
is a hyperparameter.
Trajectory Prediction Model. In this work, we allow the agent to access a short-term trajectory prediction model f(s, Onz H). Given the current state and the agent's action On, we can predict the
agent's ' trajectory [&, an, H) = (s, .1, ,, in the next steps, where &; the predicted state that
the agent will reach if the agent applies the action on for steps from the state s. The implementation detail of f is in Sec. 4.3
4 Method
4.1
Predictive Preference Learning from Human Interventions (PPL)
We propose PPL (Fig. 2). efficient interactive imitation learning method that emulates the human policy with fewer expert demonstrations and less cognitive effort. The key idea of PPL is to learn human preferences from data generated by future-trajectory prediction model. We illustrate the human-agent interactions in Fig. [ and how PPL infers human preference in Fig. (right).
During training, the human subject monitors the agent-env interaction in each state (Fig. (left)). The novice policy Tn suggests an action a, for the current state s. Instead of executing a, immediately, we query the trajectory prediction model fs, an,, H)) obtain predicted rollout T = f(s 0n H) (s, $1, HH), which we visualize for the human expert. The expert then uses determine whether the agent will fail in the next H steps, such crashing into vehicles or going off the road. so, the expert will provide corrective actions aj ~ TT(s) for the next H steps, depicted by the blue trajectory in Fig. 2 If the expert believes intervention is needed, the agent continues to use its own policy n. for the next H steps.
We introduce preference learning on the predicted trajectories because it is difficult to learn corrective behavior purely from the expert's s demonstrations in safe states. By visualizing predicted rollouts, experts anticipate unsafe trajectories before the agent actually enters them and intervenc preemp- tively. As a result, the state distribution covered by these early interventions differs substantially from the on-policy distribution of the novice policy, creating distributional shift that standard imitation or on-policy correction cannot address. Therefore, instead relying solely on expert demonstrations, collect preference labels over the predicted rollouts (Fig. Right) that the agent can learn the correct behavior those risky states.
Whenever the expert intervenes at state we interpret this as indicating that continuing with an would lead to unsafe or undesirable outcomes along the predicted trajectory. As shown Fig. 2 (right), to capture this preference, assume the expert prefers on over an at state and each of

---

2) Agent infers human preference predicted future states.
Agent's Exploratory Trajectory
Human Demonstration
Human prefers dg over on al 81
Agent's Predicted Trajectory
Oh
(8,,0
Gp
Human prefers ag over On at 82.
1) Human intervenes to prevent future crash.
Preference Buffer Dent
You should do as in state s, not dg.
(s,0x)
(3,,0"
Human Bufffer
aA
On
DA
On
(BC(TT)
E [log malas
[+]
loge a (Glog +
()
Glog xo(a" [+))]
(AAL)-D,
Figure 2: Illustration of Predictive Preference Learning. (Left) At each decision point, the agent proposes an action, and its future trajectory is predicted and visualized. The human expert reviews this rollout and intervenes only when a potential failure is anticipated. The intervention is recorded alongside the state into the human buffer Dr for behavioral cloning. (Right) Each recorded interven- tion then converted into contrastive preference pairs over the predicted future states 81,-. ,8r. These preference tuples are stored in a preference buffer Dper and used to train the policy via a contrastive classification loss, propagating expert intents into regions the agent likely explore.
the first L predicted states s.. ,5 for some preference horizon L H. For each i S L, we add the tuple ($,, at = ab, a- = an) to the preference dataset Prref- We note that in each tuple (3;, a = 4n a = a,), both Λ₁ and an are sampled at the current state s, not the predicted future states s,, because the exact human corrective actions at hypothetical future states observable. Still, the expert intervention at state implies that applying (n at the predicted states $1....,81, 8r, rather than continuing with an, is more likely to prevent the dangerous outcome in the end of the predicted trajectory ($ Hence, our construction of the preference dataset ensures that it faithfully captures the expert's corrective intent across the predicted horizon.
not directly
The preference horizon L controls the length over which we elicit preferences in the predicted trajectory. A small L may fail to capture enough risky states, while large L risks applying preferences where the corrective action d₁ at state s no longer matches what an expert would do in those imagined states 6; In Theorem 4.1 prove that under mild assumptions, the performance gap of our learned policy is bounded by terms reflecting the state distribution shift and the quality of the preference labels, implying that ideal preference horizon L should balance these two error terms. We also illustrate how the choice of affects the performance of PPL in Fig. g
We train the novice policy n. using two complementary objectives. First, we apply a behavioral cloning loss on expert demonstrations Dí:
LBC(TT) (s.0t)~-D,
E
[log (ax s)]
(3)
Second, inspired by Contrastive Preference Optimization (CPO) [45, use the preference- classification loss Eq. 2 over the predicted states in D Prrc- The final loss of the agent policy my is evaluated
[(TT) = Crre((TT)) (TT) + LBC(TT)
E
[log (8 (Blog (aa |s) - 3 log [s)]]
I )
Dlog To(ap | {)]
(s,at,e aa D
The workflow of our method PPL is summarized in Alg.
4.2 Analysis
We prove that the performance gap between the human policy Ty and the agent policy n., can be bounded by the following three eитог terms: 1) the state distribution shift ddist. 2) the quality of the preference labels 6pes, and 3) the optimization error e.
The first error term is defined as 6ais = Drv (dw. dprer), where dr,, (s) is the discounted state distribution of the agent's policy Wu. and dpecr(s) |Dportl BBss'att ) lis s). Here,
5

---

Test Success Rate of MetaDrive
Test Success Rate Table Wiping
Test Success Rate of Nut Assembly (Oust
IPL (Dus))
9.8
0.6
0.5
Rat
SuC00s
0.2
0.2
0.2
Stops
Stops
Stops
50k
100k
150k
Figure 3: The test-time performance curve of PPL and the IIL counterpart PVP (28 different environments. The x x-coordinate is the number of environment interactions, and the coordinate the agent's 's success rate in a held-out test environment, where the evaluation conducted without expert involvement. Compared to the IIL counterpart. our approach achieves much higher learning efficiency and reduces the expert's efforts needed.
under three
(Pv(P,Q) = I Q/a is the total variation distance between two distributions. This error term quantitatively measures the difference between the states actually visited by the agent and those contained in the preference dataset.
The second error term is defined as 6prel = IE DTv (Pissatt Poret), which arises from the mis-
seudgel
alignment of the positive actions in the preference dataset, the human action aş each tuple ($; an. an)) E Pret is sampled in state s instead of state &, That is, this error reflects the assumption that the human would still apply the same corrective action aд in a hypothetical future state &; reached after executing On for steps, which may not perfectly match what the expert would actually do. For any state in Prref with dprer(s) (s) > 0, the empirical preference-pair distribution in state follows
(an an) E( J~DDast I[s' s, a+ On
an
(5)
E(siaal a )rD 11s
The ideal preference-pan distribution any state in Prrer is simply the joint distribution of (an, an): Pracal (On, an)) = Ta (an [8) 8)
s) on A x A.
Finally, we define the optimization error of the agent policy n, as c = Cprer((T,) Geeer (p)) We recall that (T)
E [log a (3 log (aa s) B log T(a s))], where B is positive (s.at.m ,a )~Dptt
constant and (.) is the Sigmoid function. Under these notations, we have the following Thm. 4.1
Theorem 4.1. We denote the Q-function of the human policy as Q* (s, a). We assume that for any (s, a, a"), 1Q" a) Q*( a!)l S U, |log TA (a]s) log an(a' [s)] S M, and log Tn (a(a))) log T. (a' [ss] S M, where U, M > O are constants. When B is small enough, we have
J (m)) J (mT,) e ōpret + 8alst).
(6)
Here we explain the insights of Thm. 4.1 as follows. In our choice of the preference horizon L. the key is to balance the two error terms daist and ōprer. Recall that the distribution shift term daist measures how close the state distributions are when there is no human intervention (dr,)) and the state distribution represented in the preference dataset (dwert). Increasing L decreases daist because the preference dataset will contain more predicted states 5, from the agent's 's future trajectories. In contrast, the preference error term ópeet captures the misalignment between the true but unobserved human action at a future state t' >> t and the bootstrapped corrective action On at step t, which we assume would also apply f'. Therefore, the longer the preference horizon, the larger Speet because the difference between the human actions a, in state s and the predicted & grows as increases. In Fig. 8 visualize the effects of L the performance of PPL. We prove Thm. 4.1 in Appendix
4.3 Implementation Details
Tasks. As shown in Fig. we conduct experiments on control tasks and manipulation tasks with different observation and action spaces. For the control task, we consider the MetaDrive driving

---

experiments [16], where the agent must navigate towards the destination in heavy-traffic scenes without crashing into obstacles or other vehicles. The agent uses the sensory state vector E R as its observation and outputs a control signal = (0n, a1) E [-,,1] representing the steering angle and the acceleration, respectively. We evaluate the agent's learned policy in a held-out test environment separate from the training environments.
For manipulation tasks, we consider the Table Wiping and Nut Assembly tasks from the Robosuite environment [49]. In the Table Wiping task, the robot arm must learn to wipe the whiteboard surface and clean all of the markings. The positions of these markings are randomized at the beginning of each episode. The states are E t and actions are E IR' (3 translations in the X axes and 3 rotations around the XYZ axes). In the Nut Assembly task, the robot must grab a metal ring from a random initial pose and place it over a target cylinder at fixed location. The states are E R and actions are a E RR7, where the additional dimension in the action space represents opening or closing the gripper. In both manipulation tasks, the simulated URSe robot arm uses fxxe-impedance operational-space control to achieve the commanded pose.
fn H) = Trajectory Prediction Model. In PPL, we need to predict the future states f(s, an,
(s, st-1.-... , St) from the current state s. We implement f by running an H-step simulator rollout from the current state 8, repeatedly applying action a, to collect the sequence ($141)' 31+H). This H-step simulator rollout runs at up to 1,000 fps on CPU.
In real-world tasks such as autonomous driving, simulator rollouts often deviate from reality because vehicle dynamics parameters imperfect and other traffic participants behave unpredictably. To predict future motion with minimal overhead, prior work directly propagates the ego-vehicle's state through a physics model [18 29 13.. Following this approach,
the kinematic bicycle model [30] to simulate H = 10 steps, assuming all other traffic participants remain stationary. Compared with the data-driven approaches (50 21). this rule-based predictor requires only forward integration of a single vehicle and produces short-term trajectories whose accuracy closely matches simulator rollouts. This lightweight extrapolation method runs about 3,000 fis on CPU, enabling real-time prediction with minimal overhead. Our ablation studies confirm that replacing the simulator with our bicycle-me model predictions incurs negligible performance loss (Table 2 rows 9-10).
5 Experiments
5.1 Experimental Setting
Neural Policies as Proxy Human Policies. Experiments with real human participants are time- consuming and exhibit high variability between trials. Following the prior works on interactive imitation learning 10 27, in addition to real-human experiments, we also incorporate neural policies
in the training loop of PPL to approximate human policies Table 3 and The neural experts are trained using PPg-Lagrangian
(33 for 20 million environment steps.
In MetaDrive, the neural expert uses the following takeover rule when training all baselines and our method PPL: if the predicted trajectory
f(s On, H) contains any safety violation, such as crashes or going off the road, or the average speed is too slow, the expert takes control for the next H steps. In RoboSuite, the neural expert intervenes when the cumulative reward over the predicted trajectory falls below a threshold e. We set for the Table Wiping task and e = 2 for the Nut Assembly task.
In Table T we report experiments involving real humans in the MetaDrive safety benchmark. In 3 T and T we report experiments with the neural policy as the proxy human policy in the MetaDrive, Table Wiping, and Nut Assembly tasks, respectively.
Evaluation Metrics. In the Table Wiping task and Nut Assembly task, we report the success rate, the ratio of episodes where the agent reaches the destination. In the MetaDrive safety benchmark, we also report the episodic return and route completion rate during evaluation. The route completion rate is the ratio of the agent's 's successfully traveled distance to the length of the complete route.
We train each interactive imitation learning baseline five times using distinct random seeds. Then, we roll out 50 trajectories generated by each model in the held-out evaluation environment and average each evaluation metric as the model's 's performance. During the evaluation, no expert is involved. The standard deviation is provided. We fix H = 10 for all the interactive imitation learning baselines. In
7

---

(3) MevaDrive
(b) Robosulte: Table Wiping
(c) Robosulte: Nut Assembly
Figure 4: Human interfaces of the three tasks: MetaDrive (a), Table Wiping (b), and Nut Assembly
(c). In (a), the agent's ' forecasted trajectory (the red dots) leads collision, prompting the expert to intervene via the gamepad (blue dots show the predicted rollout of the expert). (b) and (c), the expert observes the agent's forecasted trajectory and intervenes via the keyboard if necessary.
Tutt Success Bate
Training Tecover Rate
3.4
Staoe
Sheps
Traihing Sphoode Cost
Staps
Figure 5: Training process of PPL in the MetaDrive environment with the human expert over 20K steps. We plot the test success rate (left), training takeover rate (top right), and training episodic safety cost (bottom right). During training, when the agent's ' forecasted trajectory (red dots) leads to collision, the human expert intervenes via the gamepad, and the corrected rollout shown (blue dots). When the agent's 's forecasted trajectory is safe, it is visualized in green dots. The agent becomes autonomous and performant during training, requiring fewer human interventions maintain safety.
PPL, we fix 3 = 0.1, choose L = for the MetaDrive benchmark and Table Wiping task, and set = 6 for the Nut Assembly task. In Fig. we show how the choice of L affects the performance of PPI. in the MetaDrive benchmark.
We also report the total number of human- transitions (human data usage) and the overall intervention rate, which the ratio of human data usage to total data usage. These show how much effort humans make to teach the agents.
Human Interfaces. Human subjects take control through the Xbox Wireless Controller or the keyboard and monitor the training process by visualizing environments on the screen. The predicted trajectories are updated every 10 steps (one second), so that the human expert can intervene promptly before the agent causes any safety violations and undesired behaviors.
Baselines. We test two imitation learning baselines: Behavior Cloning (BC) and GAIL | and
two confidence-bart IIL methods: Ensemble- -Dgger [23] and Thrifty-DAgger 12 Four human- in-the-loop IIL methods that learn from active human involvement are tested: Intervention Weighted Regression (IWR) [22], Human-Al Copilot Optimization (HACO) Expert Intervention Learning
(EII.) 42 and Proxy Value Propagation 28
5.2 Baseline Comparison
In Table report the performance of our PPL and all the baselines with real human experts in the MetaDrive safety benchmark. Our method PPL outperforms all the baselines and achieves a success rate of 76% within 10K steps. The whole experiment of PPL takes only 12 minutes on desktop computer with an Nvidia GeForce RTX 4080 GPU.
In Table and [ we report the performance of our PPL and all the baselines with neural experts as proxy human policies in MetaDrive, Table Wiping, and Nut Assembly tasks, respectively. We also plot the curves of the test-time success rate in Fig. 3 These tables and Fig. 3 show that PPL. achieves both fewer expert data usage and environment samples needed in both driving tasks and robot manipulation tasks while significantly outperforming baselines in testing performance. These

---

Table 1: Comparison of methods with training/testing statistics in the MetaDrive environment with the real human expert. The overall intervention rate is given together with the human data usage.
Method
Human-in- thepLoop
Training
Testing
Human Data Usage
Data Usage
Success
Episodic Return
Route Completion
Human Expert
20K
0.95 4:004
349.2 =
0.98
BC
X
20K
53.5 2
0.16 oon
GAIL
X
20K
1M
0,14
146.2 m
0.44 sue
Ensemble DAgger
/
3.8K (0.38)
10K
0.36 a:l
233,8 2
0.70 one
Thrity-DAgger
V
3,2K (0.32)
10K
0.45
221.5 2
0.62
PVP
4.9K (0.49)
10K
0.46 a.a
267.3 112
0.71 soe
IWR
/
5.2K (0.52)
10K
0.23 2:10
246.7
0.62 wuc
FII.
6.9K (0.69)
10K
0.01 2:13
137.3 2
0.40 ore
HACO PPI. (Ouns)
6.3K (0.63)
10K
0.11
154.7 1:147
0.45
2.9K (0.29)
10K
0.76 2:30
324.8 1.9
0.90 o.
Table 2: Ablation studies in MetaDrive with 10K total data usage. We use the neural expert as the proxy human policy.
PVP
Expert Data
Route Completion
Success Rate
PPL (Ours)
Method Imitation
Usage
1.9K
0.65
0.36
PPL. with random at
2.2K
0.73
0.45
PPL with random
2.3K
0.69
0.38
Figure 6: We plot the steering control sequences for both PVP and PPL the same MetaDrive
PPL with DPO
L6K
0,91
0.80
PPL with IPO PPL with SLiC-HF
2.6K
0.61
0.35
map, with arrows representing the steering an- gles every five steps. Both agents are trained to
3.0K
0.59
0.32
PPL BC only
2.0K
0.72
0.42
10K steps. Compared to PVP, our method yields smoother steering and more consistent speeds,
PPL with CPO loss only
5.8K
0.31
0.04
PPL rule-based f PPL (Ours)
1.K
0.91
0.78
especially when navigating close to obstacles.
188K
0.92
0.81
results suggest that our construction of the preference dataset accurately reflects human preferences and helps speed up imitation learning. In addition, Fig. shows that our method PPL produces smoother control sequences and generates trajectories that better align with human preferences.
5.3 Ablation Studies
In Table 2
we perform ablation studies of our PPL in the MetaDrive safety benchmark with the neural expert as proxy human policies.
Discarding positive or negative actions: In the first three rows of Table 2 we show that the advantage of our method PPL arises from the constructed preference pairs (š, a a in the preference data Dper (Fig. 2 (right)), instead of merely emulating the positive actions a or simply avoiding taking the negative actions a in the preference buffer. As shown in Table 2 discarding the negative actions and performing Behavior Cloning on the positive actions (Imitation on a at) leads poor performance, which is even worse than directly imitating the expert demonstrations in the human buffer D,, (PPL with BC loss only). In addition, replacing the positive actions by random actions (PPL with random a ) or the negative actions by random actions (PPL with random a ) also fails to solve the MetaDrive benchmark.
Preference-basee RL objectives: our learning objective Eq. 4 we use the Contrastive Preference Optimization (CPO) loss [45] to learn from the preference dataset 'D pres- In Table (rows 4-6), we also report the performance of using other preference-baseee RL objectives from Direct Preference Optimization (DPO) 31, IPO [1. and SLiC-HF (48). For DPO and IPO, we use reference policy trained by Behavior Cloning from 10K expert demonstrations. Table shows that using IPO (PPL with IPO) and SLiC-HF (PPL with SLiC-HF) objectives degrade the performance of PPL. Using the DPO objective (PPL with DPO) does not hurt the performance of PPL. However, the DPO objective requires access to pretrained reference policy, while our learning objective Eq. 4 does not.
Discarding the BC loss or preference loss: As shown in row of Table 2 discarding the CPO loss Epre in Eq. (PPL. with BC loss only) significantly damages the performance of PPI. Discarding the BC loss (PPL. with CPO loss only) also damages the performance, because the BC loss helps regularize our learned policy and avoid it deviating too much from the expert demonstrations.

---

PPL's Performance vs. Prediction Noise
PPL's Performance vs. Preference Horizon
0.80
[Ours]
Test Success Rate
0.75
PVP
0.75
Success
0.70
0.79
0.65
0.60
steps)
0.55
0.60
PPL (15K steps)
0.50
PVP (15K steps)
0.50
6.05
0:30
0.15
0.20
0.25
0.30
6
Trajectory Prediction Error
Preference Horizon
Figure Performance of PPL under varying vajectory-predictiin noise levels in MetaDrive. PPL still outperforms PVP when the trajectory predictor is imperfect.
Figure 8: Performance of PPL with different preference horizons L in MetaDrive with 10K and 15K total data usage. PPL has the best learn- ing efficiency when we set L 4 in MetaDrive.
Rule-based trajectory prediction model: Following Sec. we also implement a
trajectory prediction model f by simulating the ego-vehicle dynamics for H steps. Using a rule-based (PPL with rule-based f) has negligible effects on the performance of PPL. This shows that our method still outperforms the III. baselines even without relying on simulator rollouts.
5.4 Robustness Analysis
In Sec. 5.4] we evaluate PPL's robustness to noise in the trajectory predictor (Fig. 7p. We also visualize the effect of the preference horizon L on PPL in Fig. 8
In Fig. 7 we show that PPL is robust to noise in trajectory predictors. With an imperfect predictive model, PPL still outperforms all the baselines. We inject random Gaussian noise enoise the outputs of the trajectory predictor, and we set the norm ||enosel|2 |2 = e || Then we gradually increase the constant to test PPL's robustness to noises in trajectory predictors. We use MetaDrive. Table Wiping, and Nut Assembly environments following the same setups from Tables 3 4 and 5 respectively. We find that with noisy predictive model, PPL still outperforms all the baselines in MetaDrive and Table Wiping when the noise S 0.25. In Nut Assembly, PPL outperforms the baselines when e S 0.125.
In Fig. we visualize how the preference horizon L affects the test success rate of PPL in the MetaDrive safety benchmark with 10K and 15K total data usage. As L increases from 2 to 4, the agent gains additional corrective information from forecasted states in the preference buffer and achieves higher success rates. Beyond = 4, however, the benefit tapers off and eventually degrades, since overly long horizons yield less accurate preference labels. Therefore, we observe peak learning efficiency at L = 4. Notably, when 3 < L < 5. PPL trained for only 10K steps already outperforms PVP trained for 15K steps. With an appropriately chosen preference horizon, PPL can substantially reduce both training time and expert monitoring effort.
6 Conclusion
In this work, we propose Predictive Preference Learning from Human Interventions (PPL., a novel interactive imitation learning algorithm that applies preference learning over predicted future trajecto- ries to capture implicit human preferences. By converting each expert intervention into contrastive preference labels across forecasted states, PPL directs corrective feedback toward the regions of the state space the agent is most likely to explore. This approach substantially improves learning efficiency and reduces both the number of required demonstrations and the expert's 's cognitive load, without offline pretraining and reward engineering.
Limitations. We assume that the expert always knows the optimal corrective action and demon- strates it accurately, whereas human demonstrations can be suboptimal or inconsistent. Additionally, all our experiments are conducted in simulation. The effectiveness and safety of PPL on real robots operating in physical environments remain to be explored in future works.
Acknowledgment: This work was supported by the NSF Grants CCF-2344955 and IIS-2339769, and ONR grant N00014251211 The human experiment in this study approved through the IRB423-000116 at UCLA. ZP was supported by the Amazon Fellowship via UCLA Science Hub.
10

---

References
[1] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[2] Haoyuan Cai, Zhenghao Peng, and Bolei Zhou. Robot-gated interactive imitation learning with adaptive intervention mechanism. International Conference on Machine Learning. 2025.
[3] Sonia Chernova and Andrea L Thomaz. Robot learning from human teachers. Springer Nature, 2022.
[4] Sonia Chernova and Manuela Veloso. Interactive policy learning through confidence-basee autonomy. Journal of Artificial Intelligence Research, 34:1-25, 2009.
[5] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 027836492412236666 2023.
[6] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljaan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isahelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neurul Information Processing Systems 2017. December 4-9, 2017, Long Beach, CA. USA, pages 4299-4301 2017.
[7] Bin Fang. Shidong Jia, Di Guo, Muhua Xu, Shuhuan Wen, and Fuchun Sun. Survey of imitation learning for rohotic manipulation. International Journal of Intelligent Robotics and Applications, 3:362-369, 2019.
[8] Aditya Ganapathi, Priya Sundaresan, Brijen Thananjeyan, Ashwin Balakrishna, Daniel Seita. Jennifer Grannen, Minho Hwang, Ryan Hoque, Joseph Gonzalez, Nawid Jamali, al. Learning dense visual correspondences in simulation to smooth and fold real fabrics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11515 11515-11522. IEEE, 2021.
[9] Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subharao Kambhampati. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation. Advances in Neural Information Processing Systems, 34, 2021.
[10] Jocy Hejna, Rafacl Rafailov, Harshit Sikehi, Chelsca Finn, Scott Nickum, W Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: learning from human feedback without rl. arXiv preprint arXiv:2310.13633 2023.
[11] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg. Isabelle Guyon, and Roman Garettt editors, Advances in Neural In- formation P Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4565 4573,, 2016.
[12] Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S. Brown, and Ken Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning, 2021.
[13] Nico Kaempchen, Bruno Schiele, and Klaus Dietmayer. Situation assessment of autonomous emergency brake for arbitrary sehhicleto--vhicle collision scenarios. IEEE Transactions on Intelligent Transportation Systems, 10(4))678-687, 2009.
[14] Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Claude Sammut and Achim G. Hoffmann, editors, Machine Learning, Proceedings of the Nineteenth International Conference (ICMI. 2002), University of pages 267-274. Morgan Kaufmann, 2002.
New South Wales, Sydney, Australia, July 8-12, 2002,
[15] Michael Kelly, Chelsea Sidrane, Katherine Driggs- Campbell, and Mykel Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pages 8077 8083. IEEE, 2019.
[16] Quanyi Li, Zhenghho Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE trunsactions on pattern analysis and machine intelligence, 2022.
[17] Quanyi Li, Zhenghao Peng, and Bolei Zhou. Efficient learning of safe driving policy via human-ai copilot optimizaaion. In International Conference on Learning Representations, 2022.
11

---

[18] Chiu-Feng Lin, A Galip Ulsoy, and David J LeBlane. Vehicle dynamics and external disturbance estimation for vehicle path prediction. IEEE Transactions on Control Systems Technology, 833):508-518. 2000.
[19] Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke Zhu. Robot learning on the job: Human-in-the-lowwe autonomy and learning during deployment. The International Journal of Robotics Research, page 027836692211233901 2022.
[20] Huihan Liu, Yu Zhang, Vaarij Betala, Evan Zhang, James Liu, Crystal Ding, and Yuke Zhu. Multi-task interactive robot fleet learning with visual world models, arXiv preprint arXiv:2410.22689 2024,
[21] Yicheng Liu, Jinghuai Zhang, Liangii Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7577-7586, 2021.
[22] Ajay Mandlekar, Danfei Xu, Roberto Martin-Martín Yuke Zhu, Li Fei-Fei, and Silvio Savarese. Human- in-the-loop imitation learning using remote teleoperation. ArXiv preprint, abs/2012.067755 2020.
[23] Kunal Menda, Katherine Drigg-CCampbell and Mykel Kochenderfer. Ensembledagger. A bayesian approach to safe imitation learning. 2019 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 5041-5048. IEEE, 2019.
[24] Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198-124235, 2024.
[25] Long Ouyang, Jeff Wu, Xu liang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02153, 2022.
[26] Malayandi Palan, Gleb Shevchuk, Nicholas Charles Landolfi, and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences. In Robotics: Science and Systems, 2019.
[27] Zhenghao Peng, Quanyi Li. Chunxiao Liu, and Bolei Zhou Safe driving via expert guided policy optimization. In 5th Annual Conference on Robot Learning. 2021.
[28] Zhenghao Mark Peng, Wenjie Mo, Chenda Duan, Quanyi Li, and Bolei Zhou. Learning from active human involvement through proxy value propagation. Advances in neural information processing systems, 36, 2024.
[29] Romain Pepy, Alain Lambert, and Hugues Mounier. Reducing navigation errors by planning with realistic vehicle model. In 2006 IEEE Intelligent Vehicles Symposium, pages 300 307. IEEE, 2006.
[30] Philip Polack,
Florent Altché, Brigitte d' Andréa Novel, and Arnad de La Fortelle. The kinematic bicycle model: A consistent model for planning feasible trajectories for autonomous vehicles? In 2017 IEEE intelligent vehicles symposium (IV), pages 812-818. IEEE, 2017.
[31] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. Advances Neural Information Processing Systems, 36:5376855533333 -37411, 2023.
[32] Harish Ravichandar, Athanasios S Polydoros, Sonia Chemova, and Aude Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3:297-330, 2020.
[33] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprini arXiv:1910.01700 7(1):2, 2019.
[34] Siddharth Reddy, Anca Dragan, and Sergey Levine. Shared autonomy via deep reinforcement learning. Robotics: Science and Systems, 2018.
[35] Stéphane Ross and Drew Bagnell, Efficient reductions for imitation learning. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668. JMLR Workshop and Conference Proceedings, 2010.
[36] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[37] Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions. UC Berkeley, 2017.
12

---

[38] H. Saeidi, Justin D. Opfermann, Michael Kam, Sudarshan Raghunathan, S. Leonard, and A. Krieger. A confidence-based shared control strategy for the smart tissue autonomous robot (star). In 2018 IELE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1268-1275, 2018.
[39] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner. Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. ArXiv preprint, abs/1902.04043 2019.
[40]
William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2067-20066 International Foundation for Autonomous Agents and Multiagent Systems, 2018.
[41] Esmacil Seraj, Kin Man Lec, Zulfiqar Zaidi, Qingyu Xiao, Zhaoxin Li, Arthur Nascimento, Sanne van Waveren, Pradyumna Tambwekar, Rohan Paleja, Devieena Das, et al. Interactive and explainable robot learning: A comprehensive review. Foundations and Trends Robotics, 12(2-3):75 349, 2024.
[42] Jonathan Spencer, Sanjiban Choudhury, Matthew Baress, Matthew Schmittle, Mung Chiang, Peter Ra- madge, and Siddhartha Srinivasa. Learning from interventions. In Robotics: Science and Systems (RSS), 2020.
[43] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in neural information proessing systems, 33:3008-3021, 2020.
[44] Garrett Warnell, Nicholas R. Waytowich, Vernon Lawherm and Peter Stone. Deep TAMER: interactive agent shaping in high-dimensional state spaces. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, Proceedings of the Sheroy-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1545-1554. AAAI Press, 2018.
[45] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401 08417, 2024.
[46] Zhenghai Xue, Zhenghao Peng, Quanyi Li, Zhihan Liu, and Bolei Zhou. Guarded policy optimization with imperfect online demonstrations. arXiv preprint arXiv:2303.01728 2023.
[47] Maryam Zare, Parham M Kebria, Abbas Khosravi, and Saeid Nahavandi. A survey of imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions on Cybernetics, 2024.
[48] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint 23X5v:2355.10225 2023.
[49] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu, robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12293 2020.
[50] Alex Zyner, Stewart Worrall, and Eduardo Nebot. A recurrent neural network solution for predicting driver intention at unsignalized intersections. IEEE Robotics and Automation Letters, 3(3):1559 1764, 2018.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our Method and Experiments sections accurately address the claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Conclusion section (Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
We provide full assumptions in Thm. 4.1 and the proof in the supplemental
material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide implementation details in Sec. 4.3 and our hyperparameters Sec.
5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is included in hattps://metadriieerss wit.ug.ttiiboo io/ppl and the experiment results can be faithfully reproduced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
Moolic//guides/Coeeeubbiis www.eeeeuuuuisssiooooooo.ooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
Stteeee//ooSSSSeissSioica ssissssliccyyy for more details. / wwtp:/cccppppgguuuaaaeoeaec
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: We provide the experimental settings, the training/test details, and the hyperpa- rameters in Sec. 5.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: According to Sec. 5.1 we train our method and all the baselines five times
using distinct random seeds. We also report the standard deviation in Table and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Sec. 5.2 we report the type of GPU and the time of execution needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. cccpppps.cccp/peeeee http://ppwpppp.pppoop//acacaaaaa
Answer: [Yes]
Justification:
This research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both positive and negative social impacts in the Ethics Statement (Sec. HH.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not include pretrained language models, image generators, or scraped datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have credited MetaDrive 16 and Robosuite of assets used in our work.
49 in our paper, the creators
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswipphaeeeooccaaaapp thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not include new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We provide the instructions to the human participants in the supplemental material.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We have obtained IRB approval to conduct this project, and we provide the ethics statement in the Appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our method does not involve LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httee://hberrrreccc///foffeerrre for what should or should not be described.
20

---

A
Algorithm
We summarize our method PPL Alg.
Algorithm Predictive Preference Learning from Human Interventions (PPL)
1: Input: Hyperparameters H, B.
2: for timestep 0, H, 2H. ...... do
3: Agent samples action O₂ nn (8k)-
4: Predict future trajectory = [S8,, an H) = (8k. 8k-1, H
5:
Human observes
decide whether to take over in the next H steps.
6:
for timestep = k, k + 1.-.. , H do
7:
if Human takes
then
8:
Human takes action TT (st).
9:
Add (st, ah) to the human butter D/- Agent samples action an (st).) Predict future trajectory = f(s On L) (st, $c-1.-.. S.+L).
10:
12:
Add (3, 0n, an) to the preference dataset prer for each s in (st. 8141,' ()LL)
Observe 8t-1 ~ P(- St, an).
14:
else
Agent samples action On TT,(ss) Observe St-11 P(- St, an).
16:
17: end if
18:
Train policy with loss function Eq.
19: end for
20:
end for
21:
Output: Policy Tn.
B Additional Experimental Results
We report the performance of our PPL and all the baselines with neural experts as proxy human poli- cies MetaDrive (Table Table Wiping (Table D. and Nut Assembly (Table tasks, respectively. The test success rate curves of all three tasks shown in Fig.
Table Comparison of methods with training/testing statistics in the MetaDrive environment with the neural expert as the proxy human policy. The overall intervention rate is given together with the expert data usage.
Expert-in-the-Loog
Training
Testing
Expert Data Usuge
Total Data Usuge
Success Rate
Episodic Return
Route Completion
Neural Expert GAIL
0.83 ao
340.2
0.93
0.46 =
x
20K
IM
0.34 200
196.5 u
0.60 6:30
Emsemble-DAwe DAgger
/
(0.32)
1
Thrifty-DAgger
2.9K (0.29)
10K
0.49 an0
248, 2
0.75 2m
PVP
7
2.5K (0.25)
258.1
0.76
2.7K (0.27)
217.0:
0.67 =:26
EIL
3.9K (0.39)
10K
0.11 aus
131.8 235
0.42 a
PPL (Ours)
1.2K (0.20)
6K
0.80 cas
329,9 1M
0.92 6x)
Table 4: Results of different approaches in Table Wiping.
Table 5: Results of different approaches in Nut Assembly.
Method
Expert Data Usage
Total Data
Success Rate
Mothed
Expert Usage
Total Data
Success Rase
Neural Expert
0.84
Neural Exper
0.60
BC
IWR
2.5K
4K
0.51
IWR EII.
S4K
2008
0.29
EIL.
2.4K
4X
0.53
48K
2008
1.25
HACO
(Olr()
(Ours)
21

---

The neural experts in Table 3 and 5 are trained with PPO- -Pg-Lagran [3] for 20M environment steps, yet their test success rates still fall short of 100% for the following reasons. The MetaDrive safety environments occasionally generate rare but challenging scenarios that even a well-trained policy may fail to handle. the Table Wiping task, the neural expert sometimes fails to remove one or two markings on the whiteboard, leaving small patch of dirt uncleaned. In the Nut Assembly task, successful grasping requires the gripper to be precisely aligned with the metal ring's handle, which is highly sensitive to even minor action errors.
C Demo Video
We have attached our demo video to how we conduct human experiments and the evaluation results of our method Predictive Preference Learning from Human Interventions (PPL). This video includes five sections:
github. /ooppp This video shows
1. The first section gives an overview of Predictive Preference Learning, showing what human observes on the screen and how human provides corrective demonstrations in an episode.
2. The second section is the footage of the MetaDrive human experiment, where the human expert interacts with the driving agent via a gamepad.
3. The third section shows the evaluation results of the PPL. agent in held-out MetaDrive test environment. We compare our approach PPL with the PVP baseline [28]. and both agents are trained 10K timesteps. The evaluation results show that our approach PPL has a higher test success rate and lower safety cost.
4. The fourth section shows the applicability of our methods to manipulation tasks in Ro- bosuite 49]: Table Wiping and Nut Assembly. PPL successfully imitates the expert and accomplishes both tasks in evaluation environments.
5. In the fifth section, we provide full training session on MetaDrive. The video is played 5x speed, and it shows how a human expert trains PPL agent on MetaDrive in under 12 minutes, with approximately 1.8K demonstration steps and 10K environment steps.
D Human Subject Research Protocol
Human Participants. Five university students (ages 20-30) with valid driver's licenses and video gaming experience took part in the study voluntarily. After receiving detailed overview of the procedures and providing written informed consent under an IRB-approved protocol, each participant completed a hands-on familiarization session. During this session, they were informed how the predicted trajectories were shown on screen, and they practiced with our control interface and learning environments until performing consecutive successful runs before the main experiments.
Main Experiment. Each participant began with one or two fully manual episodes to build con- fidence, and then ceded control the agent when they felt safe. Participants were instructed to intervene only when the agent's predicted trajectory appeared unsafe, illegal, or inconsistent with their desired actions. They were directed to prioritize safe task completion and then to guide the agent toward their personal driving or manipulation preferences.
E
Notations
recall all the notations in this work. We denote the human policy as Fy and the novice policy as nne For any stochastic policy <aa s) and the initial state distribution do state space S, we define the value function JTT) the expected cumulative
Before prove Theorem 4.1
7 at),, wherein 7 = (8o, 4o, 81, 01, (1.) is the trajectory sampled from T~Pt t=0 trajectory distribution P= induced by n. dg and the state transition distribution P. We also denote
return: J(T) E IΣ
the Q-function of policy a as Q(s, a) = E 1> T 1=0l
ar) "r
8,00 a). And we define the
discounted state distribution under π as (s) = (1 - 7)
E
12 t=
>1 = s]]
22

---

In our algorithm PPL, we have a preference dataset D containing preference pairs ,a").
The preference loss function of policy и in PPL defined as
ppeet (T) prer
Σ
| log (Blog ( s) log (a |s)]],
(7)
(el,ataa)EED
where B is positive constant and ((x) = (1 + (xxp(-x))-1 the Sigmoid function.
We denote the state distribution in D pref
dprer( (8) pret)
Σ
T s].
(8)
(0' a pe
In addition, for any state s in D Preef with dieo((s) > O, we denote the empirical preference-pair distribution in state as
Σ
Tis' = 8, a Oh, a = an]
(stattia Saatta--eee PeP
Pprer an; an)
(9)
I[s' = s)
(s/aa EEDmd
which is distribution on A x A.
F
Proof of Theorem
4.1
between the human policy ид and the agent policy T. can be bounded by the following three error terms: the state distribution shift daist the quality of preference labels ōpref, and the optimization error c. We denote the total variation for any two distributions P. Q on the same space as Drv Q) Q111-
Our goal to prove that the performance gap J(TA) - (m,,
Here, we formally define the three error terms. The first state distribution shift error arises from the difference between the distribution of states in the preference dataset Prer (denoted as dste(()) ))) and the discounted state distribution of the agent's 's policy T (denoted as d,,,, (s)). ) To define the distribution shift error ddist in PPL, use the total variation between the distributions, i.e.,
ddist (DTv(d,,,) dpret).
(10)
The second error term arises from the misalignment of the positive actions in the preference dataset, as the human action aj in each tuple (š,, an a,,) E Dpeet is sampled in state instead of the predicted future state 8, In ideal preference dataset, one would observe expert and novice actions drawn directly at Sy. To quantify this error, we define the following distribution Piacal (ah, an) = Ty (a,, S)TT (an, | 8) A x A for any sampled at directly at state 8. Then we use
i.e., the distribution over pairs (an, Og) if both policies
Opeel
E
Dre(Pidcat Porct)
(11)
to define the errors in the preference dataset.
Finally, we define the optimization error of the agent policy n,
c c= C [T))
(12)
Under these notations, have the following Thm. We note that when we choose small 3 < M- (M is defined in Thm. F.1 we have
(V
4 log 2 6prel
(TT) (TT))
26
(13)
2
Theorem F.1 (Formal Statement of Theorem 4.1
We denote the Q-function of the human policy
Th as Q" (s, a). We assume that for any (8, a, a'), log Ty (a'all |s] < M, and log (als)) log nn (a'ls)) I M. where U, M are positive constants.
-",," S U, loo m (als))
Then, we have
J(TA) ((TT) < 1-9 (1
U
E + 4(3M + log 2) 8pref
3M2
+
+ 20dint
(14)
2,3
8
23

---

Proof. The key is to combine Lem.
Lem. and Lem. F.4 to obtain the bound.
From Lem. F.2 can the state distribution shift and the total variation of the policies The Tu on dprof to bound the optimality gap:
J(Th) J(Tm) -
U
(.E E (Tn(s(())) + 2ddist) sandpret
(15)
In addition, for any policy π,, we define the function
g
E
E
|- logo ( (glog x(a s) - B log Taa" |
(16)
snddpee at my (s).aT~,,,.)) (s)
which represents the preference loss on ideal preference pairs, where at a are sampled directly at each state s.
Using F.4 we can bound the total variation term
DTV(TT( by g(T,)) g(Ta):
enddpret
g(Tn) g(T))
3M
E srudgrat
Drv (T)) 1
(17)
28
8
In addition, by Lem. the misalignment of preference levels opref:
F.3
we can also bound g( (TTh) by the optimization error on ppet and
g( g(Th)) S 4(3M log 2) Spref.
(18)
Combining Eq.
15 17 and 18 yields Eq. 14
Lemma F22 (Performance Optimality Gap the State Distribution Shift). We recall that dpror(s)
(Dmet
I[s' s). and we define U
max
12" a1) (s,
(st.at.a JoDD Then, for any stochastic policies The The we have
LES,an,aEE.
J(mm) [ 5 1 (.E
U
Dry (Tv())(1)))= (-1))) + 25
(19)
where daist = Drv(dr, d.root.
Proof Sketch. The key is to use the Performance Difference Lemma (Lem. G.2) on ((T,) (mT). yielding Eq.
Then, we can apply Lem. G. which bounds the expectation on dprct and da the distribution shift term ddist- Finally, applying the assumption U
max 1Q" (s,())))) - (*, a2) bounds the difference of the Q-function by the total variation
sES,01,02EA
between and TT.
Proof. By the Performance Difference Lemma (Lem. G.2
we have
((Th) (TTT)
E
E
(s, an) Q" an)l
(20)
soudy, (s),aa mm (s)
By Lem. G.I as dxn, and dpref are two distributions on the same state space S, we have - (J]m() - (TT))
E snudgeed unr (s
QQ" an) - Q" (s,
+ 2DD dprot) SES
max
E (e), (8)
(Q*(,, Q" an))
E
E
Q* (n) - (x,,
(21)
anudgree on (s)
+ 20 max S (N
E
IQ" (n)) Q*(ssan)))
Q'(s,an)=
(*()
+ 2U -
Sowdpeet (s)
(n)

---

where we use U
max EE
(Q*(s, Q" (s. a2)l in the last inequality of Eq. 21
In addition, ny (s) and (s)
probability distributions on the same action space A. By Lem.
G. we have for any 8 SE E.
E ag (+)
(s)
E (s)
an) S.
(22)
This proves that
U
( Drv (Ta(1)))) + 25dist)
J(mh) - J((n)) S 1 serdppret
(23)
Lemma F.3 (Misalignment of Preference Pairs). We recall that the loss function of the policy и is [log (3 log xaa (s) log m(a |8)]] And the optimization loss is
(T (a.a1,0 ja-Daed
E
defined as e f (pree((TT) In addition, following Eq. 16 for any policy n, we define
g(x)=
srudpret a (an a (m) [- log ($log x(a 8) - Blgg T(a |
E
(24)
Under the assumption that for any (s, a, a'). log log TA (a'ls)] s M, and log n. (a|s) log [a){s) S. M, we have
we can bound
g( - g(TN) c 4(3 + log 2) Spref, Drv(Pideas Poret).
(25)
where Opref
Proof Sketch. The key is apply Lem. G. on the two distributions Piacaa and Pfreet that bound the difference of (T) and g for any policy by (6ppeet).
Proof. For any E S, we denote Pideaa (an, an) Tn(ap S)TT s), probability distribution on Ax A. We also denote (ap an) Pprer(s: abs an) dpree (s) for any such that dpret (s) >
where we recall that Ppret ah; an) pretl (s',at, )aDDpe
E I[ss' s, a aa,a =a,]
The key is that Pideal and Pfret are two distributions on the same space A x A, and we can apply Lem. G.T on Eq. 24 obtain the proof.
We denote 1 a=ss ,a log (3 log (aa s) 8log s)).
(1T at ), and 4max max (ITA, (in)). Then, for any policy m. We also denote j max s.a aa
o(TT)
E ( (s)
("ssattaa"
srudpee at Nes (s),a t
1 (ss gnudgeed (ot Prupidaaa
E
21
Div Ppres)
+
Fss,at,aa")
grudgrrd
(at (ata-)>pp
(26)
21 Spref
E snudpret (al Drppoct
('(s,at,a7)
21 Spref + |Dprer|
Σ
(3ssattaa
(s,at, JEDE
21 + CTe
Similarly, we can obtain that g(T) 2 - 21 Spref + Cpee((T) for any policy т. Thus we have
((TT) g(Tx) I 44max: + (Epret(TT) (Tn) - (p()((TT)) 46max +
(27)
25

---

Finally, under the condition that [aog)(a)s)) log (a'|s) Σ M for any (s, we have It" (s, a, a')) f log (--0()M))))))) ) log(1 + exp( - 5M + log2.
This implies that mmxx I 3M + log 2 and completes the proof.
Lemma E.4 (Optimization Error Bounds the Total Variation). We assume that for any (s, a, d'), llog Th (als)) log n (aaaaa(s) S M, and log Tn(aas) - log Tu (a'ls)) S M.
We recall that g
E
E
[- log (3 (aa 188 Bla |))]]
spedpret at (t)
which is defined in Eq.
16
Then we have
g(Tm) g(Th))
8M2
E seodpret
Drv (Ta()))))) S
+
(28)
2B
8
Proof Sketch. First, we define
3 f(T) 2 soudpeet at mo ( (s).a-m,,kl
logm(a+ls) - -log
x(a [s) (s)]ss)] + log
(29)
Using the Taylor's expansion on the function logo(x) at 0. when the policy satisfies log T(als) - log (a')=aalss) |s}] S M for any (s, a, a'), we can obtain that lg(x) f()) - a²M²
8
In addition, f (mT) f(TA)) bounds the KL divergence of the two policies Ta and n. So
dpref-
Pinsker's 's inequality to obtain the bound on Drv Drv(Ta (-1)s)), (-(ss))).
Proof. For any as aa), denote un as ,a") log (a log n. (a [s). and [u,(s,
un a = log Ta (a+
+ s) log nA [s). From the assumptions, we can obtain that
M and (s, a M.
By definition of the function g(), [log (B up (8, at aa )) - log a (B (s, at,,tte a ))]
we have g(Tn)
g(TT)
E
srudpret at ao
(s). (6).0 (s)
The Taylor's expansion of log or at a 0 ensures that for any x E IR, we have
1 x. S 8
logo(x) + log
(30)
2²)
This ensures that
g(Ta)
E suitpeet at atwan(s))
E
| log (B un(s, ,a-))
un(s,at,ai ,a 8822
(31)
> log 2
snddgred Not (s),
and similarly,
g(TN)
E
E
log (B-unts,at,e
sridpeed at (s),
un , B2M2
(32)
< log 2
E enddare ~ (_)
Hence, we have
8 snddprat at (s), ( (s)
B2
((TT)) ((TT))
Ju (u aaa un _a]]
log Th (a+ss)
n (a (8)
M2
E
log
(33)
2 snudpree NTT (+).a (+)
(a
nn (a [s)
4
(a(at(s)
(a 18)
B2M
log
log
g srvdpref
(2)
T Ta s)
(.)
Th (a [s)
4
26

---

By the definition of KL divergence, we have
g(TT)) -((TA)=
8
E
KL (-1ss)
mn((1ss)) + KL Tn (T((-ss) (mn((1s))] [m)(()))[
32a12
2
grudpran
4
(34)
(Pn((an(1ss) (11)) B2M
23 sevd
E
T. (-1s)) from the KL where use Pinsker's inequality to obtain the bound Drv (TA (TV( (-)), divergence. Finally, we apply the inequality E[x² 2 (E[X))² on X = Drv (TT (-ss), (-(+)))) so that we have
8²M²
((mm) - ((TA) 23
(35)
soddyee
This proves that
(Ta())(l)s),m.. 1 g( g(TA))
3M2
E srodgrot
2, +
(36)
8
G
Technical Lemmas
Lemma G.1 (Expectation Difference via Total Variation). Let P and Q two probability distri- butions a measurable space X, and let f: X - R be any measurable function satisfying the uniform bound [(x)] S M for any x E X. Then
f(x)
f(x) S 2M Dry (rv(P,Q)) P,,),
(37)
P-)(())
z~Q()
where Drv 1PP Ql1 is the total variation distance. In addition, when the measurable function g satisfies the bound (g(z1) g(x2)1 S M' for any 31,32 X, have
g(x) xNP())
E
E s(x))
S M' -Iv(P))...
(38)
2~Q()
Proof. When [f(x) M for any x, have
f(x)
(x)
E((r) (P(s)-((=)) Q(x))
P())
m)Q(
Σ [f(z)) (P(x) Q(x)
(39)
$ M IP(()) - Q())
2M DIv(P,Q).
When 1g(2;) - 5(x2) M' for any æ1, 82 E X, we set ((x) = g(x)) f.c, where c sup f(x)
xe
inf f(x). As we have [f()) < M' /2 for any x E X, we have
TEX
E g(x)) g(x))
E xx>(())
f(x)
E
f(x) S M' Drv(P,Q).
(40)
px))()
x))()
z~Q())
27

---

Lemma G.2 (Performance Gap Between Human Policy and Novice Policy), We denote the Q-function of human policy a (s,
TruP Σ 7 7/1((1, at) 8, 00 a).
For the human policy The and the novice poliey Ty whose value functions are J respec- tively. we have
J(T)) - (Tm)
E
E Q"(s,
E ancvan(e)
(" an)
(41)
seudan
(ah-vaa(t)
Proof. We denote the Q-function of novice policy as Qu(s,a) = E TNPPA
a)) =
1=0
dg al.
We denote value functions of Th, Tn
(s)
E Q' a) and Vi(()
E
Qn(8,a).
acvia(s)
respectively. And we have J(TA)
E V* (sa), and [(T)
E (sn(s).
#grado
sorda
We define the advantage function of mh as A (((ss)) a) (*( (8).
By the performance difference lemma (Lemma 6.1, 114
have
E
(Vo(8) - V*(o))
E
E an)aa()
A'(s,a)
(42)
spridy
sndzn
This implies that
1
V - (80)
((*)aa)-VV(())
sg
1 ? gudeen anvit(s))
1
[-V* V-V*(s)+
+ E
(43)
1
? gruden
a
E (*(s,a)+ E ('(s
1
gndeen
amt
Multiplying -1 on both sides, can obtain that
1
((Tn) (TTm)
E
E Q"(s,an)=
E an>
('(s,
(44)
7 seudin (akNNT((s)
H
Ethics Statement
Our Predictive Preference Learning from Human Interventions (PPL) delivers a human-friendhh numan-in-the--e training process that increases automation while minimizing expert effort, ad- vancing more intelligent Al systems with reduced human burden. All the experiments are conducted entirely simulation, ensuring no physical risk to participants. All volunteers provided informed consent, were compensated above local market rates, and could pause or withdraw from the study at any time without penalty. Individual sessions lasted less than one hour, with a mandatory rest period of at least three hours before any subsequent participation. No personal or sensitive data was collected or shared. We have obtained the IRB approval to conduct this project.
While PPL promises positive social impact by streamlining human-Al collaboration, it may also encourage overreliance on automated systems inherit biases present in expert involvement.
28


Reviews:
[{"review_id": "mydJJ2KjSP", "reviewer": "Reviewer_6PmN", "summary": "The paper proposes  a novel approach to interactive imitation learning (IIL) that enhances learning efficiency by leveraging human interventions more effectively. Traditional methods often require human experts to provide corrective demonstrations reactively, which can be inefficient and burdensome. Instead, the proposed method, Predictive Preference Learning (PPL) uses trajectory prediction models to anticipate future agent actions, allowing experts to intervene proactively. By visualizing predicted trajectories, human supervisors can identify potential failures before they occur, converting these interventions into contrastive preference labels that guide the agent's learning process.\n\nThe proposed method significantly reduces the number of interventions needed and decreases the cognitive load on human experts. PPL is evaluated on simulation-based tasks like MetaDrive and Robosuite, demonstrating improved efficiency and generalization compared to existing methods. The paper also includes a theoretical analysis that balances the preference horizon to optimize learning outcomes. Overall, the proposed method offers a promising framework for integrating human preferences into agent training, potentially streamlining the development of more intelligent and adaptable autonomous systems.", "questions": "1. The author claims PPL to be a novel IIL method, can author properly defend this claim with substantial proof? The claim written in lines 57-59 doesn't seems sufficient.\n2. Can you provide guidance on selecting the preference horizon L? How sensitive is PPL's performance to this parameter?\n3. Since LLM/VLM are so popular, did the authors tried to replace/augment human with a LLM? If not, can they discuss how their method might be used with LLM?\n4. Since the paper talks about robotic tasks application for the proposed PPL method. However, both experiments MetaDrive and Robosuite shown in the paper are simulation-based tasks. Can author perform real-world robotic task to evaluate the proposed method better? Or at least they can discuss how the proposed method would work in real world and what challenges do you anticipate in transitioning from simulation to real robotic tasks?\n5. How does the PPL scale with larger and more complex state spaces, and are there any optimizations planned to improve scalability?\n6. What are the computational costs associated with implementing PPL, and how does it compare to other IIL methods in terms of efficiency?", "limitations": "Yes the limitations to the proposed method has been listed in the Conclusion section. However, in my opinion the authors should discuss the limitations to PPL in detail in Appendix.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed method effectively integrates trajectory prediction to inform human interventions, allowing for proactive error correction and reducing the need for constant human oversight.\n- With converting interventions into preference labels over predicted trajectories, the proposed approach reduces the cognitive load on human experts and the number of necessary interventions.\n- The proposed method is tested on multiple simulation-based benchmarks like MetaDrive and Robosuite, showing improved efficiency and generalization over traditional methods, which supports its practicality and robustness.\n- The paper provides a theoretical analysis that explains the balance needed in setting the preference horizon, offering insights into the method's optimization.\n- The attached demo video helps to visualize the results better and proposed method effectiveness.\n\nWeakness:\n- Since the paper talks about robotic tasks application for the proposed PPL method. However, both experiments MetaDrive and Robosuite shown in the paper are simulation-based tasks. It makes hard for a reviewer to evaluate the paper based on simulation based tasks only. I would encourage the authors to perform some real-world experiments.\n- The proposed method assumes that human interventions are always optimal, which might not hold true in practice because of human variability.\n- Some key concepts, like \"preference horizon\" and \"contrastive preference labels,\" could be better explained or illustrated via pictures, enhancing the paper's accessibility.\n- The integration of trajectory prediction could introduce additional computational complexity, potentially affecting scalability of the proposed algorithm.\n- The paper is not well written and has few unclear statements. Ex. Second statement of the abstract (lines 2-3) is unclear.; There is a typo on line 30 \"One line of confidence\" -> \"One line of research is confidence\"; In line 61 what does \"neural experts\" means? Is it a typo??", "was_revised": true, "final_justification": "I am satisfied with the discussions during the rebuttal phase and would like to keep my score.", "rebuttal": "Thank you for your thorough review and insightful comments. Below, we address each of your questions in detail.\n\n__Other Strengths And Weaknesses:__\n\n>1. I would encourage the authors to perform some real-world experiments.\n\nWe appreciate the reviewer’s suggestion regarding real-world experimentation. However, we would like to emphasize that this work focuses on the algorithmic design and theoretical foundations of a new human-in-the-loop machine learning method. As a NeurIPS submission, our primary contribution lies in advancing the core machine learning methodology, which we rigorously validate in simulation environments with real human feedback. We agree that deploying the algorithm in physical systems is an important direction, and we plan to pursue real-world robotic experiments as part of future work.\n\n>2. PPL assumes that human interventions are always optimal.\n\n__We don’t assume human interventions are always optimal.__ We perform the following experiment in MetaDrive to show that PPL is robust to noise and perturbation in the expert. Following Table 3, we use a well-trained PPO expert as a proxy human policy, which achieves a 0.83 success rate. Then, we add Gaussian noises $\\mathcal{N}(0, \\sigma^2 I)$ to the expert actions, and we evaluate how $\\sigma$ affects the test success rate of PPL in MetaDrive with 10K total training steps. The result shows that PPL is robust to a noise $\\sigma \\leq 0.2$.\n| Noise $\\sigma$ in expert policy | Test Success Rate of PPL | \n|--------|--------|\n| 0  | 0.80  |\n| 0.1  | 0.77  |\n| 0.2  | 0.70  |\n| 0.3  | 0.62  |\n| 0.4  | 0.43  |\n\n>3. \"preference horizon\" and \"contrastive preference labels\" could be better explained or illustrated via pictures\n\nWe have provided the detailed description in Section 1 in our demo in the supplementary material, and we will better explain them and update the pictures in the revised manuscript.\n\nConcretely, the __preference horizon__ L denotes the number of future timesteps over which we solicit human preferences on a predicted trajectory. When an expert intervenes at state $s$ and uses human action $a_h$ to replace the agent action $a_n$, we know $a_h$ is more likely to avert a dangerous outcome not only at $s$ but also at each predicted state $\\tilde{s}_1, \\cdots, \\tilde{s}_L$. Therefore, for each $\\tilde{s}_i$ in the horizon, we assign a __contrastive preference label__ $a_h \\succ a_n$, indicating that the expert's action is preferred over the agent's.\n\n>4. The integration of trajectory prediction could introduce additional computational complexity\n\nThe additional computational cost is minimal compared to the human intervention cost. In our setting in Table 3, we use a very simple trajectory prediction method that can run up to 1000fps on a CPU. Using other off-the-shelf learning-based trajectory prediction models can easily run up to 100fps.\n\nIn addition, Table 3 demonstrates that PPL yields a higher success rate (PPL 0.80 vs. 0.56 for the best baseline PVP) and reduces expert takeovers (PPL 1.2 K vs. PVP 2.5 K), so the additional computation cost in fact __saves the human cognitive effort and training time__.\n\nApart from the rollout-based trajectory predictor, we also developed a faster trajectory extrapolation method by simulating a kinematic bicycle model via forward integration (Lines 320–323), which runs at 3000 fps. As reported in Table 2, this extrapolation approach achieves a 0.78 test success rate in MetaDrive, nearly matching the 0.80 achieved by the rollout predictor. This demonstrates that the choice of predictor has minimal effect on PPL’s performance.\n\n>5. (a) L2-L3 is unclear: “While learning from corrective demonstrations addresses present mistakes, it fails to avert errors in the agent’s subsequent trajectory.“ \\\n(b) There is a typo on line 30 \"One line of confidence-based IIL works\" -> \"One line of research is confidence-based IIL works\".  \\\n(c) What does \"neural experts\" mean in L61: \"We use both neural experts and real human participants\"? Is it a typo?\n\n(a) We rewrite L2-L3 as follows: Although prior IIL methods correct the agent’s action at the current state, they do not adjust its actions in the future states, which are potentially more hazardous.\n\n(b) L30 \"One line of confidence\" is a typo, and we will update it in the revised manuscript.\n\n(c) L61 “neural experts” is not a typo. According to lines 244-246, we also use well-trained PPO policies to approximate human policies in MetaDrive and Robosuite environments, and we report the results in Tables 3,4,5.\n\n__Questions:__\n\n>1. \"PPL is a novel IIL method\", can the author properly defend this claim with substantial proof? \n\nThe core innovation of PPL is the joint integration of trajectory prediction and preference learning into the IIL loop, while no existing IIL baselines (e.g., PVP, IWR, Thrifty‑DAgger) leverage predicted future trajectories to guide learning. Moreover, unlike traditional preference‑based RL methods such as RLHF and DPO that train on static, offline human‑annotated datasets, PPL continuously learns human preferences in an online setting from real‑time expert interventions.\n\n>2. Can you provide guidance on selecting the preference horizon L? How sensitive is PPL's performance to this parameter?\n\nOur experiments suggest choosing L up to the number of steps that an agent executes in one to two seconds. In addition, __PPL’s performance rises and then falls as L increases__, which makes it easy to pinpoint the optimal L. As shown in Fig. 6, for MetaDrive, the success rate climbs from 0.48 at L = 0 to 0.80 at L = 4, then drops to 0.50 by L = 8.\n\n__PPL is not very sensitive to L.__ We find that __PPL with any L between 2 and 6 outperforms all baselines__ in MetaDrive. The following table reports the test success rate of PPL with different values of L with 10K and 15K total training data usage.\n| |PPL (L = 2) | PPL (L = 3) | PPL (L = 4) |  PPL (L = 5) | PPL (L = 6) | PVP (best baseline) |\n|---|---|---|---|---|---|---|\n|Test Success Rate (trained for 10K steps)| 0.56 | 0.75 | 0.80 | 0.77 | 0.64 | 0.56 |\n|Test Success Rate (trained for 15K steps)| 0.70 | 0.79 | 0.81 | 0.78 | 0.75 | 0.68 |\n\n>3. Did the authors try to replace/augment humans with an LLM? Discuss how their method might be used with LLM.\n\nAlthough we have not yet tested PPL with an LLM/VLM as the expert, our PPL framework supports such integration. To incorporate the VLM as an expert in PPL's setting, we can feed the agent's predicted future trajectory (rendered as front‑view images) to the VLM and let it decide whether to intervene. Upon takeover, the model would choose its preferred action from a small set of predefined options, and the low-level policy will be trained to follow the VLM's preferences. In future work, fine-tuning the VLM on intervention data may further improve its guidance.\n\n>4. How would the proposed method work in the real world, and what challenges do you anticipate in transitioning from simulation to real robotic tasks?\n\nWe anticipate two main challenges: accurate trajectory predictors are harder to obtain, and human takeovers are more expensive. Additionally, distribution shifts in novel environments may degrade predictor performance. We plan to tackle the real-world deployment issues in our future work. \n\n>5. How does the PPL scale with larger and more complex state spaces, and are there any optimizations planned to improve scalability?\n\nWe added experiments in MetaDrive to demonstrate that __PPL can handle a more complex RGB state space__. We further evaluate PPL in MetaDrive using 320×320 images (three‑frame stacks) as input and a three‑layer CNN encoder within the policy. Even under this high‑dimensional setting, PPL continues to outperform all baselines with 10K training steps:\n| Method | Test Success Rate | \n|--------|--------|\n| PPL (Ours)  | 0.75  |\n| PVP | 0.46  |\n| IWR  | 0.23 |\n| EIL    | 0.08 |\n| HACO  | 0.31 |\n\nFor tasks with even more demanding observation spaces, one can employ pretrained encoders to extract compact feature embeddings and reduce computation.\n\n>6. What are the computational costs associated with implementing PPL compared to other IIL methods?\n\nIn our main experiments (Tables 1 and 3–5), PPL’s computation runs at ~1,000 fps, comparable to the baseline methods (Thrifty‑DAgger, PVP, IWR, etc.). Moreover, Table 3 demonstrates that PPL yields a higher success rate (PPL 0.80 vs. PVP 0.56) and reduces expert takeovers (PPL 1.2 K vs. PVP 2.5 K), demonstrating that PPL actually lowers human cognitive load and shortens total training time."}, {"review_id": "wUtzTDHQDI", "reviewer": "Reviewer_5Q4m", "summary": "The technique exploits a trajectory prediction model to forecast the agent's future states. This helps a human supervisor to determine when an intervention is necessary. The proposed approach then converts this intervention into contrastive preference labels over each \npredicted state. The resulting dataset provides more information than the corrective demonstrations used by previous approaches. \n\nResults are presented for three tasks (MetaDrive, Table Wiping and Nut Assembly) and comparisons are made to imitation learning baselines (BC, GAIL), confidence-based IIL methods (Ensemble-DAgger and Thifty-Dagger) and human-in-the-loop methods (WR, HACO, EIL, and proxy value propagation). Significant improvements are reported in terms of success rates and reductions in human data usage.", "questions": "* Could you justify your selection of the three tasks you use? why only these three?\n\n* minor: section 2. \"These methods do not leverage data collected...\" - I think this sentence could be reworded for clarity / precision. I think you are trying to say that while they can copy human behaviour they won't suppress undesired actions more generally where humans would anticipate a problem.  Perhaps expand slightly?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "strengths: \n* a clearly written paper and interesting contribution\n* significant improvements over previous approaches both in terms of success rates and human data usage", "was_revised": true, "final_justification": "I remain positive about this contribution and will retain my rating of 5.", "rebuttal": "We are grateful for your detailed review and helpful insights. Please find our summarized responses to each of your comments below.\n\n__Questions:__\n\n> 1. Could you justify your selection of the three tasks you use? Why only these three?\n\nAutonomous driving and robotic manipulation are representative real-time tasks where humans can play an essential role in demonstrating and improving the learning process of the AI agents. We selected MetaDrive, Nut Assembly, and Table Wiping to cover diverse challenges: MetaDrive provides a high‑dimensional (259‑D) state space, while Nut Assembly and Table Wiping in RoboSuite feature higher action dimensionality. Nut Assembly, in particular, is among RoboSuite’s hardest tasks due to its extreme sensitivity to small control errors. Even a minor gripper misalignment can prevent the item from being grasped or cause it to drop. In Sections 3 and 4 of the supplementary demos, we show that PPL consistently delivers strong performance across all three.\n\nMoreover, we added additional experiments to demonstrate that __PPL can handle a more complex RGB state space__. We further evaluate PPL in MetaDrive using 320×320 images (three‑frame stacks) as input and a three‑layer CNN encoder within the policy. Even under this high‑dimensional setting, PPL continues to outperform all baselines with 10K training steps:\n| Method | Test Success Rate | \n|--------|--------|\n| PPL (Ours)  | 0.75  |\n| PVP | 0.46  |\n| IWR  | 0.23 |\n| EIL    | 0.08 |\n| HACO  | 0.31 |\n\n> 2. L70-72 “These methods do not leverage data collected by agents or suppress undesired actions likely intervened by humans.” \\\n I think this sentence could be reworded for clarity. \\\n I think you mean that while they can copy human behaviour, they won't suppress undesired actions more generally where humans would anticipate a problem.\n\nWe appreciate your suggestions, and your understanding is correct. We rewrite L70-72 as follows:\n\n1. These IIL baselines (IWR, Thrifty-DAgger, Ensemble-DAgger, HG-DAgger) discard all agent‑collected transitions where no human intervention occurred, wasting valuable self‑exploration data.\n\n2. Moreover, although prior IIL methods correct the agent’s action at the current state, they do not adjust its actions in the future states, which are potentially more hazardous. Therefore, some dangerous situations may repeatedly appear during training."}, {"review_id": "uLiOe63amN", "reviewer": "Reviewer_RuYs", "summary": "Predictive Preference Learning from Human Interventions (PPL) is an interactive imitation learning approach that utilizes predicted future trajectory and, upon expert override, converts the corrective action into contrastive preference labels over those predicted states. By propagating feedback across multiple future states rather than only the intervention point, PPL reduces the number of required expert interventions and demonstrations, boosting sample efficiency. Experimental results on MetaDrive and RoboSuite show that PPL outperforms baseline imitation learning methods. A theoretical analysis further demonstrates that choosing the optimal preference horizon L balances distribution shift against label quality.", "questions": "- **Choice of Horizon $L$:** How did you select the preference horizon $L$ for each benchmark? Is there an intuition or heuristic for choosing $L$ when applying PPL to new tasks?\n- **Predictive Model Details:** Which trajectory prediction model was employed for RoboSuite experiments, and how sensitive is PPL to its accuracy?\n- **Data‐Budget Fairness:** Why do different methods use varying amounts of expert data? Could you compare all methods using the same data budget?\n- **Intervention Timing:** For each baseline and PPL, at what points during training or rollout are expert interventions incorporated into the learning process?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- **Intuitive Intervention‐to‐Preference Conversion**: Converting human overrides into contrastive preference labels over predicted trajectories is both simple and conceptually clear.\n- **Cross‐Domain Performance**: Demonstrates superior sample efficiency and task performance across diverse environments, including AD and robotic manipulations.\n\n### Weaknesses\n- **Dependence on Accurate Predictive Models**: Requires a high‐fidelity trajectory predictor (e.g., a rule‐based physics model); applicability may be limited in domains without such models (e.g., precise robotic manipulation, reasoning in LLMs, etc.).\n- **Theoretical Bounds Lack Practical Guidance**: While the analysis shows that the error varies with the preference horizon $L$, it does not offer intuition or heuristics for selecting $L$ in new domains, nor does it provide a proof of advantage over other methods.\n- **Missing Comparisons to Key Interactive IL Work**: Omits citation and empirical comparison to recent human‐in‐the‐loop imitation‐learning methods [1], [2].\n\n**References**\\\n[1] Liu et al., Robot Learning on the Job: Human‐in‐the‐Loop Autonomy and Learning During Deployment, IJRR 2024\\\n[2] Liu et al., Multi‐Task Interactive Robot Fleet Learning with Visual World Models, CoRL 2024", "was_revised": true, "final_justification": "Additional experiments, including those with a noised transition model and tested for robustness to $L$, as well as comparisons with other baselines, effectively address my concerns.", "rebuttal": "We appreciate the time and effort you devoted to evaluating our work and offering valuable suggestions. We summarize and respond to each question as follows:\n\n__Strengths And Weaknesses:__\n\n>1. Dependence on Accurate Predictive Models: PPL requires a high‐fidelity trajectory predictor.\n\nFirst, we emphasize that __our trajectory predictor remains lightweight yet effective__. In our main experiment (Table 1), we use an L‑step simulator rollout that runs at up to 1,000 fps on a CPU. We also developed a faster trajectory extrapolation method by simulating a kinematic bicycle model via forward integration (See Lines 320–323), which achieves 3,000 fps. As reported in Table 2, this extrapolation approach attains a 0.78 test success rate in MetaDrive, nearly matching the 0.80 achieved by the rollout predictor. This demonstrates that in domains where a rollout-based predictor is not available, __our trajectory extrapolation method can serve as a replacement without impacting PPL's performance__.\n\nIn addition, we perform additional experiments to show that PPL is robust to noises in trajectory predictors. __With an imperfect predictive model, PPL still outperforms all the baselines.__\n\nWe inject random noise $e_{noise}$ to the outputs $s’$ of the trajectory predictor, and we set the norm $||e_{noise}||_2 = \\epsilon * ||s’||_2$. Then we set the constant $\\epsilon = 0, 0.1, 0.2, …$ to test PPL’s robustness to noises in trajectory predictors.\nWe use MetaDrive, Table Wiping, and Nut Assembly environments following the same setup in Tables 3,4,5 from our paper, respectively. We find that with a noisy predictive model, PPL still outperforms all the baselines in MetaDrive and Table Wiping when the noise $\\epsilon \\leq 0.2$. In Nut Assembly, PPL outperforms the baselines when $\\epsilon \\leq 0.1$.\n\nIn MetaDrive, we report the result of PPL with 10K total training data.\n\n|MetaDrive |PPL ($\\epsilon = 0$) | PPL ($\\epsilon = 0.1$) | PPL ($\\epsilon = 0.2$) |  PPL ($\\epsilon = 0.3$) | PVP (best baseline) |\n|---|---|---|---|---|---|\n|Test Success Rate| 0.80 | 0.73 | 0.64 | 0.51 | 0.56 |\n\nIn Table Wiping, we report the result of PPL and baselines with 2K total training data.\n\n|Table Wiping |PPL ($\\epsilon = 0$) | PPL ($\\epsilon = 0.1$) | PPL ($\\epsilon = 0.2$) |  PPL ($\\epsilon = 0.3$) | PVP (best baseline) |\n|---|---|---|---|---|---|\n|Test Success Rate| 0.80 | 0.76 | 0.70 | 0.56 | 0.58 |\n\nIn Nut Assembly, we report the result of PPL and baselines with 200K total training data.\n\n\n|Nut Assembly |PPL ($\\epsilon = 0$) | PPL ($\\epsilon = 0.05$) | PPL ($\\epsilon = 0.1$) |  PPL ($\\epsilon = 0.15$) | PVP (best baseline) |\n|---|---|---|---|---|---|\n|Test Success Rate| 0.51 | 0.47 | 0.41 | 0.30 | 0.35 |\n\n\n>2. Theoretical Bounds Lack Practical Guidance: Thm 4.1 does not offer intuition or heuristics for selecting L in new domains.\n\n__Our experimental procedure for selecting the optimal L directly reflects the balance implied by the theoretical bound__.\n\nWe first recall Thm 4.1: the performance gap of a PPL agent can be bounded by three terms: the state distribution shift $\\delta_\\text{dist}$, the quality of the preference dataset $\\delta_\\text{pref}$, and the training loss $\\epsilon$.\n\nFrom the theoretical bound, __we can expect that PPL will achieve its highest performance when $L$ is balanced (neither too small nor too large).__ We explain it as follows.\n\n- When $L=0$, the first distribution shift term $\\delta_{\\text{dist}}$ is large, but $\\delta_{\\text{dist}}$ will drop when $L$ increases.\n\nBy definition of $\\delta_\\text{dist}$ (lines 190–191), $\\delta_\\text{dist}$ measures the discrepancy between states actually visited by $\\pi_n$ and those in the preference dataset; as $L$ grows, the dataset incorporates more predicted future states that will be visited by $\\pi_n$, so $\\delta_\\text{dist}$ decreases.\n\n- When $L=0$, the second error term $\\delta_{\\text{pref}} = 0$, but $\\delta_{\\text{pref}}$ will increase as $L$ increases. \n\nBy definition (line 198), $\\delta_\\text{pref}$ measures the error between distributions of the action pairs $(a_h, a_n)$ on current states $s$ and the future states $s'$ in the dataset. As $L$ increases, the predicted states $s'$ drift away from the current states $s$, so $\\delta_\\text{pref}$ increases. \n\nTherefore, for a well‑trained PPL agent (i.e., training loss $\\epsilon$ is small), the optimal selection of $L$ should balance $\\delta_\\text{dist}$ and $\\delta_\\text{pref}$. \n\nExperimentally, in MetaDrive, we indeed observe __PPL’s performance rising and then falling as $L$ increases (See Fig. 6), matching our theoretical prediction__. We report it in the following table:\n|MetaDrive |PPL (L = 2) | PPL (L = 3) | PPL (L = 4) |  PPL (L = 5) | PPL (L = 6) | PVP (best baseline) |\n|---|---|---|---|---|---|---|\n|Test Success Rate (trained for 10K steps)| 0.56 | 0.75 | 0.80 | 0.77 | 0.64 | 0.56 |\n|Test Success Rate (trained for 15K steps)| 0.70 | 0.79 | 0.81 | 0.78 | 0.75 | 0.68 |\n\nIn conclusion, our method for choosing L in experiments mirrors the trade-off predicted by the theoretical bound.\n\n>3. Missing Comparisons to Key Interactive IL Work: \\\n>[1] Liu et al., Robot Learning on the Job: Human‐in‐the‐Loop Autonomy and Learning During Deployment, IJRR 2024 \\\n>[2] Liu et al., Multi‐Task Interactive Robot Fleet Learning with Visual World Models, CoRL 2024\n\nThank you for suggesting additional IIL baselines, and we will include them in the revised manuscript. We implement the method Sirius in [1] given the short rebuttal period, which separates the training data into four categories and uses a Weighted Behavioral Cloning with a high weight given to human interventions and zero weights to pre-intervention states. Following the same setup in Tables 3,4,5 from our paper, we find that __our method PPL outperforms Sirius [1]__ in MetaDrive, Table Wiping, and Nut Assembly tasks:\n\n| MetaDrive | Training: Total Data Usage | Testing: Success Rate | Testing: Route Completion |\n|---|---|---|---|\n| PPL (Ours) | 10K | 0.80 | 0.92 |\n| Sirius [1]     | 10K | 0.42 | 0.73 |\n\n| Table Wiping | Training: Total Data Usage | Testing: Success Rate |\n|---|---|---|\n| PPL (Ours) | 4K | 0.80 |\n| Sirius [1]     | 4K | 0.61 |\n\n| Nut Assembly | Training: Total Data Usage | Testing: Success Rate |\n|---|---|---|\n| PPL (Ours)  | 200K | 0.51 |\n| Sirius [1]     | 200K | 0.34 |\n\n__Questions:__\n\n>1. How did you select the preference horizon L？\n\nOur experiments suggest choosing L up to the number of steps an agent executes in one to two seconds. As shown in Fig. 6, PPL’s performance rises and then falls as L increases, which makes it easy to pinpoint the optimal L: for MetaDrive, the test success rate of PPL climbs from 0.48 at L = 0 to 0.80 at L = 4, then drops to 0.50 by L = 8.\n\n>2. Which trajectory prediction model was employed for RoboSuite experiments, and how sensitive is PPL to its accuracy?\n\nWe apply a rollout-based trajectory prediction model for RoboSuite experiments. In the previous question “Dependence on Accurate Predictive Models”, we have conducted experiments showing that __PPL is robust to a noisy prediction model__ up to $\\epsilon \\leq 0.2$ for Table Wiping and  $\\epsilon \\leq 0.1$ for Nut Assembly.\n\n>3. Why do different methods use varying amounts of expert data? \n\nThe “amount of expert data” is the number of human takeover steps, reflecting how often the expert judged the current state to be unsafe and chose to intervene. Under the same total training steps, methods that make more errors trigger more human takeovers and thus consume more expert data. As Table 1 shows, PPL requires the fewest human interventions (PPL 2.9K vs PVP 4.9K) because its trajectory‑informed policy commits the least errors during training. You can also refer to Sections 1 and 5 of our supplementary demo to see how a human expert decides to take over during training.\n\n> Could you compare all methods using the same data budget?\n\nYes. As suggested, we compare all methods in MetaDrive with the same human data budget in a new experiment as follows. It shows that __PPL has the best performance with the same expert data usage__.\n\n| Method | Training: Human Data Usage | Testing: Success Rate | Testing: Route Completion |\n|---|---|---|---|\n| PPL (Ours) | 5K | 0.79 | 0.93 |\n| PVP | 5K | 0.46 | 0.71 |\n| IWR | 5K | 0.20 | 0.59 |\n| EIL   | 5K | 0.01 | 0.27 |\n| HACO | 5K | 0.08 | 0.31 |\n\n>4. At what points during training or rollout are expert interventions incorporated into the learning process?\n\nFor PPL and all baselines, we interleave data collection and gradient update: after collecting each environment transition, we immediately apply one gradient update. This online update schedule lets the agent use its most recent policy when interacting with the expert, rather than waiting for a large batch of data to accumulate. All baselines follow this schedule to ensure a fair comparison."}, {"review_id": "d9I7MowDtt", "reviewer": "Reviewer_p5vD", "summary": "This paper proposes Predictive Preference Learning (PPL), an interactive imitation learning framework that improves learning efficiency by converting proactive human interventions into contrastive preference signals. Unlike traditional imitation learning, which reacts to past errors, PPL anticipates future agent behavior using a trajectory prediction model, allowing humans to intervene before failures occur. These interventions generate preference labels over predicted future states, enabling the agent to learn not only from direct demonstrations but also from ranked alternatives. The learning objective combines behavioral cloning with a contrastive preference optimization loss, encouraging the agent to prefer safer trajectories.", "questions": "Theoretical analysis in Theorem 4.1 states that $\\beta$ must be “sufficiently small” to bound the performance gap, specifically assuming $\\beta \\leq M^{-2}$. In practice, would it be beneficial to conduct additional experiments varying both $\\beta$ and $M$ to empirically validate this theoretical condition?", "limitations": "See weaknesses and questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. PPL leverages predicted future trajectories to convert human interventions into contrastive preference labels over forecasted states, enabling proactive error prevention rather than reactive correction. This addresses the distributional shift in traditional imitation learning and reduces the cognitive burden on human supervisors by visualizing potential failure scenarios.\n2. Experiments on autonomous driving (MetaDrive) and robotic manipulation (Robosuite) tasks demonstrate that PPL achieves near-optimal policies with significantly fewer human demonstrations (e.g., 2.9K vs. up to 6.9K in baselines) and higher success rates (76% in MetaDrive with real humans), outperforming state-of-the-art methods like PVP and Ensemble-DAgger.\n\nWeaknesses: \n1. The paper acknowledges that all experiments are conducted in simulation (MetaDrive and Robosuite), lacking validation on physical robots. Simulated environments omit real-world challenges like sensor noise, mechanical imperfections, and unmodeled dynamics (e.g., gripper precision in Nut Assembly or traffic unpredictability in driving). A rigorous study on real robots is essential to verify PPL’s robustness.", "was_revised": true, "final_justification": "My concerns have been addressed.", "rebuttal": "Thank you for your careful reading and constructive feedback on our manuscript. We have organized our replies to each of your questions below.\n\n__Strengths And Weaknesses:__\n\n>1. The paper acknowledges that all experiments are conducted in simulation (MetaDrive and Robosuite), lacking validation on physical robots.\n\nWe appreciate the reviewer’s suggestion regarding real-world experimentation. However, we would like to emphasize that this work focuses on the algorithmic design and theoretical foundations of a new human-in-the-loop machine learning method. As a NeurIPS submission, our primary contribution lies in advancing the core machine learning methodology, which we rigorously validate in simulation environments with real human feedback. We agree that deploying the algorithm in physical systems is an important direction, and we plan to pursue real-world robotic experiments as part of future work.\n\nMoreover, __our current evaluations involve real human participants__, and Table 1 in our paper demonstrates that __PPL is a more human‑friendly and efficient interactive imitation learning method__. Table 1 shows that PPL yields a higher success rate (PPL 0.76 vs. 0.46 for the best baseline PVP) and reduces expert takeovers (PPL 2.9 K vs. PVP 4.9 K). The experiments with real human participants can help bridge the real-world robotic experiments in our future work.\n\n>  Simulated environments omit real-world challenges, so it is essential to verify PPL’s robustness.\n\nWe perform additional experiments to show that __PPL is robust to both the noise in the trajectory prediction model and the perturbation in the expert actions__.\n\nFirst, we conducted additional experiments by injecting random noise into our trajectory prediction models. __With an imperfect predictive model, PPL still outperforms all the baselines.__ We inject random noise $e_{noise}$ to the outputs $s’$ of the trajectory predictor, and we set the norm $||e_{noise}||_2 = \\epsilon * ||s’||_2$. Then we set the constant $\\epsilon = 0, 0.1, 0.2, …$ to test PPL’s robustness to noise in trajectory predictors.\n\nUsing the same setups as Tables 3–5 (MetaDrive, Table Wiping, and Nut Assembly), we find that with a noisy predictive model, PPL still outperforms all the baselines. In MetaDrive and Table Wiping, PPL keeps the best performance when the noise $\\epsilon \\leq 0.2$. In Nut Assembly, PPL outperforms all the baselines when $\\epsilon \\leq 0.1$.\n\nIn MetaDrive, we report the result of PPL with 10K total training data.\n\n|MetaDrive |PPL ($\\epsilon = 0$) | PPL ($\\epsilon = 0.1$) | PPL ($\\epsilon = 0.2$) |  PPL ($\\epsilon = 0.3$) | PVP (best baseline) |\n|---|---|---|---|---|---|\n|Test Success Rate| 0.80 | 0.73 | 0.64 | 0.51 | 0.56 |\n\nIn Table Wiping, we report the result of PPL and baselines with 2K total training data.\n\n|Table Wiping |PPL ($\\epsilon = 0$) | PPL ($\\epsilon = 0.1$) | PPL ($\\epsilon = 0.2$) |  PPL ($\\epsilon = 0.3$) | PVP (best baseline) |\n|---|---|---|---|---|---|\n|Test Success Rate| 0.80 | 0.76 | 0.70 | 0.56 | 0.58 |\n\nIn Nut Assembly, we report the result of PPL and baselines with 200K total training data.\n\n| Nut Assembly|PPL ($\\epsilon = 0$) | PPL ($\\epsilon = 0.05$) | PPL ($\\epsilon = 0.1$) |  PPL ($\\epsilon = 0.15$) | PVP (best baseline) |\n|---|---|---|---|---|---|\n|Test Success Rate| 0.51 | 0.47 | 0.41 | 0.30 | 0.35 |\n\nThen, we show that __PPL is robust to noise and perturbation in the expert__. Following Table 3, we use a well-trained PPO expert as a proxy human policy, which achieves a 0.83 success rate. Then, we inject Gaussian noises $\\mathcal{N}(0, \\sigma^2 I)$ to the expert actions, and we evaluate how $\\sigma$ affects the test success rate of PPL in MetaDrive with 10K total training steps. The result shows that PPL is robust to a noise $\\sigma \\leq 0.2$ in expert actions in MetaDrive.\n\n| Noise $\\sigma$ in expert policy | Test Success Rate of PPL | \n|--------|--------|\n| 0    | 0.80  |\n| 0.1  | 0.77  |\n| 0.2  | 0.70  |\n| 0.3  | 0.62  |\n| 0.4  | 0.43  |\n\n__Questions:__\n\n>1.  Would it be beneficial to conduct additional experiments varying both beta and M to empirically validate this theoretical condition of $\\beta \\leq M^{-2}$?\n\n__Our experiment that varies $\\beta$ in MetaDrive empirically validates this theoretical condition.__\n\nAccording to line 203, $M = \\sup_{s, a, a’} |\\log \\pi_h(a|s) - \\log \\pi_h(a|s’) |$, which is a constant determined by the human policy over the entire state-action space and thus difficult to measure directly. To empirically probe the theoretical condition $\\beta \\leq M^{-2}$, we conduct an additional experiment that varies $\\beta$ from 0.01 to 1.0 in MetaDrive, using a fixed budget of 10K training steps:\n\n| PPL | $\\beta = 0.01$ | $\\beta = 0.05$ | $\\beta = 0.1$ | $\\beta = 0.2$ | $\\beta = 0.4$ |  $\\beta = 0.6$ |   $\\beta = 0.8$ | $\\beta = 1.0$ | \n|---|---|---|---|---|---|---|---|---|\n|Test Success Rate| 0.61 | 0.73 | __0.80__ | __0.80__ | 0.77 | 0.69 | 0.58 | 0.47 |\n\nThis empirically shows that the PPL performance degrades when $\\beta$ is too large, due to the violation of $\\beta \\leq M^{-2}$. In addition, when $\\beta$ is too small, the behavioral cloning loss dominates the preference loss $\\mathcal{L}_\\text{pref}$, harming PPL’s performance empirically."}]

Author rebuttal:

Meta-review: The authors did a good job of addressing numerous concerns raised by the reviewers, ultimately bringing all reviewers into a positive view of the paper. The concerns raised by the reviewers appear, generally, to be reasonable questions that a typical reader might have about the paper, so it will be important for the authors to revise their paper in way that anticipates and answers these concerns for future readers.
Decision: Accept (spotlight)