Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution
Authors: Zhanyi Sun
Shuran Song
Venue: NeurIPS 2025 spotlight ()
Keywords: Imitation Learning
Out-of-Distribution Generalization
Robotic Manipulation
Primary area: applications

Abstract: Visuomotor policies trained via behavior cloning are vulnerable to covariate shift, where small deviations from expert trajectories can compound into failure. Common strategies to mitigate this issue involve expanding the training distribution through human-in-the-loop corrections or synthetic data augmentation. However, these approaches are often labor-intensive, rely on strong task assumptions, or compromise the quality of imitation. We introduce Latent Policy Barrier, a framework for robust visuomotor policy learning. Inspired by Control Barrier Functions, LPB treats the latent embeddings of expert demonstrations as an implicit barrier separating safe, in-distribution states from unsafe, out-of-distribution (OOD) ones. Our approach decouples the role of precise expert imitation and OOD recovery into two separate modules: a base diffusion policy solely on expert data, and a dynamics model trained on both expert and suboptimal policy rollout data.  At inference time, the dynamics model predicts future latent states and optimizes them to stay within the expert distribution. Both simulated and real-world experiments show that LPB improves both policy robustness and data efficiency, enabling reliable manipulation from limited expert data and without additional human correction or annotation. More details are on our anonymous project website https://latentpolicybarrier.github.io.

Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution
Zhanyi Sun
Shuran Song
Stanford University
pojject-teentplllyybbarrrrgggiie
Abstract
Visuomotor policies trained via behavior cloning vulnerable covariate shift, where small deviations from expert trajectories can compound into failure. Com- mon strategies mitigate this issue involve expanding the training distribution through human-in-the-loop corrections or synthetic data augmentation. However, these approaches are often labor-intensive, rely on strong task assumptions, compromise the quality of imitation. We introduce Latent Policy Barrier, a frame- work for robust visuomotor policy learning. Inspired by Control Barrier Functions, LPB treats the latent embeddings of expert demonstrations as an implicit barrier separating safe, in-distribution states from unsafe, -fiisiiiiitttnoionn (OOD) ones. Our approach decouples the role of precise expert imitation and OOD recovery into two separate modules: base diffusion policy solely on expert data, and dynamics model trained on both expert and suboptimal policy rollout data. At inference time, the dynamics model predicts future latent states and optimizes them to stay within the expert distribution. Both simulated and real-world experiments show that LPB improves both policy robustness and data efficiency, enabling reliable manipulation from limited expert data and without additional human correction annotation.
Introduction
Interactive Correction (Prior works)
Dynamics- guidee Correction (Ours)
In control theory, Control Barrier Functions (CBFs) offer a principled mechanism to en- force safety in autonomous systems by explic- itly defining a safety set and ensuring states remain within it during execution [2]. Moti- vated by the conceptual clarity of CBFs, we seek a similar mechanism for visuomotor poli- cies learned purely from data, where explicit analytical definitions of safety constraints and system dynamics are typically unavailable. In learning-bass visuomotor control, the analo- gous challenge arises from covariate shift [55]: minor deviations from demonstrated expert be- haviors can quickly compound, pushing agents into -fi-t-iiiitiitiionn (OOD) states and caus- ing task failures. Traditional behavior cloning (BC) methods are particularly vulnerable to co- variate shift, severely limiting their effectiveness and reliability in real-world applications.
Suboptimal
Fallure
Expert Expended expert distribution (w. correction demos)
Dynamics steering Rollout distribution
Figure 1: To mitigate covariate shift, prior DAgger- like methods (Left) expand the training distribution via human-in-the-loor corrections (-), often introducing suboptimal trajectories. contrast, our Latent Policy Barrier treats the expert distribution (m) as an implicit barrier, using leamed dynamics model to detect de- viations and steer the policy back toward expert behavior (O). The dynamics model trained both expert demos (-) and policy rollouts (=), requiring no additional human input.
Current approaches to address covariate shift primarily involve expanding the training distribution.
One common approach is uuman-in-the-lopg interventions, such as DAgger [11, 55, 36, 29, 40, 39], which repeatedly collect corrective demonstrations whenever agents deviate from expert behaviors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(Figure left). Another strategy involves synthetic data augmentation methods that leverage task- specific invariances [17, 42]. However, these approaches are either labor-intensive or rely heavily on strong prior assumptions about the environment. Moreover, both approaches risk introducing inconsistent or suboptimal demonstrations into the training dataset, potentially degrading overall policy performance [41]. This limitation exposes fundamental trade-off in robust visuomotor policy learning: precise imitation benefits from consistent, high-quality expert datasets, whereas robustness inherently demands exposure to diverse and often suboptimal data. Constructing a single dataset that balances these competing needs is inherently challenging.
To resolve this tension, our key insight is to explicitly decouple these conflicting objectives (Figure 1 right). Inspired by CBFs, we propose Latent Policy Barrier (LPB), which implicitly treats the latent expert demonstration distribution itself as a barrier that separates safe, in-distribution states from unsafe, out-of-distribution regions [7]. Unlike analytically defined CBFs, LPB does not require explicit safety sets or system dynamics. Instead, it maps high-dimensional expert states into learned latent space and uses their embeddings to define a barrier for detecting and correcting deviations.
To decouple precise expert imitation and OOD recovery, LPB leverages two complementary compo- nents: (a) a base diffusion policy trained exclusively on consistent, high-quality expert demonstrations, ensuring precise imitation; and (b) an action-conditions broader, mixed -uallityy dataset combining expert demonstrations and automatically generated rollout data [4, 45, 9, 62]. We collect the rollout data by executing intermediate checkpoints saved during base policy training. Importantly, the rollout data naturally covers diverse deviations around the policy's 's own distribution without requiring explicit success labels, task rewards, or additional human teleoperation. At inference time, L.BB ensures that the agent stays within the expert distribution by performing policy steering in the latent space. LPB uses the dynamics model to predict future latent states conditioned on candidate actions output from the base policy. Then LPB minimizes the distance between the predicted future latent states and their nearest neighbors from the expert demonstrations in the same latent space. This latent-space steering approach simultaneously achieves high task performance and robustness, resolving deviations without compromising imitation precision.
visual latent dynamics model trained on a
In summary, we introduce Latent Policy Barrier in the context of behavior cloning. LPB offers the following advantages: 1) improves sample efficiency by decoupling expert imitation from out-of- distribution correction enabling the policy to focus on learning from a small amount of high-quality human demonstrations; 2) enhances robustness through the use of dynamics model trained on inexpensive, lower-quality policy rollout data; and 3) plug-and-play compatibility with off the-tff pre-trained policies, improving their robustness without requiring policy retraining or fine-tuning.
Experimental results across simulated and real-world manipulation tasks demonstrate that LPB is able to enhances both the robustness and sample efficiency of visuomotor policy learning. Code and data for reproducing the result will be made publicly available.
2 Related Work
Mitigating Covariate Shift in Imitation Learning: Behavior Cloning (BC), despite its simplicity, remains a strong baseline for working solely with expert demonstrations [51]. An extensive body of works mitigates covariate shift of BC by expanding the expert distribution. Prior works use interactive expert interventions [55, 36, 29, 40. 14, 39, 25] that are iteratively integrated into the policy learning loop to improve performance. Alternative ways to expand the training distribution include synthetic data generation with task invariances [17, 42, 65]. noise injection [36. 27], or dynamics model-guided state-action pair generation [49, 28). Instead of focusing on the data, Inverse Reinforcement Learning combats covariate shift by alternately collecting on on-policy rollouts and updating reward function that penalizes deviations from expert trajectories, pulling the learner's s state distribution toward the expert's [68, 24, 18). Other lines of work include training recovery policy that automatically pushes the agent back to the in-distribution region [53] and incorporating training objectives that penalize distribution divergence [43]. Our method differs from these strategies by performing inference-time steering in latent space, bypassing the need for additional expert queries or heavy data augmentation.
Offline RL algorithms prevents the learned policy from drifting into OOD states by imposing an explicit regularization. This can take the form of policy regularization that uses a penalty to keep the learned policy close to the behavior policy or BC prior [60, 34, 20, 19, 50), conservative value estimation that pessimistically down-weights out- of-distribution actions [35, 33], or model- uncertainty quantification that learns a dynamics
Policy Learning from Suboptimal Data:
2

---

Base Policy Training (on experi data)
End training
Expert Data
Base Policy
plpt
pkptt
ckptt
Palicy
1 2
Raliout
Rellout
Rollout
Data
In-iistrbutton Stearing
Base policy denolsing
A
Latent Spaca
Minimize distance
Gradient
Expert
Dynamics Training (on both
Dynamics
8(2x) 2-T
Oynamics Model
Zt+h
export and rollout data)
future latent state
Nearest
Training Phase
Inference Phase
neighbor
Figure 2: Latent Policy Barrier: 1) During training, train base diffusion policy expert demonstration data and visual latent dynamics model on both the expert and rollout data. 2) During inference, LPB keeps the agent within expert states by steering the base policy in the latent space. We define the & distance of a latent state to its nearest expert latent state as the latent OOD score d. If ((2) exceeds a predefined threshold, back in-distribution steering invoked. Details on the anference-time steering process can be found in Algorithm 1.
model and uses an uncertainty penalty to discourage trajectories from unfamiliar regions [64, 30, 8]. These strategies can exploit large, mixed-quality datasets, but they assume external reward. Methods that learn from suboptimal demonstrations fit a reward function via enforcing trajectory rankings or regressing returns against noise-perturbed rollouts [5, 10]. Recent work shows that an agent can exploit its own rollouts using them to forecast and refine future latent trajectories [62], to autonomously accumulate new task executions [4], or label and filter heterogeneous demonstrations [9] to supplement the limited expert data. In contrast, we repurpose policy rollout data to learn dynamics model that, at inference time, nudges the agent back toward the expert distribution.
Inference-Time Policy Steering: A growing line of work improves pre-trained policy at inference time by steering, without requiring additional data fine-tuning. Guided denoising biases diffusion- based policies toward specified goals or reward signals by injecting gradient guidance into the denoising process [26, 54, 46). Value-based filtering samples actions from a generalist policy and executes the one ranked highest by value function [47]. Hluman-in-the-loee
steering treats user provided sub-goals, corrections, or preferences as constraints injected into the policy's sampler [58, 61]. Model- preeec-preiittve refinement combines base policy with a dynamics model to improve task performance or respect safety constraints [67, 56, 52, 48]. Similarly, LPB also uses a latent visual dynamics model [22, 21, 23, 59, 66] to steer the policy. Unlike prior works, LPB performs gradient- based action corrections using gradients from the latent dynamics model. The corrections pull predicted future states toward the expert manifold, unifying classifier guidance with model- predictive foresight while needing no explicit goals, rewards, or human input.
Method: Latent Policy Barrier
We consider the problem of visuomotor policy learning with behavior cloning. Given a demonstration dataset of observation-action pairs Dexpen = {(or, ax) tt the goal of the base policy is to learn
л O - AA that maps observations to actions by imitating expert behavior. Following the notation introduced by [12), each time step f, the policy xo (A,(O;) receives the most recent T, observations
O = {o:-To+1..-. } and predicts the next T, actions A, = {ar,... 4 Among these predicted actions, the first Tr steps, far...., A1..,, are executed without replanning. The process then repeats after receiving new observations.
The goal of Latent Policy Barrier is to learn such a robust policy ma from a limited set of high-quality expert demonstrations, capable of consistently performing long-horizon robot manipulation tasks with minimal compounding errors and drifting into out-of-disttibution states (Figure 2). In the following subsections, we detail our approach: base policy and dynamics model training $3.1, and test-time optimization to avoid deviations $3.2.
3.1 Policy and Dynamics Model Training
We train the policy ny ( (A) using diffusion model [12] on limited set of high-quality expert demonstrations. Given the high-dimensional visual observations, the policy uses visual encoder
3

---

hr(Or) that maps raw image observations into latent representations. The visual encoder and the noise-prediction network are trained end-to-end with behavior cloning loss.
To enhance dynamics model's generalization beyond the distribution of expert demonstrations, we collect additional dataset of exploratory rollouts. These trajectories automatically generated by rolling out intermediate checkpoints of the diffusion policy saved during training. Concretely, after an initial warm-up phase of to training epochs, during which the base policy is still highly random, we save policy checkpoints at fixed intervals. Every At epochs we save a checkpoint ckpt, and roll it out for N full episodes in the evaluation environment, recording all transitions regardless of task success or failure. This schedule continues until the final training epoch annl- The resulting rollout dataset covers both early exploratory and near-convereged behavior, yielding a much wider stute-action distribution than the original expert demonstrations. The diverse transitions in the rollout dataset enables the dynamics model to generalize more effectively, especially to novel states encountered at test time. Importantly, the rollout data is collected without human correction or reward annotations. Since it is generated as a natural byproduct of base policy training, no additional teleoperation manual labeling is required. This makes it inexpensive source of training data for learning a generalizable dynamics model that can steer the policy back toward expert-demonstrated states. We ablation the choice of data source for dynamics model training in Appendix A.
The visual latent dynamics model, denoted as dg. trained to predict future latent observations given current observations On and a sequence of future candidate actions A,. It consists of two components: a frozen visual encoder hg. shared with the base policy, and learnable dynamics predictor fa- The full model is thus written as:
de dOr. AA) = for (O.,),)
We reuse the visual encoder he from the behavior cloning policy, which is trained end-to-end to optimize for task execution. This design choice is motivated by the following consideration: during test-time optimization, the dynamics model is used to predict future latent states in the same embedding space that the policy relies on to make action decisions. Freezing the encoder ensures consistency between the latent representations used by the policy and those optimized through the dynamics model (3.2). It also stabilizes training by preventing collapse of the representation space. The dynamics predictor fo is implemented as decoder-only transformer ([57, 44, 6, 66]. Given encoded latent representations of current observations and future action sequence, the dynamics model predicts the encoded future latent observation at specified prediction horizon.
The model is trained using a latent-space mean squared error (MSE) loss between the predicted future latent and the ground-truth latent of the future observation:
Ldynamics (O,, Av, OC+T,) = [[ho(ortT,) - So(ho(O,), 2
(2)
3.2 Test-Time Optimization
At inference time, our goal is to mitigate compounding errors by steering the policy back toward in-distribution expert states whenever deviations from those states are about to occur. We detect OOD states using the latent OOD score, which measures how far the current observation lies from the expert data distribution in the latent space. Specifically, given an observation o, we first encode into a latent representation z = (a) using the frozen visual encoder he from the base policy. To quantify distributional shift, we identify the nearest neighbor of in the expert latent space, denoted as 2 NN E exppen- The latent OOD score 5(2) is defined as:
5(2) = ||2 - NN 12
A higher value indicates that a is farther from the expert data manifold and is thus more likely to be out-of-distribttion for the base policy. At inference timestep f, we employ the learned dynamics model de to refine the stochastic denoising process (Algorithm 1). At timestep t, the agent observes Oz, which is input to the base policy's denoising process. We first compute the latent OOD score for the current observation. If it's lower than pre-defined threshold т. then the action generated by the base policy is executed. Otherwise, we refine the denoising process using gradient guidance. Taking inspirations from classifier guidance in diffusion models (l15)) and its applicatons robotics-related domains ([1, 26, 63]). we take the gradient of latent OOD score of predicted future latent state through the dynamics model and use this gradient to refine the action denoising process. Denote the noisy action sample at timestep & and at denoising iteration & as A&, we extend classifier guidance to refine

---

A by minimizing the latent OOD score of predicted future state 24+h do (ho(Or), 41). We define the modified noise prediction as
{(A+) = ca (Aft) - n
- ar. V xp 6 od (2) 44)
(4)
where 7) is the guidance scale. The classic classifier guidance approach requires training a classifier on noisy data samples, which is less practical in robotics settings where executing random actions can be unsafe or infeasible. Our rollout data offers a practical alternative: actions in policy rollouts naturally exhibit greater variability than expert demonstrations. As a result, the dynamics model trained on this data well-equipped to predict the outcomes of noisy action samples. To avoid unreliable guidance from highly stochastic early denoising steps where actions diverge significantly from the expert manifold we restrict gradient- are more structured. For instance, Diffusion Policy often uses up to 100 denoising steps with DDPM, we only choose to refine only a subset of the denoising steps, e.g., the last 10 steps. By reweighting the action sampling distribution in favor of actions that lower the latent OOD score of predicted future states, the gradient guidance effectively steers the agent toward expert-like states. better understand the effectiveness of the gradient guidance strategy, we also compare against alternative optimization methods, including direct gradient descent on the action sequence and model predictive control (MPC). Ablation results are provided in Appendix A.
guidance to only the final K guide denoising steps, where samples
Algorithm Latent Policy Barrier (Inference time)
Require: Base policy wa, dynamics model dş, visual encoder has expert dataset Dexport, latent OOD score threshold T
1: Preprocess: Encode all expert observations into latent states: Zexpert (ha(o) 10 Despen)
2: for each timestep f do
3: Observe O, = (OI-T,+1,... 01
4: Encode latent state 2, = ho
5: Compute latent OOD score d(zz) by performing nearest neighbor search in Zexpen (Eq. 3)
6: if 5(2)) > then
b Out-of-distribution detected
7:
for denoising step k: K..... K K guide do
8:
Sample noisy action A& from me
9:
Predict future latent state: b4th (2t, At) Compute gradient: V (((+h)
10:
11:
Apply gradient guidance to get ((A)) (Eq. 4)
12:
end for
13:
Output final action sample At after K denoising steps
14: else
15:
Output A, To(-)O,)
16:
end
Execute first To steps of A,
18:
end for
4 Experiments
Our evaluation focuses on two questions: 1) Does LPB improve sample efficiency and robustness of visuomotor policy learning compared to baseline methods? 2) When the agent drifts, can LPB detect the deviation at the right moment and steer the agent back to expert-like states? We benchmark across a suite challenging robotic manipulation tasks in both simulated environments and on a real robot (Figure 3). In simulation, benchmark three suites: Push-T [12, 16], Robomimic [41], and the multi-task, Language-conditoons challenging tasks, Square, Tool Hang, and Transport. For the real robot experiment, test the Cup Arrangement task from [13] and the Belt Assembly task from the NIST board assembly challenge [32].
Liberol0 [38]. For Robomimic, we select its three most
4.1 Simulation Benchmarks
To evaluate the sample efficiency of our method, we focus on a limited demonstration regime, where suboptimal rollout data can play significant role in improving performance. For each Robomimic task (Square, Tool-Hang, Transport) and for Push-T, we keep 20% of the original expert demonstra- tions. For each task, base diffusion policy is trained these demonstrations. For Liberol0, use
5

---

Push-T
Square
Tool Hang
Libero-10
Arrangement
Belt Assembly
Figure 3: Benchmark Tasks. We evaluate our method on diverse set of manipulation tasks in both
simulation and real-world.
Task
Expert BC
Mixed BC
Filtered BC
CCIL
CQL
Ours
Square
0.56 0.016
0,50 0.025
0.65 0.000
0.63 0.025
0.0
0.65 000019
Transport
0.68 0.016
0.60 0.016
0.79 0.033
0.69 0.004
0.0
0.85 0.009
Tool Hang
0.27 0.009
0.24 0.016
0.29 0.000
0.14 0.025
0.0
0.39 0.009
Push-T
0.51 0.002
0.47 + 0.026
0.59 0.0.00
0,48 0.034
0.29 = 0.002
0.65 + 0.012
Libero-10
0.65 +: 0.009
0.50 0.017
0.71 0:0000
0.61 0.026
0.0 0.0
0.75 0.038
Table I: Suceess rates across simulated tasks with 20% demonstrations. Mean and std. deviation over 3 checkpoints and on 50 different held-out environment initial conditions.
all 50 provided demonstrations for each of the ten tasks to train a language-condtionee multi-task base diffusion policy. During policy training, we save intermediate checkpoints at fixed intervals and use them to collect additional rollouts. See Appendix B for further implementation details.
We compare LPB against the following baselines:
Expert BC [12]: Diffusion policy trained on only expert data.
Mixed BC: Diffusion policy trained on both rollout data and expert data.
Filtered BC [9, 45, 4]: We augment the expert data with successful rollout trajectories and
re-train the diffusion policy the aggregated dataset. We use the success criteria defined by the original benchmarks.
CQL [35]: Conservative Q-Learning (CQL) is offline RL algorithm that learns value function that explicitly penalizes overestimation of unseen actions. CQL requires task rewards. For Push-T, use the dense reward from [12]; for Robomimic and Liberol0 tasks, sparse reward, which gives on success and 0 otherwise.
the
CCIL [28]: CCIL enhances the robustness of behavior cloning by generating corrective data to augment the original expert data. Since the original method was proposed for tasks with low dimensional state space, adapt tasks with image observations by encoding images with the frozen visual encoder in the base policy trained with behavior cloning loss.
As shown in Table 1, under the limited- detiiedddmmmotttttiinn setting, LPB matches exceeds every baseline on all simulated tasks, showing strong sample efficiency. The largest gains appear on the long- horizon, precision-sensitive. tasks, Tool-Hang and Transport, highlighting LPB's ' ability to correct minor action deviations that otherwise compound over time. On the most challenging Tool- Hang task, Filtered BC offers only marginal improvement, indicating that even "successful" rollouts still contain suboptimal or inconsistent actions that hurt BC. CQL obtains zero reward on Robomimic and Libero 10 tasks, possibly due the absence of dense reward function, consistent with results reported in the Robomimic paper [41].
Inference-tine Robustness. To investigate how well LPB mitigates covariate shift, we inject noise perturbations to the output action at inference time. At every time step. with probability e (0.0, 0.1, 0.2, 0.3, 0.4]. J. Gaussian noise is added to the output action. We evaluate LPB alongside Filtered BC, Expert BC, and CCIL on the Transport task. Figure 4a shows that LPB maintains the highest success rate across all noise levels. While the baselines exhibit worsening performance p increases, LPB degrades the least, indicating robustness to inference- -tie perturbations.
Sample Efficiency on Expert Demonstrations. We next vary the fraction of expert demonstrations used to train the base policy on Tool-Hang, from 20% up to the full dataset, to see whether LPB continues to add value as more expert data becomes available. Figure 4b shows that L.BB's advantage largest in the low-data regime (up to 60% demos), the margin narrows as the dataset approaches 100%, reflecting performance saturation.

---

LPB (Ours)
Filtered BC
Expert BC
CCIL
Success Rate
0.8
0.6
0.6
0.2
0.2
0.0
(a) Inference-time Noise
(b) Amount of Expert Demo (%)
(c) Amount of Rollout Trajectories
robust under Figure 4: (a) Performance under perturbations Transport, showing that LPB is
external perturbation. (b) Impact of expert demonstrations on Hool-Hang. showing that LPB can achieve better performance under low-data regime (up to 60% demos). (c) Impact of rollout data Tool-Hang, showing that LPB consistently improves performance as more rollout data is used to train the dynamics model despite the suboptimality in the data.
Current Obs NNin
Latent OOD Score
Timesteps
Timestops
(Baselino)
LPB (Ours)
Figure 5: Simulation Rollouts. Left: Filtered BC baseline. Right: LPB. For both methods, we show RGB frames sampled from representative rollout (middle row), the corresponding nearest expert demonstration state to the sampled frames (top row), and the latent OOD score over time (bottom row). The dashed line in the latent OOD score curve marks the threshold above which LPB's s test-time optimization is invoked for. Filtered BC misses the two-arm hand-over, the latent OOD score spikes, and the episode fails. LPB ensures near 7 throughout the entire rollout, stays on the expert manifold, and completes the task.
Impact of Rollout Data Quantity. To evaluate how the performance of LPB depends on the amount of rollout data used for dynamics model training, we conduct experiments on the Tool Hang task (Figure 4c). We vary the number of rollout trajectories used to train the dynamics model: 0. 50, 150, 300, and 500. When using 0 rollouts, the dynamics model is trained solely on expert demonstrations. We compare LPB against Filtered BC, which filters successful trajectories from the same rollout data and combines them with demonstrations to retrain BC policy. Due inconsistencies and suboptimal behaviors present even in the successful rollouts, Filtered BC yields little improvement upon the base policy. In contrast, LPB consistently improves as more rollout data is added, suggesting that the dynamics model benefits from the broader state-action coverage despite the suboptimality. Performance gains of LPB saturate around 300 rollouts.
Latent OOD Score Visualization and Analysis. To validate our design choice of the latent space and the latent OOD score formulation, we plot the latent OOD score over time for LPB and the Filtered BC baseline on a representative Transport rollout (Figure 5). As defined in 3.2, the latent OOD score the & distance between the encoded observation and its nearest expert neighbor, where the encoder comes from the base policy. In the rollout generated from the Filtered BC baseline, the two arms miss the hammer hand-over, leading to unrecoverable failure (frames with red bounding box). At these failure timesteps, the latent OOD score spikes above the threshold indicating the
7

---

Policy
(Ours)
1.0
0.85
Succees Rate
C.8
0.6
0.4
0.2
0.0
In-distribution
Out-of-diatributionn
In-distribution
Out-of-distribution
Initial Poses
Initial Poses
Figure 6: Cup Arrangement Setup and Results. Evaluation of the base policy (Expert BC) and LPB on the real-world Cup Arrangement task.
Base Policy
Success Rate
LPB (Ours)
De
0.4
[1] Init
(2) Hook on 1st Pulley (3) Hook on 2nd Pulley (4) Release Finish
Figure 7: Belt Assembly Setup and Results. Evaluation of the base policy (Expert BC) and LPB on the real-world Belt Assembly task.
state has drifted far from the expert manifold. In the rollout generated by LPB the latent OOD score rises as the hand-over phase begins but stays near т; gradients from the dynamics model guides the policy back toward expert-like states (frames with green bounding box). Throughout the episode the score remains mostly below showing that LPB keeps the agent inside the demonstrated distribution. These observations support that (1) the visual encoder trained end-to-end with behavior cloning loss induces latent space in which the nearest-expert distance reflects how far the current state has drifted from demonstrated behavior, and (2) this & distance serves as reliable signal for OOD detection and correction, enabling effective latent-space steering by LPB. See Appendix A for results on alternative latent space choices.
Real-World Evaluation
4.2.1 Cup Arrangement with Off-the-Shelff Pretrained Policy
We first assess LPB's ability to improve the robustness an off-the-shelf pretrained policy on a real robot. As the availability of such pretrained policies continues to grow [31. 3, 37], the plug-and-play nature of LPB and ability to enhance their robustness makes especially valuable.
Task Setup. Specifically, we use the pre-trained diffusion policy for the Cup Arrangement task as our base policy [13]. In this task, robot arm with a wrist-mounted RGB camera and compliant finray gripper needs to first rotate the cup so its handle faces right, then pick up the cup, and finally place the cup upright on the saucer.
Base Policy and Dynamics Model. The base diffusion policy checkpoint, trained on in-the- nn-the-wild expert trajectories, serves the base policy for LPB. To collect training data for the dynamics model, we roll out the pre-trained policy from deliberately out-of-distribution initial poses, gathering 80 trajectories. Because intermediate policy checkpoints unavailable, augment the dataset with additional 40 human play trajectories recorded via the handheld UMI device [13].
Results. We perform two groups of experiments: in-distribution initial poses, where the wrist camera initially observes both the cup and the saucer (Figure left), and out-of-distribution initial poses, where the camera initially sees neither object (Figure 6 middle). As shown in Figure 6 right, LPB matches the base policy on in-distribution initial poses and substantially outperforms it on OOD initial poses, confirming that latent-space steering is able recover from distribution shifts without compromising nominal performance.
To illustrate how the dynamics model corrects distribution shift, Figure 8 plots four frames sampled from rollouts under the base policy (top row) and L.B (middle row), with LPB paired with its nearest expert demonstration image (bottom row) and the corresponding latent OOD scores. Both rollouts

---

Latent OOD Score
Base Pollcy
Latent OOD Score
LPB (Ours)
NN in Expert Data
Figure 8: Cup Arrangement Rollouts. Left: RGB frames from base policy rollouts (top). LPB (middle), and the nearest expert demonstration to LPB (bottom). Red borders indicate out-of- distribution observations; green denotes in in-distribution. Right: corresponding latent OOD scores. With LPB, the robot moves downward reveal the tabletop and complete the task, while the base policy drifts further and reaches joint-limit.
start from the same wrist-camera pose showing neither cup nor saucer. Guided by the dynamics model, LPB tilts the camera downward to reveal the objects, driving its OOD score below the threshold and successfully completing the task. By contrast, the base policy continues moving in the wrong direction with its observations remain OOD throughout. It fails to locate the objects and terminates at a joint-limit failure. Although the demonstration data contains only in in-the-wild images and the latent space is noisier than simulated tasks, the latent-space nearest neighbor selected by LPB still redirects the robot to in-distribution views, showing that LPB is robust to visual clutter and attends to task-relevant features.
4.2.2 Belt Assembly
We further evaluate on the Belt Assembly task from the NIST task board.
Task Setup. As illustrated in Figure 7, the task begins with the belt already grasped by the robot's gripper. First, the robot positions the belt to hook it over the small pullcy. Next, it moves downward while stretching the belt to loop its opposite side around the large pullcy. The robot then performs 180" rotation around the large pulley to thread the belt. Finally, it lifts the gripper to release the belt.
Base Policy and Dynamics Model. We collect 200 expert demonstrations, each performed with slight variations the board position while keeping the initial robot pose fixed. To train the dynamics model, we additionally collect 400 rollout trajectories executed under randomly initialized board positions and initial robot poses.
Results. This task poses significant challenge for naive behavior cloning due to its contact-rich nature and the high precision required for successful execution. At test time, we perform 40 rollouts for both the base policy and LPB, varying the initial robot pose and board position. All evaluations use the same set of test cases and initial robot configurations for fair comparison. The base BC policy achieves success rate of 0.55. As shown in Figure 7 (right), LPB improves this performance to 0.75. Figure shows representative rollout frames for both the base policy and LPB, along with their corresponding latent OOD score plots. At the start of the rollout, the observation lies out-of- distribution, reflected by high latent OOD score. LPB first recovers the robot to an in-distribution pose, enabling the base policy to subsequently complete the task. In contrast, the base policy alone fails to recover, resulting task failure.

---

Latent OOD Score
Base Policy
Latent OOD Score
LPB (Ours)
NN in Expert Data
Figure 9: Belt Assembly Rollouts. Left: RGB frames from base policy rollouts (top). L.B (middle), and the nearest expert demonstration to LPB (bottom). Red borders indicate out-of-distribution observations; green denotes in-distribution. Right: corresponding latent OOD scores. LPB is able to help the robot recover to in-distribution expert states and complete the two threading motions.
5 Conclusion and Discussion
In this paper, propose a novel method, Latent Policy Barrier, that transforms behavior cloning from a passive imitator into an active, self-correcting controller with dynamics model. By treating the expert manifold latent barrier and steering a base policy with a learned dynamics model, LPB achieves reliable imitation and eliminates costly human corrections. Extensive simulated and real-robot experiments show that LPB surpasses or matches baselines when trained on just small slice of the demonstration set, while maintaining robust performance under severe perturbations, highlighting a practical path toward reliable visuomotor learning.
Limitations and Future Work. Currently, LPB only corrects short-term, local deviations around the expert demonstration. Future work can expand the dynamics model training distribution and employ long-horizon reasoning to recover from more aggressive deviations. In addition, the visual latent dynamics model LPB is trained per task, limiting reuse and requiring domain. Future work can explore training multitask dynamics models that capture shared dynamics structure, offering zero-shot generalization and amortized training cost across diverse robotics domains. Moreover, the latent OOD score in LPB assumes access to expert training data to define the in-distribution manifold; in scenarios where only pretrained base policy is available without paired training data, this metric cannot be directly computed. Future work could investigate distribution- free uncertainty quantification enable robust policy learning in the absence of expert training data.
data for each
6 Acknowledgements
We thank Mengda Xu for insightful discussions throughout the various stages this project. We are also grateful to Yihuai Gao for assistance with the ARX arm setup and to Yifan Hou for support with the URS setup. We appreciate the valuable feedback from all REALab members on the manuscript. This work was supported in part by the NSF Awards #2 143601, #2037101, and #2132519, and by the Toyota Research Institute. We thank TRI for providing the UR5 robot hardware and ARX for providing the ARX robot hardware. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
10

---

References
[1] Anurag Ajay, Yilun Du Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
[2] Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC), pages 3420-3431. Icee, 2019.
[3] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. To: A Lagion-Language-Actiie Flow Model for General Robot Control. arXiv preprint 2rX1v:2410.24164, 2024.
[4] Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauzá, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, al. Robocat: A self- improving generalist agent for robotic manipulation. arXiv preprint arXiv:2306.1 11706, 2023.
[5] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. International conference machine learning, pages 783-792. PMLR, 2019.
[6] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
[7] Fernando Castaneda, Haruki Nishimura, Rowan Thomas McAllister, Koushil Sreenath, and Adrien Gaidon. In-distribution barrier functions: Self-supervised policy filters that avoid out-of- distribution states. In Learning for Dynamics and Control Conference, pages 286-299, PMLR, 2023.
[8] Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems, 34:965-979, 2021.
[9] Annie S Chen, Alec M Lessing, Yuejiang Liu, and Chelsea Finn. Curating demonstrations using online experience. arXiv preprint arXiv:2503.03707, 2025.
[10] Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on robot learning, pages 1262-1277 PMLR, 2021.
[11] Sonia Chernova and Manuela Veloso. Interactive policy learning through confidence-based autonomy. Journal of Artificial Intelligence Research, 34:1-25, 2009.
[12] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusson policy: Visuomotor policy lcarning via action diffusion. The International Journal of Robotics Research, page 027836492412736688 2023.
[13) Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402. 10:.2999..0 2024.
[14] Shivin Dass, Karl Pertsch, Hejia Zhang, Youngwoon Lee, Joseph J Lim, and Stefanos Niko- laidis. Pato: Policy assisted teleoperation for scalable robot data collection. arXiv preprint arXiv:2212.04708, 2022.
[15] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[16] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on robot learning, pages 158-168. PMLR, 2022.
11

---

[17] Peter Florence, Lucas Manuelli, and Russ Tedrake. Self-supervised correspondence in visuomo- tor policy learning. IEEE Robotics and Automation Letters, 5(2):492-499, 2019.
[18] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint urXiv:1710.12458 2017.
[19] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20144 2021.
[20] Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[21] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1912.01603 2019.
[22] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2565 PMLR, 2019.
[23] Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203. 04955,, 2022.
[24] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[25] Nils Ingelhag. Jesper Munkeby, Michael C Welle. Marco Moletta, and Danica Kragic, Real-time operator takeover for visuomotor diffusion policy training. arXiv preprint arXiv:2502.02308, 2025.
[26] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205. 199991 2022.
[27] Liyiming Ke, Jingqiang Wang, Tapomayukh Bhattacharjee, Byron Boots, and Siddhartha Srinivasa. Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation. In 2021 IEEE international conference on robotics and automation (ICRA), pages 6185-6191. IEFE, 2021.
[28] Liyiming Ke, Yunchu Zhang, Abhay Deshpande, Siddhartha Srinivasa, and Abhishek Gupta. Ccil: Continuity-based data augmentation for corrective imitation learning. arXiv preprint arXiv:2310.12972. 2023.
[29] Michael Kelly, Chelsea Sidrane, Katherine Trigs-CCambblll and Mykel J Kochenderfer. Hg- dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083. IEEE, 2019.
[30] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810 21823, 2020.
[31] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprini arXiv:2406.099: 09246, 2024.
[32] Kenneth Kimble, Karl Van Wyk, Joe Falco, Elena Messina, Yu Sun, Mizuho Shibata, Wataru Uemura, and Yasuyoshi Yokokohji. Benchmarking protocols for evaluating small parts robotic assembly systems. IEEE robotics and automation letters, 5(2):883-88) 2020.
[33] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint (xXi:2110000169) 2021.
[34] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy lelarnning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
12

---

[35] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learming for offline reinforcement learning. Advances in neural information processing systems, 33: 1179-1191, 1191, 2020.
[36] Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pages 446.156. PMLR, 2017.
[37] Fanqi Lin, Yingdong Hu, Pingyue Sheng. Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. arXiv preprint 2781472410.18647, 2024.
[38] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-44791, 2023.
[39] Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke Zhu. Robot learning on the job: Human-in-the-loop autonomy and learning during deployment. The International Journal of Robotics Research, page 02783649241273990. 2022.
[40] Ajay Mandlekar, Danfei Xu, Roberto Martin-Martin, Yuke Zhu, Li Fei-Fei, and Silvio Savarese.
-n-the-loop imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733, 2020.
[41] Ajay Mandlckar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2/08.0322 038.0998. 2021.
[42] Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang. Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17500 2023.
[43] Shaunak A Mehta, Yusuf Umut Ciftci, Balamurugan Ramachandran, Somil Bansal, and Dylan Losey. Stable-be: Controlling covariate shift with stable behavior cloning. IEEE Robotics and Automation Letters, 2025.
[44] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. arXiv preprint arXiv:2209.00S: 2022.
[45] Suvir Mirchandani, Suneel Belkhale, Joey Hejna, Evelyn Choi, Md Sazzad Islam, and Dorsa Sadigh. So you think you can scale up autonomous robot data collection? arXiv preprint arXiv:2411.018 2024.
[46] Utkarsh Aashu Mishra, Shangjie Xue, Yongxin Chen, and Danfei Xu. Generative skill chaining: Long-horizon skill planning with diffusion models. In Conference on Robot Learning, pages 2905-2925. PMLR, 2023.
[47] Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. arXiv preprint 6rX1v:2410.13816. 2024.
[48] Kensuke Nakamura, Lasse Peters, and Andrea Bajesy. Generalizing safety beyond collision- avoidance via latent-space reachability analysis. arXiv preprint wxxv:25502.00935. 2025.
[49] Jung Yeon Park and Lawson Wong. Robust imitation of few demonstrations with backwards model. Advances in Neural information Processing Systems, 15:19759-19772, 2022.
[50] Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. 2rXiv:2502.02538, 2025.
arXiv preprint
[51] Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
[52] Han Qi, Haocheng Yin, Yilun Du, and Heng Yang. Strengthening generative robot policies through predictive world modeling, arXiv preprint arXiv:2502.00622) 2025.
13

---

[53] Alfredo Reichlin, Giovanni Luca Marchetti, Hang Yin. Ali Ghadirzadeh, and Danica Kragic. states. In 2022 IEEE/RSJ Interna-
Back to the manifold: Recovering from lional Conference on Intelligent Robots and Systems (IROS), pages 8660-8666. IEFE, 2022.
[54] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-cooditioned imitation learning using score-basee diffusion policies. arXiv preprint arXiv:2304.02532, 2023.
[55] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[56] Zhanyi Sun, Yufei Wang, David Held, and Zackory Erickson. Force-constrained visual policy: Safe assit-as dressing via multi-modal sensing. IEEE Robotics and Automation Letters, 2024.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[58] Yanwei Wang. Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang. Yu- YuiWei Chao, Claudia Perez-D' Arpino, Dieter Fox, and Julie Shah. Inference-time policy steering through human interactions. arXiv preprint arXiv:2411.16622 2024.
(59] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Day- dreamer: World models for physical robot learning. In Conference on robot learning, pages 2226-2240. PMLR, 2023.
[60] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911. 1. 1361, 2019.
[61] Yilin Wu, Ran Tian, Gokul Swamy, and Andrea Bajesy. From foresight to forethought: Vlm-in- the-loop policy steering via latent alignment. arXiv preprint arXiv:2502.01828. 2025.
[62] Amber Xie, Oleh Rybkin, Dorsa Sadigh, and Chelsea Finn. Latent diffusion planning for imitation learning. arXiv preprint arXiv:2504. 16925, 2025.
[63] Xiaomeng Xu, Huy Ha, and Shuran Song. Dynamics-guid guidedd diffusion model for robot manipu- lator design. arXiv preprint arXiv:2402.15533: 2024.
[64] Tianhe Yu, Garrett Thomas, Lantao Yu. Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 14119-114442 2020.
[65] Allan Zhou, Moo Jin Kim, Lirui Wang, Pete Florence, and Chelsea Finn. Nerf the palm of your hand: Corrective augmentation for roboties via novel-view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17907-17917, 2023.
[66] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:241 1,04983, 2024.
[67] Guangyao Zhou, Sivaramakrishnan Swaminathan, Rajkumar Vasudeva Raju, J Swaroop Guntu- palli, Wolfgang Lehrach, Joseph Ortiz, Antoine Dedieu, Miguel Greeill,,,, and Kevin Murphy. Diffusion model predictive control. arXiv preprint arXiv:2410.05364 2024.
[68] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The evaluation section 4 both simulated environments and real robot demonstrates that our proposed method achieves higher samples efficiency and robustness.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are explicitly discussed in the conclusion section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not contribute theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experiment setups described in the experiment section 4 in the main paper and full detail in Appendix B. Code is provided in the supplemental and cleaned version will be released in the near future.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: For simulation experiments. the demonstration data is taken from public datasets ([41, 12]). The rollout data can be generated by executing checkpoints saved during expert policy training. For real robot experiment, the demonstration data is taken from [13]. The additional rollout data and play data will be released in the near future. A working copy of the code is provided in the supplemental material, and a cleaned version will be released in the near future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ http://uuuueeeeoeeuuusssssss/sse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n //ipp..cc. wwtppc/pccpgccciceccccooococcece httgg/udee////// hetggdde//oeeuuaiiiieeeseeseeeee /Codeeoumissstiieeeeeee ssionnoicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experiment settings are discussed in the experiments section 4 in the main paper and full details are in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For simulation experiments, we report success rate with std. deviation over policy checkpoints, For real robot experiments, we do not have intervals due to the intensive resource requirements. We note that it is common for experiments with real robots.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The full details of compute requirement provided in Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: btttps://neurippe http:////weuripscccpppliaaae Deuublic/Ethissuiieee ines?
Answer: [Yes]
Justification: We confirm that the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts of the paper in Appendix C.
18

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper uses public datasets and models. The learned policies and dynamics models do not have risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited all original owners of assets mentioned in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: pre-release anonymous version of the code with documentation is provided in the supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
The paper doesn't 't not use LLMS apart from assisting with visualization code and grammar checks.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https hupps://neurre htt/e/rrpp..ccccoeeeeeeeeeeeee for what should or should not be described.
21

---

A Additional Experiments and Results
A.1 Ablation: Gradient Guidance Hyperparameter Choice
To understand how the gradient guidance hyperparameters LPB affect the final task success rate, we run set of experiments scale 17 and the number of denoising steps that receive gradient guidance, Kguide. As shown in Figure 10 left, when is very small, the improvement over the base policy is minimal because the guidance signal is too weak. As 7) increases, the improvement becomes more pronounced; however, once reaches 0.3, performance drops slightly, indicating overshooting effect caused by overly strong guidance. Figure 10 right shows that LPB is generally robust to the choice of Kguide: even when gradients are applied only during the last 35 denoising steps. LPB still outperforms the base policy, demonstrating that the dynamics model can provide meaningful guidance even when the noisy action samples far from valid expert actions. Across all tasks, we find that injecting gradient guidance during the final 10 denoising steps sufficient for substantial performance boost.
the Transport task. We focus the two hyperparameters: the guidance
Ours
Base Policy
1.0
0.8
0.6
Success
0.4
0.2
0.0
0.01
0.05
0.1
0.2
0.3
0.4
10
15
20
25
30
(a) Guidance Scale
(b) Number of Denoising Steps
Receiving Gradient Guidance
Figure 10: (a) Performance under varying guidance scale. denoising steps that receive gradient guidance.
(b) Performance under varying numbers of
A.2 Ablation: OOD Threshold
We analyze the effect of varying the OOD threshold for the Transport and Tool-Hang tasks. From Figure 11, we observe that setting too high results in fewer corrections being triggered, leading to performance drops. Conversely, setting too low can cause the dynamics model to get stuck in local optima and stall task progress. This effect is particularly pronounced in the Tool Tool-Hang task, where the robot may get stuck at inserting the hook into the base.
1.0
0.8
Success Rate
0.6
0.4
0.2
0.0
1.6
2.2
2.8
3.4
4.0
0.8
1.4
1.7
(a) Transport
(b) Tool-Hang
Figure 11: Performance under varying OOD thresholds.
A.3
Ablation: Data Source for Dynamics Model Training
We evaluate different data sources for training the dynamics model on the Tool-Hang and Transport task. LPB leverages policy rollout data inexpensive and scalable way to improve the dynamics
22

---

model's generalization. but alternative sources are also available. We compare against two such baselines: (1) Expert Demonstration with Noise Injection. Gaussian noise injected into expert demonstrations with a probability of 0.3 to generate diverse, perturbed trajectories. (2) Epsilon- Greedy Exploration. A base policy is first trained on limited expert data, then rolled out from randomized initial configurations with Gaussian noise added to the output action at probability of 0.3 for exploration. As shown in Table 2. LPB, which relies on policy rollouts, achieves the highest success rate. We also note that rollout collection does not require manual tuning of noise magnitudes or injection probabilities, which simplifies the data collection procedure in practice. Moreover, naturally captures broad spectrum of data, spanning early exploratory trajectories to those generated by a converged policy.
Table 2: Success Rate with different data sources for dynamics model training
Policy Rollout
Noisy Demos
Epsilon-Greedy
Transport Tool-Hang
0.85 + 0,009
0.73 = 0.025
0.71 0.024
0.39 +: 0.009
0.30 = 0.031
0.30 +: 0.028
A.4 Ablation: Action Optimization
We ablate the action optimization strategies used in LPB on the Tool-Hang and Transport tasks, LPB adopts a classifier guidance-style approach for test-time action refinement. We compare this with two alternatives: (1) LPB-MPC, which uses Model Predictive Control with stochastic action sampling. and (2) LPB-GD, which directly optimizes actions via gradient descent. As shown in Table 3, LPB achieves the highest success rate. LPB-MPC also improves over the base policy, but not as much as LPB- likely because it relies on sampling, which can fail to capture rare but optimal actions if they are underrepresented in the action distribution. By contrast, LPB nudges each sampled action toward expert-like directions using a differentiable guidance signal, enabling exploration that remains on-manifold while still biasing toward high-value corrections. LPB-GD performs the worst, as its unconstrained optimization can produce actions that are out-of-distribttion for both the base policy and the dynamies model.
Table 3: Success Rate for different action optimization strategies
LPB (Ours)
LPB-MPC
LPB-GD
Tool-Hang Transport
0.39 0.009
0.32 + 0.028
0.26 : 0.028
0.85 +: 0.009
0.80 + 0.043
0.73 0.025
A.5 Ablation: Choice of Latent Representation
To study how the choice of latent space affects deviation detection and recovery toward expert states, we evaluate different latent representations on the Square and Transport tasks. We compare LPB, which uses the base policy's encoder, against two alternatives: (1) DINOv2 Encoder and (2) Encoder Trained with Reconstruction. For (1), we use a pretrained DINOV2 model as the observation encoder for the dynamics model; outputs 256 patch embeddings of dimension 384. This representation preserves fine-grained visual details, but may include task-irrelevant information, as it not directly trained with task supervision. For (2), we pretrain ResNet-18 as an autoencoder using a reconstruction loss and then freeze it to serve as the encoder for dynamics model training. As shown in Table 4, both the base policy's 's encoder and DINOv2 improve over the base policy alone, with the base policy's 's encoder outperforming DINOv2. This may be because DINOv2 encoder, trained in a task-agnostic manner, may focus on irrelevant features, while the base policy's encoder, trained via behavior cloning loss, is more aligned with task-relevant information. The reconstruutiion-asse encoder performs the worst, suggesting that the reconstruction objective alone does not produce a latent space suitable for effective action optimization.
23

---

Table 4: Success Rate with different latent space
Base Policy's Encoder
DINOv2
Encoder Trained with Reconstruction
Square
0.65 0.019
0.61 0.025
0.47 0.034
Transport
0.85 0.009
0.79 :: 0.025
0.68 1 0.043
B
Implementation Details
B.1
Implementation Details of Latent Policy Barrier
Base Policy. We adopt Diffusion Policy as the base policy in LPB due to its strong capability in modeling complex robot action distributions. It uses a ResNet-18 as the image encoder and a U-Net as the noise prediction network, with FiLM layers to condition the denoising process on both observation features and the current diffusion timestep. For all tasks, the base policy is trained using 20% the available expert demonstrations. Task-specific and shared hyperparameters provided in Table 5 and Table 6, respectively.
Dynamics Model. Once base-policy training passes an initial warm-up of to epochs, during which the policy remains highly exploratory, we begin saving checkpoints at fixed intervals. Concretely, every At epochs we store =n and roll it out for N complete episodes from randomly initialized configurations. All transitions, whether successful or not, are retained to train the dynamics model. This procedure continues until the final epoch (fiall The values of to. At, fimal, N, and the resulting total number of rollout trajectories for each simulated task are summarized Table 7.
The dynamics predictor fo is implemented as a Vision Transformer (ViT). It receives the concatenation of the latent observation token, the proprioception state token, and an action token, and predicts the next latent observation and proprioception state tokens. Training hyperparameters for fo are provided in Table 8.
Table 5: Simulation lask-dependett hyperparameters for base diffusion policy training.
Table 6: Shared hyperparameters for base dif- fusion policy training.
Training Epochs
Name
Value
Eny Name
Ta
T p
#Demo
To
Push-T
16
41
500
Image Size
140
Square
8
16
40
600
Crop Size
128
Tool-Hang
15
32
40
300
Batch Size
64
Transport
15
32
40
300
Learning Rate
1 x 10
Libero10
15
32
50
200
Diffusion Step
100
Table 7: Rollout trajectories collection schedule for
Table 8:
Hyperparameters for dynamics
simulation tasks
model training.
Eny Name
to
At
final
N
Total
Name
Value
Push-T
150
40
470
30
270
History Length
Square
70
50
470
30
270
Depth
6
Tool-Hang
70
50
270
30
150
Heads
16
Transport
70
50
270
30
150
MLP Dim
2048
Liberol0
40
40
160
50
200
Dropout
0.1
Batch Size
64
Learning Rate
5 x 10-4
Training Epoch
100
Test-time Optimization. At test time, we denoise actions with a DDPM scheduler. For each timestep t we first compute the latent OOD score; if the score is below the threshold т, we run standard denoising and execute the resulting action 4 If the score exceeds т, we apply latent steering during the final 10 denoising steps. Specifically, the current observation and the intermediate noisy action samples at denoising timestep k, A&. are fed to the dynamics model, which predicts the future latent state zeth-- We then measure the Fuclidean distance between z(+A and its nearest expert state latent space; this distance is back propaprop through the dynamics model, and the resulting gradient with
24

---

respect to the noisy action samples provides the guidance signal. The OOD threshold is chosen empirically by rolling out the final policy checkpoint, while the guidance scale r is selected via grid search. Both and for each task are listed in Table 9.
Table 9: Hyperparameters for test-time optimization
T
Push-T 0.05
3.2
Square 0.05
5
Tool-Hang
0.05
1,4
Transport
0.2
2.8
Libero10
0.2
1.1
Compute Resources. All simulated experiments are run on a single NVIDIA L40S GPU (46 GB VRAM). Base policies require roughly 24-48 h converge, while the dynamics models converge within 24 h. For each simulated task, we evaluate the final three checkpoints, each spaced 10 training epochs apart; the results reported in Table 1 averages over those three checkpoints. In the real- robot setting, the base policy is pretrained and thus incurs no additional training cost. The dynamics model is trained in parallel six NVIDIA L40S GPUs and converges in approximately h
B.2 Implementation Details of Baselines
Filtered BC. This baseline uses the same rollout data collected for LPB. We discard trajectories that do not meet the task-specific success criteria and merge the remaining successful rollouts with the original expert demonstrations. The diffusion policy then retrained from scratch on this augmented dataset.
CQL. We adopt the CQL implementation provided in the Robomimic codebase without modification.
CCIL. CCIL was originally designed for tasks with fow-dimensional state inputs. To extend it to high-dimensional image observations, cache latent representations produced by the base policy's encoder. Specifically, we first train a base diffusion policy Te on the limited expert demonstrations, identical to the setup used for LPB. We then encode every image observation in the expert dataset with aa's encoder and store the resulting latent transition pairs. CCIL's dynamics model is trained on these low-dimensional latents, after which CCIL's corrective-label generation procedure is applied to augment the expert data. Finally, we fine-tune on the augmented latent dataset. We use the authors' original implementation for all CCIL components.
B.3
Real Robot Experiment Setup
B.3.1
Cup Arrangement
Setup. We use 6-DoF ARXS robot arm with 3D-printed soft compliant fingers and a wrist-mounted GoPro camera, mirroring the sensor configuration of the original training data. Low-level controller provided by the Universal Manipulation Interface (UMI) stack ported to the ARX5 platform https://github.com// http:real-stanford/umiiie
The cup arrangement task consists of three sequential stages: (i) rotate the cup so its handle points right, (ii) grasp and lift the cup, and (iii) place upright on saucer.
Pre-trained Base Policy. We start from the publicly released diffusion policy checkpoint in the Universal Manipulation Interface repository https://ithub.com/reall stanford/universal manipulation interface. The base policy was trained on 1447 in in-the-wild demonstrations of the cup arrangement task collected in diverse environments. The test environment not included the training data. At each timestep, the policy receives a single RGB observation plus the past two proprioceptive states and predicts 16-step action sequence; the first 12 actions are executed each control cycle.
Dynamics Model Training, To collect training data for the dynamics model, we roll out the pre- trained policy from deliberately outoo--iistriuttin initial poses, gathering 80 trajectories. Because intermediate policy checkpoints are unavailable, we augment the dataset with additional 40 human play trajectories recorded via the handheld UMI device. These play trajectories are intended to broaden Mate-action coverage rather than solve the task, for example, by sweeping the gripper
25

---

through random motions to capture wide-angle views. These 120 unlabeled trajectories, together with the original demonstrations, are used train the dynamics model.
Evaluation Protocol. We evaluate two initial-poss regimes: (i) in-distribution initial poses and (ii) out-of-distribution initial poses. For each group conduct 20 trials with both the base policy and LPB, using identical initial robot and object poses for fair comparison. During deployment we denoise actions with DDIM scheduler (16 diffusion steps) and apply gradient guidance during the final five steps.
B.3.2 Belt Assembly
Setup. We use 6-DoF UR5 robot arm with 3D-printed soft compliant fingers and wrist-mounted OAK-D camera. Low-level controller is provided by https://gttub..com/ hou/hardware interfaces.
Base Policy Training. We collect 200 expert demonstrations, each performed with slight variations in the board position while keeping the initial robot pose fixed. We train the base policy for 800 epochs. At each timestep, the policy receives a single RGB observation plus the past two proprioceptive states and predicts 32-step action sequence; the first 16 actions are executed each control cycle.
Dynamies Model Training. To train the dynamics model, we collect 400 rollout trajectories executed under randomly initialized board positions and initial robot poses. We use hyperparameters to 200 At 200, finall = 800, and N 100. We train the dynamics model with combined 600 trajectories.
Evaluation Protocol. We perform 40 rollouts for both the base policy and LPB, varying the initial robot pose and board position. All evaluations use the same initial robot and object poses for fair comparison. During deployment we denoise actions with DDIM scheduler (16 diffusion steps) and apply gradient guidance during the final five steps.
C Broader Impact
Our work contributes to improving the robustness of visuomotor policies robotic manipulation settings by introducing test-time optimization framework that leverages learned dynamics and latent-space guidance. This has the potential to reduce failure rates in deployment, especially in out-of-distribution scenarios, which is critical for the safe and reliable operation of robots in human- centered environments. However, as with any data-driven system, care must taken to ensure that failure modes thoroughly evaluated before real-world deployment. Future extensions could investigate formal guarantees for test-time optimization mechanisms.
26


Reviews:
[{"review_id": "RZpaL9qqa1", "reviewer": "Reviewer_9nMP", "summary": "This paper introduces Latent Policy Barrier (LPB), a novel framework for robust visuomotor policy learning via behavior cloning. LPB decouples precise expert imitation from out-of-distribution (OOD) recovery by combining: (1) a base diffusion policy trained exclusively on high-quality expert demonstrations to ensure task performance, and (2) a visual latent dynamics model trained on both expert and automatically collected rollout data to detect and correct deviations at inference time. At test time, LPB computes a latent OOD score-the $l_2$ distance between the current latent embedding and its nearest expert neighbor-and, if it exceeds a threshold, performs gradient-based steering in latent space to pull predicted future states back toward the expert manifold. Experiments on four simulated robotic manipulation tasks (Push-T, Square, Tool-Hang, Transport) and a real-world cup-arrangement task demonstrate that LPB achieves up to a 34% success-rate improvement over strong baselines under limited expert data and maintains robustness under action noise and significant distribution shifts.", "questions": "1. How sensitive is LPB's performance to the choice of latent-OOD threshold $\\tau$? Can you provide guidance or automate its selection?\n\n2. What is the real-time computational cost of LPB's gradient-based steering during inference, and how might this scale with higher action dimensions or longer denoising chains?\n\n3. Storing all expert latent embeddings for nearest-neighbor search can be memory-intensive. Have you considered approximate methods (e.g., k-d trees or learned quantization) and their impact on robustness?\n\n4. LPB focuses on short-term, local corrections. How might the dynamics model be extended for long-horizon planning to recover from more severe drifts?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Quality\n\n**Strengths**: the empirical evaluation is thorough, covering both simulated and real-robot benchmarks. LPB consistently outperforms baselines-including Behavior Cloning variants, Filtered BC, CCIL, and CQL-especially in low-data regimes (20% demos) and under inference-time perturbations (up to 0.4 action noise).\n\n**Weaknesses**: the paper lacks theoretical guarantees on convergence or safety margins. Evaluation is limited to a narrow set of manipulation tasks; generalization to other domains (e.g., mobile navigation) remains untested.\n\n# Clarity\n\n**Strengths**: the conceptual analogy to Control Barrier Functions and the separation of training vs. inference components are well motivated. Figures (e.g., rollout comparisons in Figure 5) and Algorithm 1 clearly illustrate the LPB pipeline.\n\n**Weaknesses**: Key hyperparameters, such as the OOD threshold $\\tau$, guidance scale $\\eta$, and selection of nearest-neighbor search strategy, are deferred to the appendix, which may hinder immediate reproducibility. The presentation of computational overhead at inference (gradient-guided denoising) could be more detailed.\n\n# Significance\n\n**Strengths**: Covariate shift is a central challenge in imitation-based visuomotor control. LPB offers a plug-and-play approach that enhances robustness without additional human labeling or policy fine-tuning, potentially benefiting a broad class of pretrained vision-based policies.\n\n**Weaknesses**: While impactful within robotic manipulation, the approach's scalability to environments with very high-dimensional observation spaces or multi-agent settings is unclear.\n\n# Originality\n\n**Strengths**: LPB is the first to frame expert demonstrations as an implicit latent barrier and to perform gradient-based inference steering in a learned visual latent space for behavior cloning policies. This unifies ideas from diffusion-policy classifier guidance and model-predictive refinement in a novel way.\n\n**Weaknesses**: Related works on guided denoising and model-based policy refinement exist; the incremental novelty lies primarily in applying these to latent-space neighbor distances rather than explicit reward signals.", "was_revised": true, "final_justification": "Rebuttal of the authors (https://openreview.net/forum?id=FUd016XD4d&noteId=VuhVVwxj5q) has address all my concerns about this work. (Thanks for your detailed clarification. Good work!) I decided to change my rating to 5: Accept.", "rebuttal": "We thank Reviewer 9nMP for their time and expertise in reviewing our submission. Their insightful comments are valuable in helping us improve the quality of our work.\n> *“Evaluation is limited to a narrow set of manipulation tasks; generalization to other domains (e.g., mobile navigation) remains untested.”*\n\nWe thank the reviewer for this comment. Our work focuses on long-horizon, precise manipulation tasks where covariate shifts can cause unrecoverable failures. To further test generalization within this domain, we evaluate on Libero-10 [1], a multi-task, language-conditioned manipulation benchmark that contains 10 different tasks. For each task, Libero-10 provides 50 expert demonstration trajectories. We use those $50\\times 10$ expert trajectories to train the base policy. For each task, we collect 200 rollout trajectories. We use $200\\times 10$ rollout trajectories and the 500 expert trajectories to train the dynamics model. Both the base policy and the dynamics model are multi-task with an additional shared language encoder. We perform 50 rollouts per task across ten tasks and report the average success rate.\n\n| Method        | Libero-10 Success Rate |\n|---------------|------------------------|\n| Expert BC     | 0.65 ± 0.009           |\n| Mixed BC      | 0.50 ± 0.017           |\n| Filtered BC   | 0.71 ± 0.006           |\n| CCIL          | 0.61 ± 0.026           |\n| Ours (LPB)    | **0.75 ± 0.038**       |\n\nWe also conduct real-world evaluation on the **belt assembly task** from the NIST Task Board. We collect 200 expert trajectories to train the base policy and 400 rollout trajectories to train the dynamics model. For both LPB and the base policy, we perform 40 rollouts. LPB improves the success rate from **0.55** to **0.75**. Please see the project website for rollouts.\n\n\n> *“Key hyperparameters, such as the OOD threshold, guidance scale, and selection of nearest-neighbor search strategy, are deferred to the appendix, which may hinder immediate reproducibility. How sensitive is LPB's performance to the choice of latent-OOD threshold? Can you provide guidance or automate its selection?”*\n\nWe thank the reviewer for pointing out this presentation issue. We agree that presenting the hyperparameter design choice more explicitly in the main paper would be more informative for the readers, and **we will include them in the camera-ready version**.\n\nWe analyze the effect of varying the OOD threshold $\\\\tau$ for the Transport and Tool-Hang tasks. We observe that setting $\\\\tau$ too high results in fewer corrections being triggered, leading to performance drops. Conversely, setting $\\\\tau$ too low can cause the dynamics model to get stuck in local optima and stall task progress. This effect is particularly pronounced in the Tool-Hang task, where the robot may get stuck at inserting the hook into the base.\n\nAs described in the appendix, we select the threshold by first rolling out the base policy to identify the minimum OOD score associated with failure, then sweeping $\\\\tau$ below that value during evaluation with our method.\n\n**Transport Task:**\n\n| $\\\\tau$ | Success Rate        |\n|------------------------|---------------------|\n| 1.6                    | 0.78 ± 0.028        |\n| 2.2                    | 0.82 ± 0.025        |\n| 2.8                    | 0.85 ± 0.009        |\n| 3.4                    | 0.76 ± 0.033        |\n| 4.0                    | 0.67 ± 0.038        |\n\n**Tool-Hang Task:**\n\n| $\\\\tau$ | Success Rate        |\n|------------------------|---------------------|\n| 0.8                    | 0.28 ± 0.028        |\n| 1.1                    | 0.33 ± 0.034        |\n| 1.4                    | 0.39 ± 0.009        |\n| 1.7                    | 0.34 ± 0.016        |\n| 2.0                    | 0.29 ± 0.034        |\n\n> *“What is the real-time computational cost of LPB's gradient-based steering during inference, and how might this scale with higher action dimensions or longer denoising chains?”*\n\nWe thank the reviewer for this important question. Below we provide detailed measurements of the real-time computational cost of LPB's gradient-based steering during inference on the real world cup arrangement task. **We will update the paper to make this more clear.**\n\nWe predict 16 action steps and execute the first 12. On a GeForce RTX 3080 GPU, it takes around 5 minutes to load the expert dataset and caching them to latent representations. The base policy has an initial overhead of 0.36 seconds and 2.0 GB GPU usage. After warm-up, inference runs at **77 ms** per chunk with peak usage of **1.6 GB**. LPB incurs an initial overhead of 0.87 seconds and 3.3 GB usage, and after warm-up, inference runs at **420 ms** per chunk with peak memory usage of **3.0 GB**. \n\nIn summary, LPB’s inference cost scales reasonably with an action dimension of 10 (3D translation, 6D rotation, and 1D gripper control) and a horizon of 16. For quasi-static tasks, this cost is not a bottleneck in practice. However, the computational cost would increase with longer denoising chains due to additional gradient steps during backpropagation. \n\n> *\"Storing all expert latent embeddings for nearest-neighbor search can be memory-intensive. Have you considered approximate methods (e.g., k-d trees or learned quantization) and their impact on robustness?\"*\n\nWe thank the reviewer for the insightful question. Indeed, storing all expert latents can be memory intensive. In this work, the out-of-distribution detection is performed in the most straightforward way - latent l2 distance to the nearest expert latent. Exploring more scalable alternatives, such as approximate nearest-neighbor search, is definitely a promising direction that could reduce memory and compute overhead.\n\n> *\"LPB focuses on short-term, local corrections. How might the dynamics model be extended for long-horizon planning to recover from more severe drifts?\"*\n\nWe thank the reviewer for the insightful question. As noted, LPB is most effective for short-term, local corrections. Extending it to longer horizons would require a dynamics model with stronger generalization and broader data coverage. In principle, sampling-based approaches combined with a reliable dynamics model can support long-horizon recovery [2,3], and integrating such long-term planning capabilities is an interesting direction for future work.\n\n> *\"the paper lacks theoretical guarantees on convergence or safety margins\"*\n\nWe thank the reviewer for the thoughtful question. We fully acknowledge that our paper does not provide theoretical guarantees. Providing theoretical guarantees for learning based systems is still an open challenge [4,5] and is an exciting future direction to our paper. Despite lack of theoretical guarantees, our extensive experiments have shown the practical value of gradient-based correction with a dynamics model. \n\n> *\"While impactful within robotic manipulation, the approach's scalability to environments with very high-dimensional observation spaces or multi-agent settings is unclear.\"*\n\nWe thank the reviewer for this thoughtful comment. Our method is explicitly designed for high-dimensional observation spaces and all tasks in our evaluation rely on raw RGB images (some with more than one camera view) as input. We demonstrate strong empirical performance across a diverse set of vision-based, long-horizon manipulation tasks, indicating that our approach scales well in high-dimensional observation domains.\n\nRegarding multi-agent settings, our method is not directly formulated to handle joint policies or shared dynamics between agents. LPB assumes a single-agent setting where the dynamics model and OOD guidance operate on individual observations and actions. Extending this framework to multi-agent scenarios is an exciting future direction and would require rethinking several core components, such as modeling joint dynamics, defining agent-specific guidance objectives, and addressing non-stationarity introduced by other agents. We view this as a non-trivial and interesting research direction but leave it outside the scope of the current work.\n\n\n### References\n[1] Liu, Bo, et al. *Libero: Benchmarking knowledge transfer for lifelong robot learning.* Advances in Neural Information Processing Systems 36 (2023): 44776-44791.\n\n[2] Feng, Yunhai, et al. *Finetuning offline world models in the real world.* arXiv preprint arXiv:2310.16029 (2023).\n\n[3] Zhou, Gaoyue, et al. *Dino-wm: World models on pre-trained visual features enable zero-shot planning.* arXiv preprint arXiv:2411.04983 (2024).\n\n[4] Dixit, Anushri, et al. *Perceive with confidence: Statistical safety assurances for navigation with learning-based perception.* 8th Annual Conference on Robot Learning. 2024.\n\n[5] Majumdar, Anirudha, Alec Farid, and Anoopkumar Sonar. *PAC-Bayes control: learning policies that provably generalize to novel environments.* The International Journal of Robotics Research 40.2–3 (2021): 574–593."}, {"review_id": "78CYOgOH5L", "reviewer": "Reviewer_76MU", "summary": "This paper presents a new approach to improve OOD recovery of the diffusion policy. The authors achieve this by learning a separate dynamics model and using classifier guidance to guide the generation process to in-distribution states to encourage recovery. The result is verified on well-designed experiments both in simulation and in the real world.", "questions": "1. How does the method compare to using a two-stage framework? For example, how about using planners such as MPPI with the learned dynamics model to push back into the expert distribution first, then utilize the expert diffusion model to do inference? Is this the MPC baseline in your appendix?\n2. How much overhead (both memory for loading the expert dataset latents and computation for calling the learned dynamics model and calculating the gradient-based guidance) does the method add to the original diffusion policy inference pipeline?\n3. When expert data size is scaled up, it seems both the proposed approach and the base expert policy lead to similar performance. I am curious what is the trend of the correction rate (i.e., percentage of times that the guidance is activated) as one increase the amount of expert data.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Presented a novel approach to recover from OOD situations in diffusion policies.\n2. Well-designed experiments that showcase the effectiveness of their work.\n3. Very effective improvements in the low (expert) data situations compared to previous works.\n4. Overall clearly written and presented paper.\n\nWeaknesses:\n\n1. Classifier-based guidance in diffusion policy has been widely used already.\n2. The proposed method claims to be Plug-And-Play. However, it still requires non-trivial data collection and training for the dynamics model.", "was_revised": true, "final_justification": "My final recommendation for this paper is 5 (Accept). The methods are well explained, and the experiments are well documented and demonstrate the good performance of their method in OoD recovery. The rebuttal also clarified several important questions I had on the baseline design and speed considerations. I am not fully convinced by their explanations on the claim that the method is fully plug-and-play. I think training a separate dynamis model is not a small task: it requires extra data gathering, network design, and hyperparameter tuning. Overall, I think the proposed method is a good improvement upon diffusion policies for OoD recovery.", "rebuttal": "We thank Reviewer 76MU for their time and expertise in reviewing our submission. Their insightful comments are valuable in helping us improve the quality of our work.\n> *“How does the method compare to using a two-stage framework? For example, how about using planners such as MPPI with the learned dynamics model to push back into the expert distribution first, then utilize the expert diffusion model to do inference? Is this the MPC baseline in your appendix?”*\n\nWe thank the reviewer for this important question. The MPC baseline in our appendix corresponds to a two-stage framework where action initializations proposed by the expert diffusion policy are refined using MPC with the learned dynamics model. We initially experimented with sampling actions from a Gaussian distribution for MPPI and MPC, but found that this leads to poor performance, as most samples fall out of distribution for both the dynamics model and the base policy, resulting in unreliable predictions. Using action proposals from the expert policy as initialization significantly improves stability and performance. While this MPC baseline does improve upon the base policy, we found it to be less effective than our gradient-based LPB correction, which offers more direct and fine-grained alignment with the expert manifold.\n\n> *“When expert data size is scaled up, it seems both the proposed approach and the base expert policy lead to similar performance. I am curious what is the trend of the correction rate (i.e., percentage of times that the guidance is activated) as one increase the amount of expert data.”*\n\nWe thank the reviewer for this thoughtful question. Indeed, as the size of the expert dataset increases, the base policy becomes more reliable, and the benefit of LPB begins to saturate. Correspondingly, we observe that the frequency of invoking gradient-guided correction decreases with more expert data. Below, we report the correction rate across different fractions of the demonstration set. As the coverage of demonstration data grows, the latent OOD score will decrease, causing the correction to be invoked less often. LPB is most beneficial in low-data regimes and naturally defers to the base policy when sufficient coverage is achieved. **We will include this discussion in the final paper**.\n\n**Tool-Hang Task:**\n| Demo Data Perc.      | 20%  | 40%  | 60%  | 80%  | 100% |\n|----------------|------|------|------|------|-------|\n| Correction Perc. | 77%  | 62%  | 55%  | 39%  | 35%  |\n\n> *“How much overhead (both memory for loading the expert dataset latents and computation for calling the learned dynamics model and calculating the gradient-based guidance) does the method add to the original diffusion policy inference pipeline?”*\n\nWe thank the reviewer for this important question. Below we provide detailed measurements of the real-time computational cost of LPB's gradient-based steering during inference on the real world cup arrangement task. **We will update the paper to make this more clear.**\n\nWe predict 16 action steps and execute the first 12. On a GeForce RTX 3080 GPU, it takes around 5 minutes to load the expert dataset and caching them to latent representations. The base policy has an initial overhead of 0.36 seconds and 2.0 GB GPU usage. After warm-up, inference runs at **77 ms** per chunk with peak usage of **1.6 GB**. LPB incurs an initial overhead of 0.87 seconds and 3.3 GB usage, and after warm-up, inference runs at **420 ms** per chunk with peak memory usage of **3.0 GB**.\n\n> *“The proposed method claims to be Plug-And-Play. However, it still requires non-trivial data collection and training for the dynamics model.”*\n\nWe thank the reviewer for this insightful comment. We fully acknowledge that training a dynamics model requires additional rollout data, which can be non-trivial in real-world settings. However, we emphasize that our method remains plug-and-play in the sense that it does not require task-specific supervision, reward engineering, or labeled failure cases, only passive rollout data, which can be collected in an unsupervised manner. This makes LPB broadly applicable and easy to integrate in practice."}, {"review_id": "wDdCB69W4m", "reviewer": "Reviewer_jkE6", "summary": "This paper aims to address the covariate shift problem that is prevalent in visuomotor policies trained by behavior cloning (BC). The authors propose a new framework called Latent Policy Barrier (LPB). The core idea is to regard the embedding distribution of expert demonstrations in the latent space as an implicit \"safe\" region, and inspired by the control barrier function (CBF), ensure that the agent does not deviate from this region during policy execution. A key design of this framework is to decouple the two core goals in imitation learning - accurate imitation and out-of-distribution (OOD) recovery. It uses a base policy (the diffusion model in this paper) trained only on high-quality expert data to ensure the accuracy of imitation, and uses a dynamics model trained on expert data and suboptimal policy rollout data to learn how to recover from deviations. In the inference stage, the dynamics model \"guides\" the actions of the base policy through online optimization so that its predicted future latent states remain within the manifold of the expert data distribution, thereby improving the robustness of the policy without sacrificing the quality of imitation. Experiments on simulation and real robots demonstrate the effectiveness of the proposed method.", "questions": "On the quality of the latent space: The effectiveness of the entire LPB framework relies on a high-quality latent space. Can you discuss under what circumstances the visual encoder learned by the BC strategy may not form a \"good\" latent space (i.e., a space in which distances do not effectively reflect the degree of OOD)? For example, how does the performance of LPB change in the presence of a large amount of visual distractors that are irrelevant to the task?\n\nOn the sensitivity of hyperparameters: The setting of the threshold $\\tau is crucial for the algorithm to trigger. How should this value be set in practice? Is it a sensitive parameter that needs to be fine-tuned for each task, or is it robust over a wide range? Can you provide more sensitivity analysis on \\tau and the guidance strength \\eta$?\n\nComparison with explicit recovery strategies: Your method can be viewed as an online, implicit recovery strategy. Another idea is to explicitly train a separate recovery strategy using the same rollout data. Can you fundamentally analyze the advantages and disadvantages of your proposed \"online guidance\" method compared with the \"offline training recovery strategy\" method and what are the applicable scenarios?", "limitations": "Yes. The authors clearly discuss the limitations of the current work in the conclusion section, including that the current method mainly deals with local deviations, the dynamics model is task-specific, and the method requires access to expert demonstration data to define the inner manifold of the distribution. The discussion is sufficient and honest.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe core idea of ​​the paper cleverly generalizes and applies the concept of \"control barrier function\" (CBF) in control theory to data-driven imitation learning, and proposes to use \"latent space distribution of expert data\" as an implicit barrier, which is a novel and insightful analogy and framework.\n\nThe paper contains in-depth qualitative and quantitative analysis. For example, the analysis in Figure 5 associates the latent space OOD score with the actual task failure moment, providing strong intuitive evidence for the core mechanism of the LPB method. Its ability to improve the \"out-of-the-box\" pre-trained model is verified on a real robot, which enhances the credibility and practical value of the paper.\n\nWeaknesses:\n\nStrictly speaking, the various components of the algorithm (diffusion strategy, latent space dynamics model, gradient-based guidance) are all existing technologies. The originality of this work is mainly reflected in the combination of these components and the guiding ideas behind them (CBF analogy and decoupling principle).\n\nThe method introduces some hyperparameters that need to be adjusted, such as the threshold $\\tau of the latent space OOD score and the guidance strength \\eta$. The sensitivity analysis of these hyperparameters is not sufficient in the main text of the paper, which may affect its reproducibility in different tasks.\n\nThe guidance mechanism during inference (Formula 4) combines the ideas of diffusion model denoising and gradient optimization. It is conceptually dense and may require readers to have a certain understanding of the relevant background to fully digest it.", "was_revised": true, "final_justification": "my decision is keeping my score", "rebuttal": "We thank Reviewer jkE6 for their time and expertise in reviewing our submission. Their insightful comments are valuable in helping us improve the quality of our work.\n> *“The method introduces some hyperparameters that need to be adjusted, such as the threshold and guidance strengths. How should $\\tau$ be set in practice? Is it a sensitive parameter that needs to be fine-tuned for each task, or is it robust over a wide range? The sensitivity analysis of these hyperparameters is not sufficient in the main text of the paper, which may affect its reproducibility in different tasks.”*\n\nWe thank the reviewer for pointing out this important issue. We agree that analyzing the hyperparameter design choice more explicitly in the main paper would be more informative for the readers, and **we will update the camera-ready version to include them.** \n\nWe analyze the effect of varying the OOD threshold $\\tau$ for the Transport and Tool-Hang tasks. The performance is robust to $\\tau$ within a small range. We observe that setting $\\tau$ too high results in fewer corrections being triggered, leading to performance drops. Conversely, setting $\\tau$ too low can cause the dynamics model to get stuck in local optima and stall task progress. This effect is particularly pronounced in the Tool-Hang task, where the robot may get stuck at inserting the hook into the base.\n\nAs described in the appendix, we select the threshold by first rolling out the base policy to identify the minimum OOD score associated with failure, then sweeping \\tau below that value during evaluation with our method. The ablation experiment on guidance scale already in Appendix A.1. The optimal guidance scale is found by grid search. \n\n**Transport Task:**\n\n| $\\\\tau$ | Success Rate        |\n|------------------------|---------------------|\n| 1.6                    | 0.78 ± 0.028        |\n| 2.2                    | 0.82 ± 0.025        |\n| 2.8                    | 0.85 ± 0.009        |\n| 3.4                    | 0.76 ± 0.033        |\n| 4.0                    | 0.67 ± 0.038        |\n\n**Tool-Hang Task:**\n\n| $\\\\tau$ | Success Rate        |\n|------------------------|---------------------|\n| 0.8                    | 0.28 ± 0.028        |\n| 1.1                    | 0.33 ± 0.034        |\n| 1.4                    | 0.39 ± 0.009        |\n| 1.7                    | 0.34 ± 0.016        |\n| 2.0                    | 0.29 ± 0.034        |\n\n> *“On the quality of the latent space: The effectiveness of the entire LPB framework relies on a high-quality latent space. Can you discuss under what circumstances the visual encoder learned by the BC strategy may not form a \"good\" latent space (i.e., a space in which distances do not effectively reflect the degree of OOD)? For example, how does the performance of LPB change in the presence of a large amount of visual distractors that are irrelevant to the task.”*\n\nWe thank the reviewer for this insightful question. Indeed, the effectiveness of LPB depends on a latent space where distances reflect task-relevant similarity. While learning the right representation for planning and control is still an open research question, our motivation for using the encoder trained with BC loss is that it encourages control-aware representations by aligning observations with expert actions. Below we discuss two scenarios in which the encoder learned in this way may sometimes struggle: (1) visually cluttered environments, where irrelevant features can have a dominating effect, and (2) limited visual diversity in demonstration, which can lead to overfitting and poor generalization.\n\nFor (1), in our real-world setup, we use the pretrained in-the-wild policy encoder. The encoder and policy were trained on visually diverse images with task-irrelevant features. As shown in Fig. 7, the OOD score and nearest-neighbor matches are noisier in these settings, reflecting challenges in the latent space. For (2), in simulated tasks, where backgrounds are cleaner, we still observe occasional overfitting to visual features such as the robot’s appearance, which is present in the observation space.\n\nFuture work on improving the latent space includes incorporating contrastive learning [1,2] or auxiliary objectives [3] that explicitly promote control-aware latent structure.\n\n> *“Comparison with explicit recovery strategies: Your method can be viewed as an online, implicit recovery strategy. Another idea is to explicitly train a separate recovery strategy using the same rollout data. Can you fundamentally analyze the advantages and disadvantages of your proposed \"online guidance\" method compared with the \"offline training recovery strategy\" method and what are the applicable scenarios?”*\n\nWe thank the reviewer for this insightful question. Explicitly training a recovery policy from offline data is often challenging, as it typically requires access to task rewards [4] or prior assumptions about tasks such as equivariance [5]. In contrast, our proposed method does not require such additional assumptions. In addition, our method provides online correction through gradient-based action optimization guided by a learned dynamics model, conceptually similar to classifier guidance in diffusion models. This ensures that the corrected actions remain close to the expert manifold, unlike a separately trained recovery policy, which may drift outside the expert distribution and introduce instability.\n\nThat said, offline recovery policies may be more sample-efficient when recovery behaviors are well-represented and consistent in the offline dataset. In comparison, LPB incurs higher test-time computational cost due to online optimization. We believe the two approaches are complementary: LPB offers adaptability and robustness to pre-trained base policies in novel online situations, while explicit recovery policies are well-suited for structured environments with known failure modes.\n\n### References\n\n[1] Laskin, Michael, Aravind Srinivas, and Pieter Abbeel. *Curl: Contrastive unsupervised representations for reinforcement learning.* International conference on machine learning. PMLR, 2020.\n\n[2] Yan, Wilson, et al. *Learning predictive representations for deformable objects using contrastive estimation.* Conference on Robot Learning. PMLR, 2021.\n\n[3] Hansen, Nicklas, et al. *Self-supervised policy adaptation during deployment.* arXiv preprint arXiv:2007.04309 (2020).\n\n[4] Thananjeyan, Brijen, et al. *Recovery rl: Safe reinforcement learning with learned recovery zones.* IEEE Robotics and Automation Letters 6.3 (2021): 4915-4922.\n\n[5] Reichlin, Alfredo, et al. *Back to the manifold: Recovering from out-of-distribution states.* 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022."}, {"review_id": "tit6FHgc8i", "reviewer": "Reviewer_hbP6", "summary": "This paper introduces an approach to keep diffusion policies in distribution during execution. The core idea is to learn a local dynamics model by gathering data using policy rollouts from a trained policy (or checkpoints from the policy during training). The trained dynamics model is then used as a form of classifier guidance to keep the observed latents close to a set of latent states gathered from an expert's demonstration. This helps keep the policy close to the support of the expert training data, and improves robustness. Results show this reduces the amount of data required for robust model training. Experiments are conducted in simulation and on real robots.", "questions": "In Fig 4c. How is the impact of rollout data computed and contribution to success rates? Eg. Where does the 0.4 number come from at 300 trajectories.\n\nI may have missed this, but what sort of coverage do you need for the expert latents? How big is the buffer, and do you do anything special to ensure they adequately cover the space of interest?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe idea is simple, and results show that it is robust to noise, and policies require fewer expert demonstrations to be effective.\n\nI think it is a sensible approach, and like that it is modular and can be attached to a pre-trained policy. \n\nThis meets my internal criteria of would I send it to a PhD student.\n\nThe paper is well written, and experiments are nicely executed.\n\nWeaknesses:\n\nThe data collection process for dynamics means that only a small local model can be trained, this means that the approach may struggle in complex environments with greater environmental dynamics and for longer horizon tasks. This is discussed in the limitations.\n\nMany of the demonstration tasks show the approach successfully driving the robot to the expert data distribution in areas like free space motion, but I would have liked to see an example of this happening in a more complex interactive setting like contact, where classical planning techniques couldn't work. \n\nThis is nitpicking, but the distance used to determine whether something is out of distribution is an l2 distance between the nearest expert latent. Given that the policy being corrected is a probabilistic one, it seems that a less heuristic approach would be to directly determine where a latent lies in distribution or not, although this may require solving the flow equation.", "was_revised": true, "final_justification": "The authors included additional experiments that strengthen and already clear paper. I think this should be accepted.", "rebuttal": "We thank Reviewer hbP6 for their time and expertise in reviewing our submission. Their insightful comments are valuable in helping us improve the quality of our work.\n> *“Many of the demonstration tasks show the approach successfully driving the robot to the expert data distribution in areas like free space motion, but I would have liked to see an example of this happening in a more complex interactive setting like contact, where classical planning techniques couldn't work.”*\n\nWe thank the reviewer for this insightful question. To demonstrate LPB’s effectiveness in contact-rich settings, we evaluate it on an additional simulation benchmark and a real-world task.\n\nIn simulation, we evaluate on Libero-10 [1], a multi-task, language-conditioned manipulation benchmark that contains 10 different tasks. For each task, Libero-10 provides 50 expert demonstration trajectories. We use those $50\\times 10$ expert trajectories to train the base policy. For each task, we collect 200 rollout trajectories. We use $200\\times 10$ rollout trajectories and the 500 expert trajectories to train the dynamics model. Both the base policy and the dynamics model are multi-task with an additional shared language encoder. We perform 50 rollouts per task across ten tasks and report the average success rate.\n\n| Method        | Libero-10 Success Rate |\n|---------------|------------------------|\n| Expert BC     | 0.65 ± 0.009           |\n| Mixed BC      | 0.50 ± 0.017           |\n| Filtered BC   | 0.71 ± 0.006           |\n| CCIL          | 0.61 ± 0.026           |\n| Ours (LPB)    | **0.75 ± 0.038**       |\n\nFor the real-world evaluation, we consider the **belt assembly task** from the NIST task board, which involves hooking a rubber band onto two pulleys, a contact-intensive manipulation task. We collect 200 trajectories to train the base policy and 400 rollout trajectories to train the dynamics model. We conduct 40 rollouts using both the base policy and LPB. LPB boosts the success rate from **0.55 to 0.75**. Please refer to the original project website for rollout videos.\n\n> *“This is nitpicking, but the distance used to determine whether something is out of distribution is an l2 distance between the nearest expert latent. Given that the policy being corrected is a probabilistic one, it seems that a less heuristic approach would be to directly determine where a latent lies in distribution or not, although this may require solving the flow equation.”*\n\nWe thank the reviewer for the insightful question. We agree that our use of L2 distance to the nearest expert latent is a simple heuristic. While effective in practice, it does not fully leverage the structure of the underlying diffusion model.\n\nWe fully agree that more principled approaches to uncertainty quantification that are grounded in the generative structure of diffusion or flow-based models could offer a richer and more theoretically motivated signal. Prior work has proposed various OOD detection methods specifically tailored to generative models. Interestingly, [2] highlights that likelihood alone is not a reliable indicator for OOD detection. For diffusion models in particular, recent approaches include: using reconstruction consistency across denoising steps [3], modeling the distribution over predicted diffusion timesteps [4], using curvature along the diffusion paths [5], and leveraging approximate score matching loss computed at test time [6]. While these methods can be more computationally involved, they hold strong potential for improving both the interpretability and robustness of online guidance in frameworks like ours.\n\nWe consider this an exciting direction for future work and appreciate the reviewer’s suggestion. **We will include this discussion in the final paper.**\n\n> *“In Fig 4c. How is the impact of rollout data computed and contribution to success rates? Eg. Where does the 0.4 number come from at 300 trajectories?”*\n\nWe thank the reviewer for raising this question. In Fig. 4c, we evaluate the same base policy while varying the amount of rollout data used to train the dynamics model. The curve shows how the success rate of LPB-guided rollouts improves as the dynamics model becomes more accurate and generalizable with more rollout data. The success rate at each point is computed by averaging the task success across 50 randomized test configurations and random seeds. A success rate of 0.4 simply means 20 out of 50 rollouts are successful.\n\nThe improvement with increasing rollout data reflects better coverage of the state-action space, which enables more reliable gradient-based corrections. However, we observe that the benefit plateaus around 300 trajectories, likely because the additional rollouts no longer significantly expand the distribution of reachable states encountered at test time. **We will make this more clear in the final paper**. \n\n> *“What sort of coverage do you need for the expert latents? How big is the buffer, and do you do anything special to ensure they adequately cover the space of interest?”*\n\nWe thank the reviewer for this insightful question. Our method does not assume any explicit coverage requirement over the expert latents. The key requirement is that the expert demonstrations are optimal, high-quality, and consistent, so that the base policy trained on them performs precise imitation within the expert distribution. As shown in Fig. 4b, 100% demonstration corresponds to 200 trajectories. We found that even a modestly sized expert dataset (e.g., 20% of the data, or 40 expert trajectories) is sufficient to enable effective online correction with LPB.\n\n### References\n\n[1] Liu, Bo, et al. *Libero: Benchmarking knowledge transfer for lifelong robot learning.* Advances in Neural Information Processing Systems 36 (2023): 44776-44791.\n\n[2] Nalisnick, Eric, et al. *Do deep generative models know what they don't know?.* arXiv preprint arXiv:1810.09136 (2018).\n\n[3] Graham, Mark S., et al. *Denoising diffusion models for out-of-distribution detection.* Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[4] Livernoche, Victor, et al. *On diffusion modeling for anomaly detection.* arXiv preprint arXiv:2305.18593 (2023).\n\n[5] Heng, Alvin, and Harold Soh. *Out-of-distribution detection with a single unconditional diffusion model.* Advances in Neural Information Processing Systems 37 (2024): 43952-43974.\n\n[6] Lee, Sung-Wook, Xuhui Kang, and Yen-Ling Kuo. *Diff-dagger: Uncertainty estimation with diffusion policy for robotic manipulation.* arXiv preprint arXiv:2410.14868 (2024)."}]

Author rebuttal:

Meta-review: The paper proposes a novel framework to enhance behavior cloning diffusion policy, by learning a separate dynamics model and using classifier guidance to guide the generation process to in-distribution states to encourage recovery. In the discussion stage the reviewers are convinced by further experiments on additional environments and sensitivity to hyper-parameters. They are in consensus that this paper should be accepted.
Decision: Accept (spotlight)