Title: DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance
Venue: NeurIPS 2025
Accepted: yes
Keywords: robotics

diffusion policy

generative model for robotics

inference-time steering
Abstract: Deploying large, complex policies in the real world requires the ability to steer them to fit the needs of a situation. Most common steering approaches, like goal-conditioning, require training the robot policy with a distribution of test-time objectives in mind. To overcome this limitation, we present DynaGuide, a steering method for diffusion policies using guidance from an external dynamics model during the diffusion denoising process. DynaGuide separates the dynamics model from the base policy, which gives it multiple advantages, including the ability to steer towards multiple objectives, enhance underrepresented base policy behaviors, and maintain robustness on low-quality objectives. The separate guidance signal also allows DynaGuide to work with off-the-shelf pretrained diffusion policies. We demonstrate the performance and features of DynaGuide against other steering approaches in a series of simulated and real experiments, showing an average steering success of 70% on a set of articulated CALVIN tasks and outperforming goal-conditioning by 5.4x when steered with low-quality objectives. We also successfully steer an off-the-shelf real robot policy to express preference for particular objects and even create novel behavior.

DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance

Abstract

Deploying large, complex policies in the real world requires the ability to steer them to fit the needs of a situation. Most common steering approaches, like goal-conditioning, require training the robot policy with a distribution of test-time objectives in mind. To overcome this limitation, we present DynaGuide, a steering method for diffusion policies using guidance from an external dynamics model during the diffusion denoising process. DynaGuide separates the dynamics model from the base policy, which gives it multiple advantages, including the ability to steer towards multiple objectives, enhance underrepresented base policy behaviors, and maintain robustness on low-quality objectives. The separate guidance signal also allows DynaGuide to work with off-the-shelf pretrained diffusion policies. We demonstrate the performance and features of DynaGuide against other steering approaches in a series of simulated and real experiments, showing an average steering success of 70% on a set of articulated CALVIN tasks and outperforming goal-conditioning by 5.4x when steered with low-quality objectives. We also successfully steer an off-the-shelf real robot policy to express preference for particular objects and even create novel behavior. Videos and other visualizations can be found on the project website: https://dynaguide.github.io

1 Introduction

The rise of large datasets and expressive policies has enabled complicated skills in robots like folding clothes [3, 55], making sandwiches [44], and washing dishes [7]. When these large policies are deployed in the dynamic real world, we face the challenge of steering them to match the needs of a specific scenario. This means finding a subset of the policy's behaviors that are appropriate for the scenario [48]. The policy's complexity means that we ideally want to accomplish this steering without needing to retrain the policy, sample excessively from it, or anticipate all the possible steering during policy training.

Most existing works for policy steering, however, rely on at least one of those assumptions. Even language or goalconditioned policies are trained on a set of

Pre-trained Policy with

on Diverse Data

Steering to

Steering to

Behaviors

Don't Open Drawer

Switch On or Drawer Open

a

k

t

DynaGuide

goal distributions, which requires foresight for the goal distribution that satisfies all future steering needs [45]. In our work, we distance ourselves from these assumptions. We propose separating the steering forces from the base behavior policy and combining the influence from the steering model with the strong action prior of the base policy to generate guided behaviors that satisfy inference-time objectives while leveraging the base policy's existing skills.

Steering robot behavior requires understanding how current actions influence future outcomes, a feat well-suited for dynamics models [39, 16]. To adapt a dynamics model for policy steering, we craft a latent visual dynamics model that predicts the final or far-future observation of a trajectory given the current observation and action, allowing it to understand long-horizon dependencies. This dynamics model learns from unstructured environment interaction data and operates on visual observations projected into the expressive DinoV2 latent space [37] previously shown to be useful for robot planning [56]. In this latent space, we can compute the distance between the predicted future observation and observations of other desired/undesired outcomes, creating a differentiable metric that encompasses the guidance objectives.

Previous methods that steer policies with external metrics will sample actions from the policy and pick the action that best satisfies the metric [39, 34]. Instead, our approach leverages the denoising stochastic process of diffusion-based policies. Taking inspiration from classifier guidance [9] and dynamics diffusion guidance in other applications [53], we take the gradient of the metric through the dynamics model and use this dynamics gradient to directly modify the action denoising process.

This separate dynamics model and the active diffusion guidance make up DYNAGUIDE, a new way of steering pretrained diffusion policies towards complex objectives. This design offers several key advantages over existing steering approaches:

Flexible Steering Structure. Goal-conditioned formulations have rigid input structures that cannot be modified during inference-and most policies accept only a single goal condition. DynaGuide accepts a collection of guidance conditions encompassing any number of positive and/or negative objectives (§4.3).

Increased Steering Robustness. Goal-conditioned approaches require training the input condition alongside the policy, causing out-of-distribution (OOD) failures when new or lower-quality objectives are introduced during inference. The DinoV2 embedder and separation of guidance and policy in DynaGuide means that no individual part of the system is OOD under this same situation, and the averaging effect of the guidance metric (Eq. 2) means that DynaGuide still extracts a meaningful guidance signal (§4.2). The additional experience present in the dynamics model also enables guidance towards novel behavior (§4.5).

Plug-and-Play Modularity. Unlike fine-tuning approaches, DynaGuide only modifies the inference process of the diffusion policy, allowing different dynamics models to be swapped in and out without changing the base policy. This modularity also allows an off-the-shelf policy to be steered without further modifications (§4.5).

Enhancing Underrepresented Behaviors. Sample-based steering approaches [39, 34] select the best action from a set of proposed actions by the base policy. Not only does this require many inferences of the policy per step, these sampling approaches also struggles with expressing behaviors that are underrepresented in the base policy (§4.4). In contrast, our Dynamics Guidance directly influences the denoising process. Through one denoising sequence, we guide the action towards the specified objectives, even if the behavior is less common in the base policy (§4.4).

To investigate these claims, we conduct five experiments using simulated CALVIN environment tasks [30] and three experiments on a real robot arm. We show that DynaGuide successfully guides the base policy up 70% of the time (§4.1) and outperforms goal conditioning by 5.4x on lower quality objectives (§4.2). It successfully amplifies underrepresented behaviors over sampling methods (§4.4) and accommodates multiple positive and negative objectives (§4.3). DynaGuide is also successful on an existing real robot policy, achieving up to 80% guidance success and even creating a novel behavior (§4.5). We will make code and collected data publicly available. Videos and visualizations can be found on our website, https://dynaguide.github.io

2 Related Works

Conditioning Policies on Goal Representations. Acommon approach that influences robot behavior is goal conditioning , where the policy is given a representation of a desired outcome. The common representation is natural language [5, 57, 23, 36], which can also be used as a steering mechanism for human-in-the-loop execution [45, 8]. Language-conditioned policies require special data and/or supervision to obtain and use the language labels, and it may be difficult to specify the exact steering needs through language alone. Other methods with fewer assumptions can learn goal-conditioned policies from trajectories by conditioning on learned latent embeddings [21, 28] or using future observations directly as the goal [41, 4, 38, 10, 2]. Like DynaGuide, these conditioned policies learn to accomplish the outcome represented by the goal observation. However, unlike DynaGuide, these approaches require training a policy that can take the goal condition as the input. DynaGuide's external dynamics model means that it works on top of any diffusion policy. It also supports multiple desirable and undesirable outcomes, whereas the observation-based goal conditioning typically supports only one desirable outcome. In our experiments, we compare the performance of DynaGuide with an observation goal-conditioned policy.

Leveraging Dynamics Models to Improve Robot Policies. Models can be trained to predict future states through latent dynamics by taking in an observation latent, action(s), and outputting a predicted future latent state [42, 31, 16, 14, 15, 56, 25]. These latent dynamics are commonly used for training reinforcement learning agents by simulating trajectories through the dynamics model [16, 14, 15, 40, 24] and have seen success on real robots [51]. They can also be used for Model Predictive Control (MPC) to generate trajectories directly [56, 43, 17, 13, 12, 47] and com-

bined RL/Planning approaches [18, 19]. Especially relevant to DynaGuide are approaches that use dynamics models to filter samples from policies according to predicted value [34], VLM feedback [52], or engineered reward [39]. Like some of these prior works and especially Dino-WM [56], DynaGuide uses a transformer-based dynamics model to guide action creation. However, instead of sampling from the base policy, running RL, or MPC, DynaGuide steers the trained base policy directly through the action diffusion process.

Inference-Time Steering . After a policy has been trained, its behavior can still be changed without modifying its weights or requiring goal conditioning [48]. Safety value functions can intervene with a recovery policy or request human assistance when abnormal behavior is predicted [35, 27, 11]. Diffusion policies can be steered by skewing the initial noise distribution [49] or applying classifierfree guidance [20] that influences the action diffusion process [41] or the inverse-dynamics planning process [1]. While classifier-free guidance is effective for steering, it still requires training the policy on supplied conditions, akin to goal conditioning. The alternative steering method is classifier guidance , which leverages an external classifier to influence a diffusion generation process [9]. Classifier guidance in diffusion policies is akin to seeking an optimal trajectory defined by the classifier [22]. Applied to robot policies, classifier guidance can apply post-hoc constraints [32] or seek objects near a human-supplied keypoint [50]. DynaGuide is a classifier guidance approach, but it seeks to guide the policy with more complicated objectives using feedback from a trained dynamics model acting as the classifier. Dynamics-based diffusion guidance has succeeded in other applications like hardware generation [53] and DynaGuide brings it to robot policies.

3 DynaGuide Method

As shown in Fig. 2, we consider the problem of steering a diffusion policy π θ ( a | o ) during its inference process. We are given a set of guidance conditions G = g + ∪ g -, which are represented as image observations that contains partial state information. Some guidance conditions show desirable outcomes g + = { g + 1 , ...g -j } and others are undesirable outcomes g -= { g -1 , ...g -i } . Using G , we want to create a π ′ θ ( a | o ) such that for every a ∼ π ′ θ ( ·| o t ) , the probability p ( g + | o t , a ) is as high as possible and the probability p ( g -| o t , a ) is as low as possible. To accomplish this, we train a

G

1

G

2

G

3

ε

p

o

t

o

t

G

4

∇

a

k

t

d

ˆ

z

t

+

H

DinoV2

Guidance Gradient

Policy Denoising

Gradient

Selected

Mode

Other

Mode

Observation

Actions

(Noised)

Pretrained

Policy

a

k

-

1

t

Online

Guidance

Dynamics Model

(A) DynaGuide

(B) Dynamics Model Guidance

(C) One Denoising Step

a

k

t

a

k

t

d

1

d

2

d

4

d

3

dynamics model and use the model to approximate these probabilities (§3.1) and then use the gradient of that dynamics model to create π ′ by modifying its action denoising process (§3.2).

3.1 A Dynamics Model Capable of Guidance

Creating a model to approximate p ( g + | o t , a ) and p ( g -| o t , a ) has two requirements: 1) the model needs to predict future outcome ˆ o t + H from current observation o t and action sequence a , 2) we need to compare o t + H to to the guidance conditions. Here, H can be a large value or t + H can be the end of the trajectory.

Past works on predicting future observations for planning and reinforcement learning have used latent dynamics models to satisfy these two requirements. These dynamics models h θ ( ϕ ( o t ) , a ) operate in a learned latent space z t = ϕ ( o t ) . The recent work of Dino-WM [56] demonstrated the usefulness of latent distances in the pretrained DinoV2 image embeddings [37] for planning. Taking inspiration from Dino-WM, we leverage the same DinoV2 image latent space for our observation comparisons. Concretely, we use the patch embeddings from DinoV2 as z t = ϕ ( o t ) and train a transformer h θ ( a , z t ) to predict a latent outcome representation ˆ z t + H . Because ϕ is frozen and we have access to full trajectories of data during training, it is sufficient to train h θ through a regression objective (Eq. 1). For more details, refer to Appendix B.

With this dynamics model trained, we define a guidance metric that compares the predicted outcome with the guidance conditions. Given one guidance condition g + i , we want to approximate p ( g + i | o t , a ) , which can be done by comparing the predicted outcome ˆ z t + H to g + i . The condition g + i is a visual observation, so we project it into the same latent space z + i = ϕ ( g + i ) . This allows us to directly compare z + i and ˆ z t + H . If we roughly assume that the latent space is gaussian in structure, a reasonable approximation of log p ( g + i | o t , a ) is proportional to -|| ˆ z t + H -z + i || 2 2 , a metric found in similar forms elsewhere for latent reinforcement learning [54, 33] and planning [56].

Algorithm 1 DynaGuide (Inference-Time)

When we try to maximize log p ( g + | o t , a ) across the set of guidance conditions g + , we can try to maximize the combined probability of achieving each outcome log ∑ i p ( g + i | o t , a ) . The negative is true for the undesired outcomes. We use our approximation of log p ( g + i | o t , a ) and a variance hyperparameter σ to get the guidance metric d ( g + , g -, o t , a ) (Eq. 2). The log-sum-exp acts as a soft maximum, a feature useful for diverse objectives (§4.3) and lower quality guidance conditions (§4.2). For a more detailed derivation of d and explanation of design choices, refer to Appendix B.3.

We train the dynamics model using diverse robot interaction data. For the Calvin experiments §4.1-4.4 we use existing human-collected play data from the benchmark. For the real-world experiments §4.5, we use open-source data collected on the UMI interface [6]. Because we will be querying the dynamics model with noisy actions during the guidance process, we augment the training by adding gaussian noise to the actions using the same noise scheduler used during inference time, with a noise step picked from a geometric distribution. For more details about implementation, see Appendix B.

3.2 Guiding the Action Diffusion Process

To sample an action from a diffusion policy π ( ·| o t ) , the policy predicts a sequence of denoising steps ϵ ( a k , o t ) , that moves an initial noise sample a K ∼ N to a denoised action sequence a 0 . Diffusion models can be guided by a classifier p ( y | a ) towards a particular set of samples that maximize p ( y | a ) [9]. The guidance signal comes from the gradient of the classifier ∇ a log p ( y | a ) and is combined with ϵ ( a k , o t ) during the denoising process.

In our case, y is the guidance conditions, and the log p ( y | a ) is our derived metric d in Eq. 2. Like past work [53, 41], we use the Denoising Diffusion Implicit Models (DDIMs) [46] as our sampling method. Under DDIM, the combined denoising and guidance signal is

where ¯ α k = ∏ k l α l , α l is the noise scheduler, and s is a guidance scaling parameter [9]. The higher the s , the stronger the signal to adhere to the guidance requirements. However, past work has discovered that higher s also create less smooth or even incoherent trajectories [50], creating a delicate balance between trajectory validity and adherence to guidance conditions. This phenomenon happens because ∇ a d can pull ϵ out of distribution, leading to erroneous noise predictions that compound as the denoising process continues. To resolve this, Wang et al. [50] proposed the Stochastic Sampling solution of running each denoising step k multiple ( M ) times to stabilize the guidance signal's influence on the action denoising process through MCMC sampling. Empirically, this trick allows s to be pushed higher to increase guidance success without sacrificing trajectory validity. See Algorithm 1 for an overview of DynaGuide. We investigate these hyperparameters in Appendix A.4.

4 Experiments

To understand the features, benefits, and limitations of DynaGuide, we conduct five sets of experiments on the Calvin environment (Figure 3, §4.1-4.4). To demonstrate its practical feasibility on real robots, we steer a publicly available robot policy in a real task using DynaGuide (§4.5). We also conduct experiments on a toy 2D navigation environment, which can be found in Appendix A.1. Wherever relevant, we compare DynaGuide to a representative set of baselines.

Base Policy. This is the diffusion policy [7] that we steer with DynaGuide. This policy offers a fairly uniform distribution across valid behaviors.

DynaGuide-Sampling (GPC) [39]. Instead of diffusion guidance, we sample from the base policy multiple times and pick the action sequence that best satisfies metric d (Eq 2), an idea demonstrated in GPC-Rank with engineered reward functions [39]. We re-implement this idea with our dynamics model and use this baseline to explicitly test the benefits of diffusion guidance.

Articulated Parts

Movable Objects

Randomize Other Objects and

Robot

Interact with Some Objects but Not Others

Fully-Specified Objective

Under-Specified Objective

Multiple Objectives

Goal Conditioning. We train a policy to take an additional visual observation as the goal. Past works in visual goal conditioning have used future observations as goals [2, 36, 41]. In our baseline, we take the common approach of using the last observation of each trajectory as the goal. We compare this model to DynaGuide by sampling from g + as our goal during inference-time. We also test an intermediate goal-conditioning baseline in Appendix A.2.

Position Guidance (ITPS) [50]. Past work has achieved diffusion policy guidance that steers the robot to a particular location in 3D space supplied by a human. For objects that do not move, we replicate this guidance by finding the average location of the robot after accomplishing the desired behavior and using position guidance on that location.

4.1 Experiments 1 & 2: Steering in Complex 3D Environments

Our first two experiments examine how DynaGuide can steer robots to particular behaviors in the CALVIN environment [30]. The CALVIN setup allows a robot to interact with desks with drawers, switches, buttons, and cabinets (Fig. 3). It can articulate these objects and also rearrange a set of three colored cubes that are placed randomly on the table. For these two experiments, we focus on Guidance Conditions g + that represent desired outcomes, taken as the last observation of demonstration trajectories showing the target behavior. We use these same observations for the goal-conditioned baseline by sampling g + i ∈ g + for each rollout as the goal.

In this set of experiments, we are interested in the ability of DynaGuide to steer the base policy towards behaviors that interact with the articulated table elements ( ArticulatedParts ) and randomized cube objects ( MovableObjects ). Overall, for every target behavior across both experiments, DynaGuide significantly increases the frequency of the target behavior over that of the base policy (Fig. 4, Top, Bottom-Left).

On ArticulatedParts , DynaGuide boosts the base policy's behavior by 8.7x and achieves an average target behavior success of 70%. Because the objects in ArticulatedParts stay in place, we can run the ITPS baseline [50]. Using position guidance boosts the target behavior over the base policy, but the reasoning abilities of the dynamics model allow DynaGuide to outperform it across the tasks (Fig. 4, Top). The location consistency of ArticulatedParts also means that the g + outcomes closely match the true outcome of the current environment if the desired behavior is executed. Therefore, the goal-conditioned baseline is in-distribution and performs near perfectly. This is expected, as the goal-conditioned model was trained for this exact setup.

The MovableObjects experiment poses a new set of challenges. Instead of being fixed in location, the cube objects are small and randomly arranged every environment reset. All steering performances decrease in MovableObjects , but two notable differences arise. First, the GPC baseline performance drops to the base policy. Sampling approaches like GPC rely on actions drawn from the base policy. If these actions have high variance, then the selected actions may also have high variance. Compared to the steady guidance of DynaGuide, the increased variance leads to more erratic actions and impacts performance on precision tasks. The second difference is the drastic drop in goal conditioned performance. Unlike ArticulatedParts , the randomized cubes mean that not all outcomes in g + are attainable in the current setup. For example, some outcomes in g + interact with the blue cube on the left side of the table, while a test-time setup might have the blue cube on the right side. This

DynaGuide

DynaGuide Sampling

Articulated Parts

Underspecified Objectives

Movable Objects

Success Rate

Position Guidance

Base Policy

8.7x

boost

5.4x

boost

2.1x

boost

mismatch means that the goal conditioned policy is out of distribution and is affected more than DynaGuide. We explore this phenomenon in the next experiment.

4.2 Experiment 3: Robustness to Lower Quality Guidance Conditions

The cube randomization in MovableObjects created a mismatch between desired outcome observation and attainable outcome in the current environment, a challenge that disproportionally impacted goal conditioning over DynaGuide. While MovableObjects had an inevitable mismatch, we explore a deliberate mismatch in this experiment that models a more practical use case of inference-time steering.

The two prior experiments had used g + drawn from trajectories where the robot achieves the desired behavior. For each guidance condition, the combined robot's position and the target object's position fully indicate the desired behavior. For practical deployment, it is easier to take pictures of the desired scene without collecting trajectory demonstrations or placing the robot in the correct position. We mimic this requirement in UnderspecifiedObjectives , where we set the target articulated table part to the desired pose and randomize all other objects, including the robot. We use images from this setup as g + .

As expected, the lower quality guidance conditions pushed the goal-conditioned policy out of distribution, leading to average success rates below 10% (Fig. 4, bottom right). In contrast, the dynamics model guidance approaches (DynaGuide and DynaGuide-Sampling) were still able to increase the target behavior over base policy rates and outperformed the goal conditioned policy by 5.4x on average. DynaGuide also performed more consistently than its sampling counterpart, further illustrating the importance of active guidance. We attempt to strengthen the goal-conditioned baseline in Appendix A.2 by adding training data and implementing an intermediate goal-conditioning algorithm, but none of these approaches yielded consistent improvements over the tested goalconditioned baseline here (Appendix A.2).

These under-specified objectives showcase the benefits of separating the policy and guidance. Although the guidance conditions are still mismatched for DynaGuide, the inputs ( g + , o t , a ) were all individually in-distribution to the parameterized encoder, base policy, and dynamics model of DynaGuide. The mismatch, therefore, occurs in the latent space of a large visual encoder. With nothing out-of-distribution, we face an easier challenge of extracting a meaningful signal from these latent distances. Fortunately, DynaGuide also makes use of all the conditions in the latent space for the guidance metric (Eq. 2). While each latent distance d i might be noisy and inaccurate, the combined metric will cancel some of the noise, leading to meaningful guidance.

Desired Behaviors

Button

Switch

Door

Drawer

Block

Underrepresented Behaviors

Multiple Objectives

DynaGuide

Sampling

Base Policy

Percent Data Included

Success Rate

Positive Conditions

G = DynaGuide, S = Sampling

G

S

G

S

G

S

G

S

G

S

G

S

Negative Conditions

Undesired Behaviors

No behavior / Failure

4.3 Experiment 4: Steering with Multi-Objectives

Separating the guidance and the policy improves the robustness of DynaGuide to lower quality guidance conditions, and it also allows the guidance signals to be combined , steering the policy towards a multimodal set of desired outcomes-and even away from a set of negative outcomes. This capacity is not possible for an unmodified goal-conditioned policy that takes a single observation as the goal.

In MultiObjectives , we add multiple target behaviors to g + and populate g -with undesired behaviors. Good steering will allow all represented behaviors in g + to be executed while minimizing behaviors in g -and overall behavior failure rate (e.g., doing nothing). Indeed, DynaGuide steers the policy towards multiple target behaviors with fair representation of each behavior (Fig. 5, Left), achieving up an average of 80% success with nearly no full behavior failures. When faced with avoiding behaviors in g -, DynaGuide achieves nearly perfect success in avoiding the undesired behavior while executing a variety of other behaviors (Fig. 5, Middle). For an additional comparison against Classifier-Free Guidance (CFG) [20], refer to Appendix A.5.

Because DynaGuide and GPC both use the same dynamics model for steering, they can both steer to multiple objectives. However, the sampling approach of GPC had lower overall success of desired behaviors and allowed more execution of undesired behaviors, especially when trying to avoid outcomes in g -(Fig. 5, Middle). When avoiding g -, GPC also produces more behavior failures where nothing is executed during the trajectory horizon. Sampling approaches like GPC rely on the base policy to readily produce actions that satisfy the objective. However, especially when the robot is close to an object, the base policy might only sample actions for one behavior, making this behavior impossible to avoid. In contrast, active guidance during the diffusion process can seek rare modes in the action distribution, leading to a higher chance of accomplishing complicated objectives like the ones in this experiment.

4.4 Experiment 5: Enhancing Underrepresented Behaviors

In MultiObjectives , we observed that the GPC sampling approach struggles to satisfy complicated objectives because it relies on adequate samples from the base policy. In UnderrepresentedBehaviors , we test this feature explicitly. Trained robot policies will offer behaviors based on its representation in the training data. However, when we steer a policy, we also want to steer it into behavior modes that are underrepresented in the training data.

To simulate data underrepresentation, we reduce the presence of Switch-On behavior in the training set of the base policy. As expected, the base policy offers more Switch-On behavior as more data is included (Fig. 5, Right). GPC also increases with more data representation, as it can boost the behavior by selecting the right action. Most importantly, DynaGuide consistently outperforms its sampling counterpart at these lower data regimes, achieving 40% success rate with only 1% of the original Switch-On behavior included in the trained base policy (Fig. 5 Right). By directly influencing the

Cup Preference

Base Policy

Base Policy

Guide Red

Guide Red

Base Policy

Guide Mouse

Guide Grey

Hidden Cup

Novel Behavior

Grey Cup

Red Cup

Computer Mouse

action diffusion process, DynaGuide can leverage the knowledge of the dynamics model to seek areas in the action distribution that satisfy an objective, even if they are underrepresented.

4.5 Experiment 6: Steering Off-The-Shelf Policies with a Real Robot

In the simulated Calvin environment, we used existing data to train both the base policy and the dynamics model. DynaGuide works with this setup, but it should also work with any pretrained diffusion policy and it should also work on real robots . To test these two claims, we set up a real robot experiment leveraging a publicly available pretrained policy [6] that picks up a cup and places it on a saucer. We train the dynamics model using open-source data from the UMI collection interface [6] and a set of demonstrations collected on the experimental setup. For more details about the setup, refer to Appendix B.5.

We conducted three experiments on this real-world setup. The first two, CupPreference and HiddenCup , present the robot with two different-colored cups equidistant from the robot's starting gripper. The base policy will select a cup at random and place it on the saucer. Applying DynaGuide in CupPreference creates a preference for a cup color (Fig. 6 Left), leading to an average target behavior success of 72.5%. In HiddenCup , the red mug is hidden behind the grey mug. The base policy will typically go for the closer grey mug, but the active guidance in DynaGuide enables the robot to find the red cup 80% of the time (Fig. 6 Middle).

Inspired by the performance of DynaGuide with data representation at 1% (§4.4), we tested the ability of DynaGuide to steer the pretrained policy towards a novel behavior: touching a computer mouse. In NovelBehavior , the dynamics model was trained on various object manipulations, including mugs and computer mice from other open-source datasets [26]. We used the same off-the-shelf base policy that was only trained to arrange mugs. Although the steered policy still expressed preference for the mug, DynaGuide was able to double the number of interactions with the novel object.

5 Conclusion and Discussion

In this paper, we proposed a novel method of steering pretrained diffusion policies by using a separately trained latent dynamics model. We demonstrated the ability of DynaGuide to enhance target behavior performance across simulated and real experiments, outperforming baselines like position guidance and goal conditioning in some or all setups. Finally, we demonstrated that DynaGuide can be steered to achieve multiple behaviors, avoid undesirable behaviors, and enhance behaviors that were underrepresented in the training of the base robot policy. Real-world robot deployment requires robots to be highly steerable in their skills, and DynaGuide shows one potential avenue for this ability.

5.1 Limitations and Future Work

DynaGuide can successfully steer pretrained policies towards particular objectives, but currently it is difficult to specify the method of achieving this objective. This limitation is due to the current form of

guidance conditions as observations that represent desired/undesired outcomes . There are other, more complicated guidance modalities, including language and kinesthetic demonstrations. Future work includes enabling multi-modal guidance conditions, which can hopefully enable very fine-grained guidance for all parts of the trajectory. In the future, we also hope to add an ability for the base policy to remember past guidance, consolidate the knowledge, and apply it automatically to new tasks.

5.2 Acknowledgments

Maximilian Du is supported by the Knight-Hennessy Fellowship and the NSF Graduate Research Fellowships Program (GRFP). This work was supported in part by the NSF Award #2143601, #2037101, and #2132519, and Toyota Research Institute. We would like to thank ARX for the robot hardware and Yihuai Gao for assisting with the policy deployment on the ARX arm. We appreciate Zhanyi Sun for discussions on classifier guidance and all members of the REAL lab at Stanford for their detailed feedback on paper drafts and experiment directions. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.

The Calvin Experiments (Sections 4.1 - 4.4) were made possible with the Calvin benchmark codebase [30]. The diffusion policy was adapted from the Robomimic repository [29]. The dynamics model was inspired by the Dino-WM implementation [56] and leverages representations from Dino-V2 [37].

References

Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? URL http://arxiv. org/abs/2211.15657 .

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc. URL https://proceedings.neurips.cc/paper_files/paper/2017/ hash/453fadbd8a1a3af50a9df4df899537b5-Abstract.html .

Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. Pi-0: A vision-language-action flow model for general robot control, . URL http://arxiv.org/abs/ 2410.24164 .

Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models, . URL http://arxiv.org/abs/2310.10639 .

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems XIX . Robotics: Science and Systems Foundation. ISBN 978-0-9923747-9-2. doi: 10.15607/RSS.2023.XIX.025. URL http: //www.roboticsproceedings.org/rss19/p025.pdf .

Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In Robotics: Science and Systems XX . Robotics: Science and Systems Foundation. ISBN 979-8-9902848-0-7. doi: 10.15607/RSS.2024.XX.045. URL http://www.roboticsproceedings.org/rss20/p045.pdf .

Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin CM Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Proceedings of Robotics: Science and Systems , Daegu, Republic of Korea, July 2023. doi: 10.15607/RSS.2023.XIX.026.

Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, and Dorsa Sadigh. No, to the right: Online language corrections for robotic manipulation via shared autonomy. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction , HRI '23, pages 93-101. Association for Computing Machinery. ISBN 978-14503-9964-7. doi: 10.1145/3568162.3578623. URL https://dl.acm.org/doi/10.1145/ 3568162.3578623 .

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems , volume 34, pages 87808794. Curran Associates, Inc. URL https://proceedings.nips.cc/paper/2021/hash/ 49ad23d1ec9fa4bd8d77d02681df5cfa-Abstract.html .

Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc. URL https://proceedings.neurips.cc/paper_files/paper/2019/ hash/c8d3a760ebab631565f8509d84b3b3f1-Abstract.html .

Maximilian Du, Alexander Khazatsky, Tobias Gerstenberg, and Chelsea Finn. To err is robotic: Rapid value-based trial-and-error during deployment. URL http://arxiv.org/abs/2406. 15917 .

Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. URL http://arxiv.org/abs/1812.00568 .

Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 2786-2793. doi: 10. 1109/ICRA.2017.7989324. URL https://ieeexplore.ieee.org/document/7989324/ .

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. . URL http://arxiv.org/abs/1912.01603 .

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models, . URL http://arxiv.org/abs/2010.02193 .

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. 640(8059):647-653, . ISSN 1476-4687. doi: 10.1038/ s41586-025-08744-2. URL https://www.nature.com/articles/s41586-025-08744-2 . Publisher: Nature Publishing Group.

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551 , 2018.

Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In ICML , 2022.

Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control, 2024.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. URL http://arxiv.org/ abs/2207.12598 .

Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In Proceedings of the 5th Conference on Robot Learning , pages 991-1002. PMLR. URL https://proceedings.mlr.press/v164/jang22a.html . ISSN: 2640-3498.

Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , 2022.

Gi-Cheon Kang, Junghyun Kim, Kyuhwan Shim, Jun Ki Lee, and Byoung-Tak Zhang. CLIPRT: Learning language-conditioned robotic policies from natural language supervision. URL http://arxiv.org/abs/2411.00508 .

Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In Advances in Neural Information Processing Systems , volume 33, pages 741-752. Curran Associates, Inc. URL https://proceedings.neurips.cc/paper/2020/hash/ 08058bf500242562c0d031ff830ad094-Abstract.html .

Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. URL http://arxiv.org/abs/2503.00200 .

Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. arXiv preprint arXiv:2410.18647 , 2024.

Huihan Liu, Yu Zhang, Vaarij Betala, Evan Zhang, James Liu, Crystal Ding, and Yuke Zhu. Multi-task interactive robot fleet learning with visual world models. In 8th Annual Conference on Robot Learning (CoRL) , 2024.

Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Proceedings of the Conference on Robot Learning , pages 1113-1132. PMLR. URL https://proceedings.mlr.press/v100/ lynch20a.html . ISSN: 2640-3498.

Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.03298 , 2021.

Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. URL http://arxiv.org/abs/2112.03227 .

Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=vhFu1Acb0xb .

Utkarsh Aashu Mishra, Shangjie Xue, Yongxin Chen, and Danfei Xu. Generative skill chaining: Long-horizon skill planning with diffusion models. In Proceedings of The 7th Conference on Robot Learning , pages 2905-2925. PMLR. URL https://proceedings.mlr.press/v229/ mishra23a.html . ISSN: 2640-3498.

Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. URL http://arxiv.org/abs/1807.04742 .

Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. Conference on Robot Learning (CoRL) , 2024.

Kensuke Nakamura, Lasse Peters, and Andrea Bajcsy. Generalizing safety beyond collisionavoidance via latent-space reachability analysis. 2025.

Iman Nematollahi, Branton DeMoss, Akshay L Chandra, Nick Hawes, Wolfram Burgard, and Ingmar Posner. Lumos: Language-conditioned imitation learning with world models. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , Atlanta, USA, 2025. URL http://ais.informatik.uni-freiburg.de/publications/papers/ nematollahi25icra.pdf .

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. URL http://arxiv.org/abs/2304.07193 .

Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. Zero-shot visual imitation. In ICLR , 2018.

Han Qi, Haocheng Yin, Yilun Du, and Heng Yang. Strengthening generative robot policies through predictive world modeling. URL http://arxiv.org/abs/2502.00622 .

Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Proceedings of the 3rd Conference on Learning for Dynamics and Control , pages 1154-1168. PMLR. URL https://proceedings. mlr.press/v144/rafailov21a.html . ISSN: 2640-3498.

Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal conditioned imitation learning using score-based diffusion policies. In Robotics: Science and Systems , 2023.

Jan Robine, Marc Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world models are happy with 100k interactions. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=TdBaDGCpjly .

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. 588(7839):604-609. ISSN 1476-4687. doi: 10.1038/s41586-020-03051-4. URL https://www.nature.com/articles/s41586-020-03051-4 . Publisher: Nature Publishing Group.

Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchical vision-language-action models. URL http://arxiv.org/abs/2502.19417 .

Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z. Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell at your robot: Improving on-the-fly from language corrections. arXiv preprint arXiv: 2403.12910 , 2024.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. URL http://arxiv.org/abs/2010.02502 .

E. Todorov and Weiwei Li. A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005. , pages 300-306 vol. 1. doi: 10.1109/ACC.2005.1469949. URL https://ieeexplore.ieee.org/document/1469949 . ISSN: 2378-5861.

Masatoshi Uehara, Yulai Zhao, Chenyu Wang, Xiner Li, Aviv Regev, Sergey Levine, and Tommaso Biancalani. Inference-time alignment in diffusion models with reward-guided generation: Tutorial and review. URL http://arxiv.org/abs/2501.09685 .

Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning, 2025. URL https://arxiv.org/abs/2506.15799 .

Yanwei Wang, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Perez-D'Arpino, Dieter Fox, and Julie Shah. Inference-time policy steering through human interactions. URL http://arxiv.org/abs/2411.16627 .

Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Daydreamer: World models for physical robot learning. Conference on Robot Learning , 2022.

Yilin Wu, Ran Tian, Gokul Swamy, and Andrea Bajcsy. From foresight to forethought: Vlmin-the-loop policy steering via latent alignment, 2025. URL https://arxiv.org/abs/2502. 01828 .

Xiaomeng Xu, Huy Ha, and Shuran Song. Dynamics-guided diffusion model for sensor-less robot manipulator design. In 8th Annual Conference on Robot Learning .

Ruihan Zhao, Ufuk Topcu, Sandeep Chinchali, and Mariano Phielipp. Learning sparse control tasks from pixels by latent nearest-neighbor-guided explorations, . URL http://arxiv.org/ abs/2302.14242 .

Tony Z. Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Seyed Kamyar Seyed Ghasemipour, Chelsea Finn, and Ayzaan Wahid. ALOHA unleashed: A simple recipe for robot dexterity. In Proceedings of The 8th Conference on Robot Learning , pages 1910-1924. PMLR, . URL https://proceedings.mlr.press/v270/zhao25b.html . ISSN: 2640-3498.

Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. URL http://arxiv.org/abs/2411. 04983 .

Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Proceedings of The 7th Conference on Robot Learning , pages 2165-2183. PMLR. URL https://proceedings.mlr.press/v229/zitkovich23a.html . ISSN: 2640-3498.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: average steering success on articulated task : §4.1. Goal condition improvement: §4.2. All other results in the introduction already point to sections in the paper results: §4.1-4.5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: The limitations are explicitly discussed in §5 and also mentioned in other parts of the paper, including the results §4.1-4.5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Although a brief mathematical derivation is shown in §3.1 and fully worked out in §B.3, they are only meant to show the reasoning behind the guidance metric (Eq. 2). This paper does not contribute theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experiment setups are described in the main paper §4 and in full detail in §B. For the simulated results, the data is publicly available and instructions will be provided on how to use it. Code is provided in the supplemental and a cleaned version will be released in the very near future.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Data in the simulated CALVIN environment was taken from a public dataset [30]. In the real-world setup, the pretrained policy is publicly available as well as the data [6]. An additional dataset was collected in the experimental environment, which will be released publicly in the very near future. A working copy of the codebase will be included in the supplemental material, and a cleaned codebase implementation will be supplied in the very near future.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental setting is summarized in 4 and detailed in B. The detailed training, testing, and model hyperparameters are described in B.1. The code instructions will contain other practical details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All simulation bars and plots contain confidence intervals, with details on these intervals found in §B.4. As mentioned in B.4, the real robot results are resource-intensive and therefore do not have intervals. This is very common for real robot experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details on compute provided in §B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts in §C

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper uses publically available datasets and publically available trained policies for real robots. The dynamics model predicts furture states in an robot environment and has no easy way of misuse or dual use.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Credits for datasets and codebases are appropriately cited throughout the paper and detailed in Appendix D.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: A pre-release version of the code in the supplemental material is provided with documentation that will be fully fleshed out in the near future.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing was used for this project.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No study participants were used, and therefore no IRB was needed.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: Apart from assisting with visualization code and grammar checks, LLMs were not used in this project.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Additional Experiments, Visualizations, and Ablations

A.1 Steering in a Simple 2D Block Task

Early Decision

Base Policy

Guidance: Blue

Guidance

Base Policy

Guidance: Blue

Base Policy

Guidance: Blue

A

B

C

D

Late Decision

Go to Furthest Cube

Policy

Agent

Example of Guidance During Diffusion

As an additional way of understanding how guidance works in DynaGuide, we look at a toy BlockTouch environment. In BlockTouch , the agent navigates a 2D environment and touches a colored square. We train the vision-based base policy and dynamics model on synthetic data that shows navigation to squares in any location of any color. Then, during test time, we use the dynamics model to guide the policy towards cubes of one particular color. In Figure 7A, we visualize the creation of an action chunk under dynamics guidance. In the initial noisier steps, the guidance signal and the denoising policy signal compete for the action's direction. In the example, the agent is nearly touching the yellow square, so the base policy wants to denoise the action towards the yellow square. The guidance wants to move the agent towards the blue square. The vector sum of these two forces pushes the actions towards the blue square. In the later stages of denoising, the guidance and the policy signals start to work together to craft the final action sequence that points to the blue square. This guidance works in situations where early decisions are important for hitting the correct color (Figure 7B) and in situations where late decisions are important (Figure 7C).

Even though the dynamics and base policies are both trained on random navigation behavior, we can leverage the dynamics model to accomplish novel tasks not explicitly present in the base policy. The Late Decision setup (Figure 7C) contains squares that are closer together than the training data, requiring high navigational precision. The Go to Furthest Cube setup (Figure 7D) places the target much further away than three squares, requiring deliberate navigation around the closer squares. The base policy goes to the closer squares, but the dynamics model was able

to reliably guide the policy between the squares and to the target. Note that the visuals in B-D show square distributions and the black dot at the yellow end of the trajectory shows the first and only contact with a square in the environment.

A.2 Additional Goal-Conditioning Baselines

In Experiments 1-3, we use a simple goal conditioning baseline by using the last trajectory state as the goal representation. To maintain fairness with the base policy, we also train the goal conditioned model on only the test environment's data in Calvin. However, we note that the dynamics model in

DynaGuide is trained with multi-environment data. To maintain fairness with the total data exposed to all parts of a method, we train another goal-conditioned policy on the same multi-environment data. When evaluated on UnderspecifiedObjectives , this additional data improved performance across some steering targets, although DynaGuide still outperformed this augmented goal conditioning baseline (GC w/ All Data, Table 3).

Finally, we try to apply an algorithmic improvement on the goal conditioning baseline by implementing Hindsight Experience Replay [2]. Instead of conditioning the policy on only end states during training, we also sample intermediate future states to broaden the goal distribution. We find that this improves steering success across some tasks but not others (GC w/ HER, Table 3). Overall, we find that goal conditioning can be improved for select tasks with additional data or algorithmic changes, but these augmentations do not raise the performance of this baseline to that of DynaGuide.

A.3 Additional Real Robot Baseline

The real robot experiments (Section 4.5) demonstrated the ability for DynaGuide to steer pretrained policies. To further demonstrate the performance of DynaGuide on real robots, we add DynaGuideSampling baseline for one real robot task. As seen in Table 2, influencing the action sampling with the diffusion model leads to an improved success rate of the target task over baseline, but the classifier guidance of DynaGuide still yields a higher success rate of the target task.

A.4 Ablations and Hyperparameter Investigation

To understand how various hyperparameters and components of DynaGuide contribute to the final success rate, we run a set of experiments on the Switch-On task in the ArticulatedParts experiment. In general, we observe that the two main hyperparameters σ (Eq. 2) and s (Eq. 3) are robust to reasonable changes, meaning that DynaGuide is not difficult to tune. At very low values of s , success rates are lower because there is not enough guidance. Success rates increase with higher strength until the strength becomes too high and creates instability. Very low values of σ decreases guidance success by making the guidance conditions too far away for any meaningful signal. Increasing the number of stochastic sampling steps M generally increases the performance of guidance at a cost of computational expense. Surprisingly, DynaGuide is very robust to the number of guidance conditions G , achieving comparable guidance even with one guidance condition.

In practice, we use a guidance strength and σ chosen from a gridsearch for each experiment. These parameters are often very similar across tasks in the same environment (Table 4), which means that switching target tasks will not require extensive hyperparameter search. We use stochastic sampling M = 4 for our experiments as a balance of stability and computation efficiency. We use 20 guidance conditions per task, which is more than sufficient for Switch-On but is more important for underspecified goals (§4.2) and harder objectives.

Critically, we discover that pretraining the dynamics model with a noised action is essential for performance (Fig. 9). This makes sense as the noise would otherwise be out of distribution.

across tasks. In experiments 1-5, we leverage a more diverse dataset collected across four different Calvin environments to train the dynamics model. This data magnitude improves training stability and re- duces overfitting. However, because we test only on a single Calvin environment, we conduct a

data ablation and train the dynamics model only on data from that environment. As seen in Table 5, reducing the dynamics training data also reduces the steering success, although the performance is still above baseline (Experiment 4.1). This ablation demonstrates that DynaGuide can transfer information from related environments to help steer a policy in a target environment, thereby reducing the data requirement for training a dynamics model in a new environment.

button_on

button_off

switch_on

switch_off

drawer_open

drawer_close

door_left

door_right

All Environments

0.70

±

0.031

0.70

±

0.038

0.74

±

0.027

0.76

±

0.037

0.60

±

0.015

0.66

±

0.029

0.74

±

0.025

0.68

±

0.026

Target Environment Only

0.66

±

0.023

0.69

±

0.038

0.54

±

0.043

0.58

±

0.031

0.51

±

0.022

0.53

±

0.024

0.65

±

0.019

0.57

±

0.036

A.5 Classifier-Free Guidance Baseline

An additional relevant baseline is the Classifier Free Guidance (CFG) [20] that leverages versions of the diffusion policy itself to guide the diffusion process. We implement and run CFG as a baseline for MultiObjectives . CFG can guide towards multiple objectives, a feat that the goal-conditioned baseline was not able to accomplish. Table 6 presents these additional results for a CFG baseline, which should be compared to those of DynaGuide and DynaGuide-Sampling in Figure 5. As expected, CFG can successfully guide the policy towards multiple objectives, including accomplishing multiple behaviors and avoiding other behaviors. However, the total success rate for positive guidance (first three rows in Table 6) is lower than that of DynaGuide (Figure 5).

The CFG is also able to steer the policy away from behaviors, as seen in the last three rows of Table 6. The success rates of the undesired behaviors for CFG and DynaGuide are within one standard deviation. However, while DynaGuide nearly always offers a valid alternative behavior (Figure 5), the CFG baseline fails to execute a behavior more than 10% of the time for all three tested conditions (No Behavior, Table 6). This is an expected result, as the log-sum-exp latent selection method of DynaGuide allows the guidance to find an appropriate behavior and minimize the influence of other contradictory behaviors. In contrast, the CFG implementation gives equal weight to all guidance signals. The lack of a latent distance makes it difficult to select the most relevant guidance. Contradictory guidance signals can influence the diffusion process negatively, leading to poorly-formed actions and unsuccessful behaviors.

B Implementation Details

B.1 DynaGuide Implementation Details

Base Diffusion Policy The diffusion policy is trained to take a 2 -step history stack of visual observations and robot proprioception and predict a chunk of 16 actions. It uses a Resnet-18 image encoder that conditions a U-Net with 4 encoding and 4 decoding layers. This U-Net predicts the noise during training using the standard diffusion policy objective [7]. During inference, we use the noise predictor with the DDIM noise scheduler to craft the generated action chunk. Together, the diffusion policy has around 18 million parameters. We use the Adam optimizer with a learning rate of 1e-4. We train the model for 200k gradient steps with a batch size of 16. All parts of the diffusion policy are trained

together using expert data. During execution, 14 actions are executed in the environment open-loop before the policy is queried again.

Stochastic Sampling Steps

per Denoising

Number of Guidance

Conditions

σ

Parameter

Success Rate

Dynamics Model . We use the DinoV2 patch embeddings as the latent space, which provides 256 patches of embeddings with size 384 . While this is a large embedding size ( 256 × 384 ), past work has shown the effectiveness of this exact embedding in robot planning [56]. Empirically, both in past work and in our development, we discovered that the CLS token in DinoV2 ( 1 × 384 ) does not carry as much spatial information needed for meaningful latent guidance. For the latent dynamics model, we use a 6 -layer Transformer Encoder with 8 heads that takes in the 256 patches, a token for robot proprioception, and an action token. This action token is created by embedding each action in a k using the same MLP and concatenating the embeddings into the token. We use a learned position embedding. The first 256 outputs of the transformer are taken as the predicted future state. We regress these patches to the DinoV2 embeddings of the last (or later) observation of the trajectory. For the CALVIN experiment, we regressed to the final latent of the trajectory. For the real-world experiment, we regressed to the latent of the observation after executing the proposed chunk of actions in the environment. We use the Adam optimizer at a learning rate of 1e-4. Because the DinoV2 embedding is frozen, the transformer can be trained end-to-end using a regression objective. This transformer has approximately 16 million parameters. We train the model for 600k gradient steps with batch size 16 , a point past model convergence. During development, we showed that validation performance depends strongly on the action's presence, indicating that the dynamics model has learned to listen to actions.

To account for noisy actions during guidance, we add noise to the actions used to train the dynamics model. 50% of the fed actions are noiseless, and 50% are noised based on the same DDIM scheduler used during inference. We select the noising step using a geometric random variable with an expected noise step of 20 . The scheduler starts at step 100 with pure gaussian noise, so at step 20 , it is a noisy but still meaningful action that the dynamics model can use.

Inference-Time Steering . Before inference-time, we take the guidance conditions and pre-compute their embeddings. Then, during inference-time, we compute the objective (Eq. 2). Empirically, the Euclidean distance yields more stable results in lieu of squared distance in Eq. 2. We backpropagate this metric through the dynamics model and use the gradient with respect to the input actions as the guidance signal. Like the base policy, we use the DDIM sampler to reduce inference-time computation from 100 steps to 10 steps. To implement stochastic sampling, we repeat the same denoising step multiple times within the denoising loop.

Compute Hardware . All policies and dynamics models were trained on single RTX 3090 GPUs with 24GB VRAM, taking 24-48 hours to convergence. The dynamics model is ≈ 15M trainable paramters, which takes up ≈ 4GB of GPU memory during training and inference. All experiments were conducted on single RTX 3090 GPUs taking 10-20 minutes per seed per task. Development of the method and experiments were also conducted on RTX 3090 GPUs, with the total compute-hours for development estimated at 10-20 times the results shown in the paper. Most of this compute was spent developing the dynamics model.

B.2 Implementation of Baselines

Goal Conditioning . We use the provided state-conditioning implementation in Robomimic [29] and condition the diffusion policy on an additional input consisting of the final observation in the trajectory. We train this policy using the same hyperparameters as the base diffusion policy.

Position Guidance (ITPS) [50] . The ITPS algorithm excels at steering the policy towards a target point in 3D specified by a human. To compare ITPS to our approach that does not require human specification, we manually compute the average final position of the robot for a set of 20 trajectories showing the desired behavior. We then use this position as the target point in ITPS.

Sampling (GBC) [39] . Previous work that introduced GBC-Rank used an objective function to rank action samples from a policy to select the best sample. We take that idea and apply it to our guidance metric (Eq. 2). We sample the base policy 5 times per inference and pick the best action to execute based on the metric.

Noise Pretraining Ablation

Success Rate

Classifier-Free Guidance [20] . We use the same goal-conditioned architecture as the goalconditioned baseline. However, during training, we randomly zero the goal condition to represent the unconditional input needed in classifier-free guidance. During inference, we compute the classifier-free guidance noise estimation using the same conditions used in the goal-conditioned policy baseline.

B.3 Distance Metric: Derivation and Design Choices

In this section, we will conduct a more detailed derivation of the metric shown in Eq. 2. While other works [56] simply assume the metric to be Euclidian, we use a probabilistic motivation to understand the log-sum-exp, which we discovered empirically to be very important for the performance of DynaGuide. For the first part of this derivation, we will only talk about the desired outcomes z + , and the undesired outcomes follow by negation.

We start from the very rough approximation that the latent space is Gaussian with diagonal variance Σ . It is an approximation that methods using squared latent distances already implicitly make. Here, the probability p ( z + i | ˆ z t + H ) = N ( z + i , µ = ˆ z t + H , Σ = σI ) . The log probability can be computed as follows:

There are different ways of combining the influence of multiple z + i in z + . One way that empirically does not work is adding the log-probabilities together:

This is not a surprise, as adding log-probabilities is equivalent to multiplying the probabilities together. As previously discussed, not all the z + i 's are achievable and some may be mutually exclusive. For example, in the multi-objective setup (§4.3), some z + might correspond to pressing the button and

others to opening the drawer. The influence of any single z + i is too powerful in this setup because of the product.

In contrast, it is more sensible to add the probabilities . This way, changes that generally increase the likelihood of the z + i 's will increase the metric, even if it decreases the likelihood of some other competing z + i 's. To add the probabilities, we can do the following:

Which creates an intuitive result, as the log-sum-exp is a soft maximum. The metric focuses on the distances that are closer to the desired outcome. The value of σ modulates the sharpness of the soft maximum.

We compute the same metric for z -. Because we want results in z -to be mutually exclusive to z + , we want to divide the probabilities:

Intuitively, it means that there is a strong gradient incentive to push p ( z -) as small as possible, which is not as strong if we were to use p ( z + ) -p ( z -) . We observe the benefits of the division empirically. This setup corresponds to subtracting the undesirable outcome metric from the desirable outcome metric. To create the final d , we absorb the constant 2 into σ and discard the constant C . This gives us the expression in Eq. 2:

Empirically, we discover that Eq. 2 works best with the squared L2 distance substituted with Euclidean distance. This deviation from the theoretical result is best attributed to the Gaussian latent assumption of DinoV2 not being fully true to the real embedding space.

In practice, g + can be either a future state ( t + H ) or a final state ( T ). Using a future state t + H is easier to train for the dynamics model but it is harder to find a correct g + during inference-time guidance, as it must be a state achievable H steps from the current timestep. Alternatively, predicting a final state T means that final states from demonstration trajectories can be used as g + , making the choice of inference-time guidance examples straightforward. However, training a dynamics model to predict final state can be difficult for long-horizon tasks. We find that the Calvin environment enables end state prediction ( T ), but the real world robot environment requires setting H = 48 steps into the future. We feed entire trajectories as g + and rely on the log-sum-exp latent selection to dynamically pick the most relevant guidance points. The choice of g + is an important design choice and it depends on the environment and the data.

B.4 Experimental Setup: Simulation

CALVIN Environment: Data . Although CALVIN is used for benchmarking, we just use the CALVIN tasks for the data and our own sets of experiments. The CALVIN data is provided as a

continuous set of transitions. We used privileged state information to segment these transitions into trajectories showing one behavior per trajectory: switch on, switch off, drawer open, drawer close, door left, door right, button on, button off, red touch, red displace, blue touch, blue displace, pink touch, pink displace. We did our own segmentations to be consistent with our evaluation criteria (See later paragraphs). The segmentations were also important to extract end observations for the goal-conditioning baseline and training the dynamics model.

CALVIN Environment: Base Policy . We use the CALVIN-D dataset to train the base policy. We use third-person observation, wrist camera observation, and full robot proprioception for the base diffusion policy. Because the demonstrations show a large variety of behavior, the trained base policy also offers a wide variety of behaviors.

CALVIN Environment: Dynamics Model . Because the dynamics model can take a wider range of data, we train the dynamics model on the CALVIN-ABCD dataset, which is the full data split provided by the benchmark. We discover that adding the non-relevant environments improved convergence and reduced overfitting. We use the third-person observation and full robot proprioception as inputs to the dynamics model. The short tasks horizon of the CALVIN tasks meant that it was advantageous to train the dynamics model to predict the final state observation of the single task trajectory. We use the trajectory segmentations to obtain the final observation target during training.

CALVIN Environment: Evaluations . We segment the provided validation CALVIN dataset and randomly select 20 trajectories per desired behavior to extract g + , g -using the last state observation (except for the UnderspecifiedObjectives experiment). For each evaluation, we perform 50 trials with a horizon of 400. We use privileged state information to monitor the object interactions and we stop the rollout when an object is sufficiently articulated or moved. If no object is sufficiently moved or articulated after 400 steps, we count the trial as a failure (no behavior). For each target behavior (horizontal axis in Fig. 4), we compute the success rate by finding the frequency of trials that show the target behavior. We reset the robot randomly by sampling a starting pose in a validation set of trajectories. All error bars in simulation results (Figures 4, 5) are 1-sigma error bars. The standard error is calculated by evaluating DynaGuide on six policies trained on individually separate train-validation splits of the Calvin dataset. Because each of the six success rates is already an average across 100 trials, we use the standard error of the mean as the sigma. We assume Normally-distributed errors, so the standard error is computed with the standard σ/ √ n formula where σ is calculated through a Numpy function. For the line plot (Fig. 5 Right), the shaded regions were computed as standard error of the mean of six success rates-the same as the bar graphs.

ArticulatedParts Experiment . To count as being sufficiently articulated, we require buttons and switches to be fully pressed/flipped such that the light changes state. For drawers and doors, they must be articulated past halfway from the starting location to their end location.

MovableObjects Experiment . In this task, we focused on the ability to steer the policy towards the colored cubes. To be successful, we required the robot to be touching the cube and displace it a slight distance, which can be possible by lifting the cube or nudging it.

UnderspecifiedObjectives Experiment . In this task, we used the ArticulatedParts experiments but fed all the guidance approaches with a lower quality guidance condition. Instead of taking the observations from the final states of validation trajectories, we manually set the target object to the desired pose (e.g. drawer open, or door to the left). We sample states for all the other objects, and we randomize the robot by setting it to a start pose sampled from the validation trajectories. The critical difference between these g + and the g + used in the other experiments is that the robot is no longer shown directly interacting with the object of interest. Even though the robot position is randomized, it is still possible to figure out the target behavior because it is the one object state that stays constant between conditions.

MultipleObjectives Experiment. This task, we extracted guidance conditions from validation trajectories that represented multiple types of behaviors. We are not interested in chaining the behaviors, but rather a steered policy that offer multiple desired behaviors with comparable frequency while avoiding the undesired behaviors. During evaluation, we still terminated the rollout after a single behavior and we compute the success rates in the same way as all previous experiments. However, instead of reporting a single success rate per task, we report the whole task distribution (Fig. 5, Left).

UnderrepresentedBehaviors Experiment. This task required retraining the base policies with modified training data. Using privileged state information to get trajectory behavior labels, we intentionally removed Switch-On behavior from the training set of the base diffusion policy to create sets where 1%, 2%, 5%, 10%, 20%, 40%, 60%, and 80% of the original Switch-On data was kept.

BlockTouch Experiments . We collect synthetic data of the agent navigating to a randomly selected square by creating a bezier curve with 0-2 intermediate points. This bezier curve means that the dynamics model can't infer the final destination with perfect accuracy based on the current direction of travel. Instead of using the metric d , we directly train the dynamics model to classify the output square color as a 4-dimensional vector categorical distribution. During inference-time, we use a cross entropy metric between this distribution and a one-hot vector representing the desired square color. During training and all data collection, we randomize the location of the squares. During test-time, we introduce specific square arrangements to test the properties of the guidance In the EarlyDecision test, we still randomize the squares but ensure that the cubes stay in their own regions, forcing the agent to make an early directional decision. In LateDecision , we keep the squares close together, and in Go to Furthest Cube , we always have the blue square on the opposite side of the environment as the starting agent location. This agent must move past the three other squares to find the blue square. We terminate the environment once any square has been touched.

B.5 Experimental Setup: Real Robot

Base Policy . We use a publicly available trained diffusion policy provided by the Universal Manipulation Interface repository (github link). It was trained on ≈ 2k trajectories of cup grasping, reorientation (to move the handle to the left), and placement on a saucer. The policy takes images from a gopro with a Max Lens mod and outputs 16 actions per generation. These relative actions represent the change in the robot from its current position [6]. This policy has mostly seen single cups and saucers in the environment, but when provided with multiple cups, it will choose a cup at random. To demonstrate DynaGuide on existing policies, we do not modify this policy. Although the policy offers random choice, it also has a bias towards the left side of the environment. To account for this bias, we randomize the location of the desired cup in CupPreference . We also place the computer mouse in the area opposite to this bias in NovelBehavior to ensure that all improvements are due to steering and not base policy bias.

Data . We train the dynamics model on the open-source cup rearrangement data provided by the Universal Manipulation Interface repository (data link). Because we want to steer the policy towards one of two cups, we need to train the dynamics model on these decisions. Using the Universal Manipulation Interface hardware, we collect 500 demonstrations that shows two cups and one saucer in the experiment scene. We pick a cup and place it on a saucer. We use different cups and saucers, with no correlation between cup pairs and saucers. We combine this data with the existing cup arrangement data for the dynamics model. For the NovelBehavior experiment, we additionally train the dynamics model on 3648 publicly available trajectories of computer mouse rearrangement [26], which gives it the experience needed to steer the base policy towards the computer mouse.

Setup . We use an ARX5 robot arm equipped with soft Fin Ray fingers and a gopro with Max Lens mod that mimics the setups used to collect base policy's training data. We use the hardware controller stack provided by the Universal Manipulation Intervace adaptation for ARX5 arms (Github Link) and a power supply with overcurrent protection for safety during deployment.

Evaluations . We conduct 20 trials for each base/guided policy. Real-world evaluations are very resource intensive, so we did not create confidence intervals. This is very common for real robot results, even for large projects [5, 57].

CupPreference Experiment . We arrange the red and grey cups such that the starting distance to either cup is roughly equidistant from the robot. We randomize this distance by sometimes placing the two cups close, and other times placing the two cups far away. We place the saucer randomly in the environment. For 50% of the trials, the red cup is on the left, and the other 50% the red cup is on right. We measure success as picking up a cup and placing it on the saucer.

HiddenCup Experiment . We arrange the red cup in front of the grey cup, such that the grey cup is always closer to the robot than the red cup. We randomize the distance between the red and grey cup, as well as the overall location of the two cups from the robot. We randomly place a saucer in the environment, although we count a cup grasp as a success without needing this saucer. We

discovered that the two cup setup in this configuration is out of distribution enough for the base policy to increase the mistake of cup placement. Because we are trying to steer for cup color preference, the act of grabbing the cup is sufficient for this experiment. The previous experiment tested full behavior success.

NovelBehavior Experiment . Like CupPreference , we arrange the red cup and black computer mouse equidistant from the robot's starting location. As previously mentioned, we place the mouse in the area opposite of its side bias to ensure that the observed effects are attributed to steering and not base policy bias. Like in HiddenCup , we count a grasp attempt of the computer mouse as a success. This is because the computer mouse is very out of distribution for this base policy and it will not successfully grab the mouse. What matters in this experiment is being able to reach for a novel object through dynamics guidance.

C Broader Impact

DynaGuide contributes to the field of robotics by improving the ability for pretrained policies to be steered without retraining, potentially reducing energy consumption otherwise needed during retraining. Adding steering onto pretrained policies also potentially increases accessibility for labs that would otherwise be unable to retrain large policies. This steering approach can also be an effective way of removing unwanted or problematic biases in trained robot policies after the training process. The dynamics model is trained on data that is either publically available or easy to collect, and the data contains no private or sensitive information.

Because DynaGuide allows steering of existing policies, there is a risk of bad actors steering offthe-shelf policies to do dangerous or malicious behaviors, a risk also present for other generative models. To mitigate these risks, trainers of the base policy can ensure that malicious behaviors are not represented at all. Knowing the mechanism of DynaGuide, it may also be possible to train base policies to act adversarially to the guidance signal if asked to do malicious behaviors, making the guidance as difficult as retraining the policy from scratch.

D Code, Assets, and Licenses

All our models are trained on publicly available data and leverages codebases and assets under these licenses:

CALVIN Environment and Data (MIT License) [30]

Robomimic Codebase (MIT License) [29]

DinoV2 Model (Apache license) [37]

Universal Manipulation Interface Codebase, pretrained model, and data (MIT License) [6, 26]

Our code will be publicly released under the MIT license in the near future. Provided in the supplemental material is a barebones version of the code.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose DynaGuide to steer pre-trained diffusion policies at inference time without needing to retrain them. It uses a separately trained dynamics model to predict the future outcome of an action. The goal is specified through positive and negative goal images. By calculating the distance in a latent space between the predicted outcome and the goal images, the system generates a guidance signal. The gradient of this signal is used to directly modify the action denoising process of the diffusion policy, steering it towards desired behaviors and away from undesired ones.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- **Plug-and-Play:** It can steer existing, off-the-shelf diffusion policies without any modification or fine-tuning.\n    \n- **Robustness to Low-Quality Goals:** By separating the policy and the dynamics model, the system is more robust to \"under-specified\" goal.\n    \n\n### Weakness\n\n- **Guidance is Outcome-Based:** The method can only specify the desired *final outcome* of a task via an image. It cannot control *how* the robot should perform the task (i.e., the specific trajectory or method of execution).\n    \n- **Reliance on Dynamics Model:** The performance is entirely dependent on the quality and data coverage of the dynamics model. If the dynamics model cannot accurately predict the consequence of an action, the guidance will be ineffective.\n    \n- **Limited Baselines:** The paper does not compare against methods that more tightly integrate a world model into the diffusion process itself, like the original diffuser paper + inpainting [1] , which could be a relevant alternative approach.\n\n[1] Janner, Michael, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. “Planning with Diffusion for Flexible Behavior Synthesis.” arXiv, December 20, 2022. http://arxiv.org/abs/2205.09991.\n\nQUESTIONS:\n- **Negative Goal Selection:** How sensitive is the method to the choice of negative goal images? For high-precision tasks, a failure state might look very similar to a success state, which could make providing effective negative guidance difficult.\n    \n- **Alternative Guidance Modalities:** As noted in the paper's future work, can guidance be extended beyond static images to other forms like language (\"push the red block gently\") or kinesthetic demonstration to provide more fine-grained control over the trajectory?\n\nLIMITATIONS:\n- **Lack of Trajectory Control:** The guidance mechanism steers towards a final state but offers no direct control over the path taken to reach that state.\n    \n- **Computational Cost:** To use strong guidance without destabilizing the trajectory, the method relies on MCMC sampling (running each denoising step multiple times), which increases the computational overhead at inference time.\n    \n- **No Online Learning:** The guidance is applied only at inference time and is forgotten afterward. The base policy does not adapt or improve from the guidance it receives.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces DynaGuide, a method for guiding diffusion policies towards desirable goals by using classifier guidance. The guidance mechanism includes a learned latent dynamics model operating on DinoV2 embeddings. The dynamics model predicts a future state from the current state and action chunk, and the gradient of the distance from the desired goal embedding is used for guidance. This method allows decoupling guidance from policy, allowing dynamics models to be swapped out as needed, and can be used with an off-the-shelf policy. Experiments on CALVIN tasks show that DynaGuide can help in cases where the goal is misspecified or corresponds to an underrepresented behavior of the base policy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The experiment design is done quite well. The authors study different settings designed to test different capabilities, such as working with low-quality goal conditions, multiple objectives, and testing underrepresented behaviors of the base policy.\n- It is not common to see experiments on real robots, and the fact that DynaGuide works in this setting is a strong point in its favour.\n- The authors have provided details of their implementation and baselines in the appendix, as well as provided their code in the supplementary material. While I did not study the code, the details in the appendix are quite thorough, and the authors’ efforts in promoting reproducibility are appreciated.\n- The paper is written quite clearly and is easy to follow.\n\n**Weaknesses**\n\n- DynaGuide unifies classifier guidance with learned dynamics, which is new for robotics, but inherits much from prior dynamics-distillation work (DINO-WM) and inference-time steering (ITPS). The advance is incremental yet practical; clearer articulation of conceptual novelty over these works would strengthen the contribution.\n- One primary motivation of DynaGuide is that typical goal-conditioned policies require anticipating the set of goals during training such that they cover the possible goals encountered during inference. DynaGuide does not seem to solve this problem, since the steering mechanism relies on the base policy already having the ability to solve the required task. Therefore, it seems that the claim in lines 34-42 that sets DynaGuide apart from existing methods should be reconsidered.\n- In practical settings, it is rarely feasible to specify the exact goal observation that the agent needs to accomplish. In line 103, the authors claim that “Other methods with fewer assumptions…using future observations directly as the goal”. Assuming access to goal observations for guidance is a much stronger assumption than relying on goal specification via natural language, for example.\n- The dynamics model uses much broader data than the policy (CALVIN-ABCD vs D). For truly off-the-shelf steering, users must still collect a large amount of unstructured interaction frames to fit a task-specific dynamics model.\n- One essential baseline not included in this work is classifier-free guidance, which has been proposed in prior work for goal-conditioned policies [1,2] and works quite well. Like DynaGuide, this also allows specifying multiple positive and negative goals.\n- DynaGuide’s stochastic sampling loop increases per-step computation by roughly $M\\times$ relative to a conventional goal-conditioned policy, resulting in significantly slower inference. This overhead should be discussed in the paper. By contrast, the sampling baseline’s need for multiple forward passes per step is not as serious a drawback since those passes can run in parallel, so its wall-clock latency should actually be much less than DynaGuide’s inherently sequential guidance routine. Apart from a discussion, including some wall clock time comparisons during inference would strengthen the experiment sections.\n\n*[1] Reuss, Moritz, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. \"Goal-Conditioned Imitation Learning using Score-based Diffusion Policies.\" In Robotics: Science and Systems. 2023.*\n\n*[2] Ajay, Anurag, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. \"Is Conditional Generative Modeling all you need for Decision Making?.\" In The Eleventh International Conference on Learning Representations.*\n\nQUESTIONS:\n- The guidance metric in Eq. 2 compares the latent future state $z_{t+H}$ with desired goal state $g$. This would only make sense if the goal $g$ is achievable within $H$ steps from the current state. Since this is generally not true, I am not sure why this provides a meaningful guidance signal. In fact, it may actually be detrimental since the agent can try to reach some unrealizable state to minimize distance to goal within $H$ steps.\n- The value of $H$ is not explicitly mentioned, but appendix line 951 specifies that the policy produces action sequences of length 16 - does that mean that $H=16$ for all experiments? Why was this specific value chosen?\n- Why can’t position guidance be used for the moving objects experiments? In [3], the authors use block stacking and kitchen rearrangement tasks - what makes the cube moving experiment fundamentally different from these tasks?\n- The goal-conditioned baseline is trained by using the last observation of each trajectory as the goal. However, in lines 218-220, the authors mistakenly cite some papers which use goal relabeling techniques (such as hindsight experience replay [4]) to improve training of goal-conditioned policies. This has shown to work in visual domains as well [5,6]. Then, despite referencing such works, why were these techniques not used when training the goal-conditioned baseline?\n- The dynamics model in Eq. 1 predicts a future state $H$ steps away from the current state and action chunk, which is tractable only for deterministic environments. I wonder if the authors have any thoughts on extending this approach to the stochastic setting?\n- Typing error in Algorithm 1: the stochastic sampling loop does not correctly apply $M$ denoising steps at the same noise level $k$. See lines 6-9 in Algorithm 1 of [3].\n- Typing error in Algorithm 1: line 1 should be “positive guidance conditions $g^+$, negative guidance conditions $g^-$”\n\n*[3] Wang, Yanwei, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Perez-D'Arpino, Dieter Fox, and Julie Shah. \"Inference-Time Policy Steering through Human Interactions.\" arXiv preprint arXiv:2411.16627 (2024).*\n\n*[4] Andrychowicz, Marcin, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. \"Hindsight experience replay.\" Advances in neural information processing systems 30 (2017).*\n\n*[5] Nair, Ashvin V., Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. \"Visual reinforcement learning with imagined goals.\" Advances in neural information processing systems 31 (2018).*\n\n*[6] Florensa, Carlos, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Riedmiller. \"Self-supervised learning of image embedding for continuous control.\" arXiv preprint arXiv:1901.00943 (2019).*\n\nLIMITATIONS:\nThe authors have briefly discussed the limitations of their work with respect to the type of guidance provided and lack of any mechanism to guide how to achieve the desired outcome. There are several important limitations that have been left out, including the computational cost of DynaGuide compared to baselines, the fact that it seems limited to deterministic environments, and that the base policy must already be able to solve the task in order for guidance to elicit the desired behavior.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDynaGuide proposes to guide diffusion based robot policies by learning an independent dynamics model which influences the gradient field of the diffusion model.\n\nThis approach is evaluated on three simulated tasks using CALVIN and a real-robot task and compared to goal-conditioning and position guidance approaches.\n\nIn addition to help guide achieving desired behavior, this approach works with pretrained, off-the-shelf policies, with many potential applications for diffusion based robot learning.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper is very well written and easy to understand.\n- Solid motivation and theoretical foundation.\n- Multiple experiments in simulation and on real robot.\n\n### Weaknesses\n**I am worried about the data split for the experiments**. The details are a bit hidden in Appendix B4:\n- The base policy is trained only on the CALVIN-D dataset, but the dynamics model is trained on ABCD. As the other baselines are not detailed, I have to assume they were also only trained on CALVIN-D.\n\n    I am not entirely convinced that this is a good basis for a fair comparison regarding the generalization capabilities of the different baselines, as the guiding model has seen more data. I would welcome some discussion on this issue.\n\n- Additionally, I would welcome an evaluation in the more common settings of ABC-D and ABCD-D (or D-D) on CALVIN. \nThis could help to estimate if the guiding model benefits from being trained once on a large dataset and guiding specialized policies, or alternatively, if both models basically require access to the same dataset anyway, to also be trained on the same data.\n\nQUESTIONS:\n## Main Question:\nRegarding CALVIN Datasplit (see Weaknesses). I do not believe running the full ABC-D ABCD-D CALVIN benchmark is reasonable to demand as a reviewer. I would welcome any results with the DynaGuide model trained only on D, as the other policies, if results are available. \n\nOverall I am already very impressed with the method and hope that answering these questions will lead to even higher impact in the research community.\n\n### Minor Questions to help improve my understanding:\n- I am confused by the CALVIN underspecified objective. I assume any highlighting in Figure 3 is just for illustrative purposes? So what does it mean to \"randomize robot states (and other states)\"? Because I am quite surprised by goal-conditioning's performance drop.\n    - Could DynaGuide guidance counteract BC's performance drop? Such an experiment would also more nicely highlight DynaGuide capabilities to work with different, \"off-the-shelf\" policies.\n- On multi-objective guidance: Am I understanding correctly, that each objective is used to produce a guidance signal independently, which is then combined across all objectives, for example by addition? Or does the Dynaguide model actually receive a set of objectives at once?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present DynaGuide: a method for steering a base diffusion policy to desired task outcomes using the gradient feedback from an additional predictive dynamics model which matches positive and negative goal visual references.\n\nThe overall approach is illustrated in Figure 2. An additional dynamics guidance model is included to predict future latent state outcomes (trained by equation 1, page 4). Using these predicted outcomes, a denoising guidance signal is derived (equation 3, page 5) based on the action gradient with respect to a guidance metric $\\mathbf{d}$ (equation 2, page 5). The guidance metric encodes positive $\\mathbf{g^+}$ and negative $\\mathbf{g^-}$ image references for visual scenarios to attain and avoid respectively. The authors hypothesize that their novel steering approach is well suited to finding under-represented samples and meeting multi-objective scenarios.\n\n__Novelty and Contributions:__\n- Novel guidance signal approach for diffusion models using gradient from a dynamics prediction model to steer the denoising process\n- Versus existing baselines, their method boasts improved performance in achieving multi-goal objectives and under-represented behaviours.\n\n__Experiments:__\n\nA number of experiments explore the performance and properties of DynaGuide using a robot arm in the CALVIN environment [30].\n\nIn experiment 1 (section 4.1, figure 4 top) their approach fails to beat a more conventional goal-conditioned policy when interacted with standard articulated parts, however in experiments 2 (section 4.1, figure 4 bottom left) and 3 (section 4.2, figure 4 bottom right), their method significantly outperforms other baselines for movable and under-specified objects. The authors assert that this is because their method is more robust to situations that would normally be out-of-distribution for a normal goal conditioned policy given that goal, observation, and action inputs are all individually in-distribution to their modular setup while also using additional guidance signals in the latent space (described on lines 278-285).\n\nIn experiments 4 (section 4.3) and 5 (section 4.4) they also illustrate how their method is suitable for multi-objectives and its ability to find under-represented scenarios versus a sample based DynaGuide variant (GPC) as the sample-based method is restricted by the frequency of the desired behavior in the dataset being rare in this case.\n\nFinally, in experiment 6, they test their method with a real robot versus the base policy. While the base policy is limited to the representative behaviors in the dataset, their policy effectively steers towards the conditioned goal.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths:__\n\n- __Novelty and performance (originality and significance):__ Relatively novel and modern approach adapting recent diffusion models to a planning context using steering with primary advantages in executing multi-goal objectives and under-represented behaviours. Method outperforms state of the art baselines. See __Novelty and Contributions__ section above for more details.\n- __Pertinent use cases (significance):__ As discussed in the experimental section, the proposed method excels in multi-goal and under-represented behaviours. The usage for under-represented behaviour may be especially important for real world applications such as robotics where training data may be more difficult to gather and thus some behaviours may have limited representation in the dataset.\n- __Experimental rigour (quality):__ Experiments are done well for the most part (quality): Various aspects of the model were tested and modern relevant baselines compared against. Error bars are included. Through various experiments, they well illustrate the advantages of the proposed method compared to existing approaches for multi-goal objectives and under-represented behaviours (see __Experiments__ section above for more details). Ablation and parameter sweep results can be found in appendix A.2 with intuition concerning the parameter selection. However, the real-world experiments were somewhat lacking (see next section). \n- __Manuscript clarity:__ The manuscript is well written and contained sufficient detail.\n\n__Weaknesses:__\n\n- __Real-world experiments baselines (quality):__ The real-world experiments (experiment 6, section 4.5) only compared a base diffusion policy versus the full DynaGuide model. No other baselines were included (quality). This is a relatively weak baseline which does not seem to have any goal conditioning? I have asked for clarification regarding this issue in the questions section.\n- __Additional parameter tuning (significance):__ The method modulates between a base diffusion policy and task-based goal arrival (equation 3) weighted by a scaling guidance parameter. Selection of this scaling parameter might require additional tuning to achieve a reasonable policy versus other more direct optimization methods.\n\n__Current Assessment and Suggestions for Improvement:__\n\nI have currently marked this work as borderline accept. I think the submission is of good quality (based on the discussion related to strengths and weaknesses above) however I would of appreciated more baselines comparisons in the real-world experimentation section.\n\nQUESTIONS:\n- Can the authors clarify why additional baselines were not included in the real-world experiment section (experiment 6, section 4.5)?\n\nLIMITATIONS:\nYes. Limitations are listed in section 5.1 (lines 353-360).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}