DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance
Authors: Max Du
Shuran Song
Venue: NeurIPS 2025 poster ()
Keywords: robotics
diffusion policy
generative model for robotics
inference-time steering
Primary area: applications
TL;DR: We steer pretrained robot diffusion policies by using a dynamics model to guide the action diffusion process, allowing us to modify robot behavior without changing its policy weights or requiring goal-conditioning.

Abstract: Deploying large, complex policies in the real world requires the ability to steer them to fit the needs of a situation. Most common steering approaches, like goal-conditioning, require training the robot policy with a distribution of test-time objectives in mind. To overcome this limitation, we present DynaGuide, a steering method for diffusion policies using guidance from an external dynamics model during the diffusion denoising process. DynaGuide separates the dynamics model from the base policy, which gives it multiple advantages, including the ability to steer towards multiple objectives, enhance underrepresented base policy behaviors, and maintain robustness on low-quality objectives. The separate guidance signal also allows DynaGuide to work with off-the-shelf pretrained diffusion policies. We demonstrate the performance and features of DynaGuide against other steering approaches in a series of simulated and real experiments, showing an average steering success of 70% on a set of articulated CALVIN tasks and outperforming goal-conditioning by 5.4x when steered with low-quality objectives. We also successfully steer an off-the-shelf real robot policy to express preference for particular objects and even create novel behavior.

DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance
Maximilian Du Stanford University maxj du@staffortdddddcomm edu
Shuran Song Stanford University shuran@stanf ord.. edu
dynaguide.gihuub.i
Abstract
Deploying large, complex policies in the real world requires the ability to steer them to fit the needs of situation. Most common steering approaches, like conlicoonii require training the robot policy with distribution of test-time objectives in mind. To overcome this limitation, we present DynaGuide, a steering method for diffusion policies using guidance from external dynamics model during the diffusion denoising process. DynaGuide separates the dynamics model from the base policy, which gives it multiple advantages, including the ability steer towards multiple objectives, enhance underrepresented base policy behaviors, and maintain robustness on low-quality objectives. The separate guidance signal also allows DynaGuide to work with off-the-sheef pretrained diffusion policies. We demonstrate the performance and features of DynaGuide against other steering approaches in series of simulated and real experiments, showing an average steering success of 70% on set of articulated CALVIN tasks and outperforming goal-conditioning by 5.4x when steered with low-quality objectives. We also suc- cessfully steer an off-the-shelf real robot policy to express preference for particular objects and even create novel behavior. Videos and other visualizations can be found on the project website: https://dynaguuie, github.io
Pre-trained Policy with Diverse Behaviors
Dynamics Model Trained Diverse
Introduction
The rise of large datasets and expressive policies has enabled complicated skills robots like folding clothes [3, 55], making sandwiches [44], and washing dishes [7]. When these large policies deployed in the dynamic real world, we face the chal- lenge of steering them match the needs of specific scenario. This means finding a subset of the policy's 's behaviors that ap- propriate for the scenario [48]. The policy's 's complexity means that ideally accomplish this steering without needing to retrain the policy, sample excessively from it. or anticipate all the possible steering dur- ing policy training.
DynaGuide
Press Button Steering Single
Switch Steering Steering Away From
Don't
Multiple
Figure 1: DynaGuide steers pretrained diffusion policies by adding guidance from a dynamics model into the action denoising process. This dynamics dynamics-based guidance can take diverse behavior base policy and steer it towards single hehavior (left), multiple behaviors (middle), and ing hehavior (right) all without fine-tuning.
Most existing works for policy steering, however, rely least of those assumptions. Even language or goal- conditioned policies are trained on of
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

goal distributions, which requires foresight for the goal distribution that satisfies all future steering needs [45]. In our work, we distance ourselves from these assumptions. We propose separating the steering forces from the base behavior policy and combining the influence from the steering model with the strong action prior of the hase policy to generate guided behaviors that satisfy inference-time objectives while leveraging the base policy's 's existing skills.
Steering robot behavior requires understanding how current actions influence future outcomes, a feat well-suited for dynamics models [39, 16]. To adapt dynamics model for policy steering, we craft a latent visual dynamics model that predicts the final or far-future observation of trajectory given the current observation and action, allowing it to understand long-horizon dependencies. This dynamics model learns from unstructured environment interaction data and operates on visual observations projected into the expressive DinoV2 latent space [37] previously shown to be useful for robot planning [56]. In this latent space, we can compute the distance between the predicted future observation and observations of other desired/undesired outcomes, creating a differentiable metric that encompasses the guidance objectives.
Previous methods that steer policies with external metrics will sample actions from the policy and pick the action that best satisfies the metric [39, 34]. Instead, our approach leverages the denoising stochastic process of diffusion- dynamics diffusion guidance in other applications [53], we take the gradient of the metric through the dynamics model and use this dynamics gradient to directly modify the action denoising process.
policies. Taking inspiration from classifier guidance [9] and
This separate dynamies model and the active diffusion guidance make up DYNAGUIDE, new way of steering pretrained diffusion policies towards complex objectives. This design offers several key advantages over existing steering approaches:
Flexible Steering Structure. Goal-conditioned formulations have rigid input structures that cannot be modified during inference-and most policies accept only a single goal condition. DynaGuide ac- cepts collection of guidance conditions encompassing any number of positive and/or negative objectives ($4.3).
Increased Steering Robustness. Goal-conditioned approaches require training the input condi- tion alongside the policy, causing out-of-disttibution (OOD) failures when new or lower-quality objectives are introduced during inference. The DinoV2 embedder and separation of guidance and policy in DynaGuide means that no individual part of the system is OOD under this same situation, and the averaging effect of the guidance metric (Eq. 2) means that DynaGuide still extracts meaningful guidance signal ($4.2). The additional experience present the dynamics model also enables guidance towards novel behavior ($4.5).
Plug-and-Pla Modularity. Unlike fine-tuming approaches, DynaGuide only modifies the inference process of the diffusion policy, allowing different dynamics models to be swapped in and out without changing the base policy. This modularity also allows an off-the-shelf policy to be steered without further modifications ($4.5).
Enhancing Underrepresented Behaviors. Sample-based steering approaches [39, 34] select the best action from a set of proposed actions by the base policy. Not only does this require many inferences of the policy per step. these sampling approaches also struggles with expressing behaviors that are underrepresented in the base policy ($4.4). contrast, our Dynamics Guidance directly influences the denoising process. Through one denoising sequence, we guide the action towards the specified objectives, even if the behavior is less common in the base policy (64.4).
To investigate these claims, we conduct five experiments using simulated CALVIN environment tasks [30] and three experiments on real robot arm. We show that DynaGuide successfully guides the base policy up 70% of the time ($4.1) and outperforms goal conditioning by 5.4x on lower quality objectives ($4.2). It successfully amplifies underrepresented behaviors over sampling methods ($4.4) and accommodates multiple positive and negative objectives ($4.3). DynaGuide is also successful on an existing real robot policy, achieving up to 80% guidance success and even creating novel behavior ($ ($4.5). We will make code and collected data publicly available. Videos and visualizations can be found our website, https:/dynaguiid..enm iitthugg
2 Related Works
2

---

Conditioning Policies on Goal Representations. A common approach that influences robot behavior goal conditioning, where the policy is given a representation of desired outcome. The common representation is natural language [5, 57, 23, 36], which can also be used as steering mechanism for cuman-in-thh-llee execution [45, 8]. Language-conditine supervision to obtain and use the language labels, and it may be difficult to specify the exact steering needs through language alone. Other methods with fewer assumptions can Iearn goal-conditioedd policies from trajectories by conditioning
policies require special data and/or
learned latent embeddings [21, 28] or using future observations directly as the goal [41, 4, 38, 10, 2]. Like DynaGuide, these conditioned policies learn to accomplish the outcome represented by the goal observation. However, unlike DynaGuide, these approaches require training a policy that can take the goal condition as the input. DynaGuide's ' external dynamics model means that it works on top of any diffusion policy. It also supports multiple desirable and undesirable outcomes, whereas the observation-base goal conditioning typically supports only one desirable outcome. In our experiments, we compare the performance of DynaGuide with an observation goal-conditioned policy.
Leveraging Dynamies Models to Im- prove Robot Policies. Models can be trained to predict future states through la- tent dynamics by taking in an observation latent, action(s), and outputting predicted future latent state [42, 31, 16, 14, 15, 56, 25]. These latent dynamics commonly used for training reinforcement learning agents by simulating trajectories through the dynamics model [16, 14, 15, 40, 24] and have seen success on real robots [51]. They can also be used for Model Predic-
Ours
PG
GC
GPC
Untrained Goals (54.2)
X
Movable Objects ($4.1)
x
/
Multiple Conditions ($4.3)
/
Enhance Rare Behavior ($4.4)
/
x
Table I: Method Ability Comparisons. Goal-conditioned (GC) and sampling methods (GPC) [39] can steer policy be- havior, but GC is not prepared for untrained goals, and rare behaviors are challenging for GPC. External guidance meth- ods address these shortcomings. and unlike Position Guidance (PG) [50], DynaGuide (Ours) can guide towards complicated
tive Control (MPC) to generate trajectories directly [56, 43, 17, 13, 12, 47] and com- objectives without precise coordinate input.
bined RL/Planning approaches [18. 19]. Especially relevant to DynaGuide are approaches that use dynamics models to filter samples from policies according to predicted value [34]. VLM feedback [52], or engineered reward [39]. Like some of these prior works and especially Dino-WNy [56]. DynaGuide uses a transformer-bast based dynamics model to guide action creation. However, instead of sampling from the base policy, running RL, or MPC, DynaGuide steers the trained base policy directly through the action diffusion process.
Inference-Time Steering, After a policy has been trained, its behavior can still be changed without modifying its weights or requiring goal conditioning [48]. Safety value functions can intervene with a recovery policy or request human assistance when abnormal behavior predicted [35, 27. 11]. Diffusion policies can be steered by skewing the initial noise distribution
[49] or applying classifier- free guidance [20] that influences the action diffusion process [41] or the inverse-dynamies planning process [1]. While classifier-free guidance is effective for steering. it still requires training the policy on supplied conditions, akin to goal conditioning. The alternative steering method is classifier guidance, which leverages an external classifier to influence diffusion generation process [9]. Classifier guidance in diffusion policies is akin to seeking an optimal trajectory defined by the classifier [22]. Applied to robot policies, classifier guidance can apply post-hoc constraints [32] or seek objects near a human-supplied keypoint [50]. DynaGuide a classifier guidance approach, but it seeks to guide the policy with more complicated objectives using feedback from trained dynamics model acting as the classifier. Dynamics-bass diffusion guidance has succeeded in other applications like hardware generation [53] and DynaGuide brings it to robot policies.
3 DynaGuide Method
As shown in Fig. 2, we consider the problem of steering a diffusion policy To during its inference process. We are given a set of guidance conditions g+ Ug- which are represented as image observations that contains partial state information. Some guidance conditions show desirable outcomes g* + [9- ...9, and others are undesirable outcomes g = {9r :..9. J. Using G. we want to create a T/ (alo) such that for every a ~ (oo))) the probability p(g (or, a) is as high as possible and the probability lot, a) is as low possible. To accomplish this, we train
3

---

Actions
Guidance Conditions
(Noised)
6F2
Denoising
Gradient
Protrained Policy
Ot
Dinovz
Dynamics Model
Guidance
d2
+
Vodd
Mode
Selected
a
Mode
(A) DynaGuide
(B) Dynamics Model Guidance
(C) One Denoising Step
Figure 2: Achieving Dynamics Guidance. (A): DynaGuide combines action denoising gradients 6p from pretrained policy with guidance gradient V a. that increases the likelihood of accomplishing a set of guidance conditions G. (B): Inside the guidance module, dynamics model predicts future outcomes & and compares them to guidance conditions g (desired / undesired outcomes). We use the latent distances d to define guidance metric d (Equation 2) and take the gradient get the guidance signal v.g. used by DynaGuide. (C): An example of one denoising step. The pretrained policy seeks behavior modes in the data, while the guidance gradient selects particular mode.
dynamics model and use the model approximate these probabilities ($3.1) 1) and then use the gradient of that dynamies model to create by modifying its action denoising process ($3.2).
3.1 A Dynamics Model Capable of Guidance
Creating model to approximate p(g+lo a) and p(g- needs to predict future outcome ô from current observation and action sequence 2) we need to compare 0 to guidance conditions. Here, can be large value H can be the end of the trajectory.
lo., a) has two requirements: 1) the model
Past works on predicting future observations for planning and reinforcement learning have used latent dynamics models to satisfy these two requirements. These dynamics models ()())); a) operate a learned latent space 2, o(o,). The recent work of Dino- WM [56] demonstrated the usefulness of latent distances in the pretrained DinoV2 image embeddings [37] for planning. Taking inspiration from Dino-WM, we leverage the same DinoV2 image latent space for our observation comparisons. Concretely, we use the patch embeddings from Dino V2 as
Φ(or) and train
transformer (a zt) to predict a latent outcome representation & H. Because á is frozen and we have access to full trajectories of data during training, it is sufficient train he through regression objective (Eq. 1). For more details, refer to Appendix B.
C(or, O a) [|Ø(0+H)) ho(o(o)),
With this dynamics model trained, we define a guidance metric that compares the predicted outcome with the guidance conditions. Given
Algorithm 1 DynaGuide (Inference-Tima)
Input: Guidance Conditions g+ .g. g Dynamics
model h9
one guidance condition 9; we want approx- imate p(g, (o: a), which can be done by com- paring the predicted outcome ê to g. The condition 9, a visual observation, so we project into the same latent space 21 = =(gg") This allows us to directly compare 2, and 4+H-- If we roughly assume that the latent space gaussian in structure, a reasonable ap- proximation of log p(g," Joz, a) is proportional
2: Input: Action denoiser (a, o), current obs
3: a +- Sample from (0,
4: for i K to do
& Action Denoising
5: for i in to M do
Stochastic Sampling
< (a (aa (af.)) o.)
dl Eq.
8:
i(a h 0.) t c s a V..d
9:
if < M then
10:
a² + Denoise a using è
else
to - 2, I1 metric found in similar
12:
+ Denoise a using è
forms elsewhere for latent reinforcement learn-
13: end
ing [54, 33] and planning [56].
14: end for
15:
end for

---

When we try to maximize log p(g" loz, a) across the set of guidance conditions g". we can try to maximize the combined probability of achieving each outcome log >; p(g," lo, a). The negative is true for the undesired outcomes. We use our approximation of log p(g," [ot, a) and variance hyperparameter a to get the guidance metric d(g gg Ot. a) (Eq. 2). The log-sum-exp acts as soft maximum, feature useful for diverse objectives ($4.3) and lower quality guidance conditions ($4.2). For a more detailed derivation of and explanation of design choices, refer to Appendix B.3.
-1(((tt) ho(p(or), a)
-1lo(g7) ho( a)
d = log
Σ exp
- loog Σ exp
(2)
We train the dynamics model using diverse robot interaction data. For the Calvin experiments $4.1 14.1 we use existing human-collected play data from the benchmark. For the real-world experiments $4.5, we use open-source data collected on the UMI interface [6]. Because we will be querying the dynamics model with noisy actions during the guidance process, we augment the training by adding gaussian noise to the actions using the same noise scheduler used during inference time, with a noise step picked from geometric distribution. For more details about implementation, see Appendix B.
3.2 Guiding the Action Diffusion Process
To sample an action from a diffusion policy rT-1o,), the policy predicts a sequence of denoising steps e(a*, ot). that moves an initial noise sample a A a denoised action sequence a". Diffusion models can be guided by a classifier p(y|a aa towards a particular set of samples that maximize p(yla) [9]. The guidance signal comes from the gradient of the classifier V. log p(yla) and combined with c(a", o.) during the denoising process.
In our case, y is the guidance conditions, and the log (y)aa is our derived metric d in Eq. 2. Like past work [53, 41], we use the Denoising Diffusion Implicit Models (DDIMs) [46] as our sampling method. Under DDIM, the combined denoising and guidance signal is
i(a", o.) c(aaa - sV oi a :g o0,
where oi = II ar, on is the noise scheduler, and s is a guidance scaling parameter [9]. The higher the s, the stronger the signal adhere to the guidance requirements. However, past work has discovered that higher also create less smooth or even incoherent trajectories [50], creating delicate balance between trajectory validity and adherence guidance conditions. This phenomenon happens because V d can pull e out of distribution, leading to erroneous noise predictions that compound as the denoising process continues. To resolve this, Wang et al. [50] proposed the Stochastic Sampling solution of running each denoising step h multiple (M) times to stabilize the guidance signal's s influence on the action denoising process through MCMC sampling. Empirically, this trick allows to be pushed higher to increase guidance success without sacrificing trajectory validity. See Algorithm 1 for an overview of DynaGuide. We investigate these hyperparameters in Appendix A.4.
Experiments
To understand the features, benefits, and limitations of DynaGuide, conduct five sets of experiments on the Calvin environment (Figure 3, 1-4.4).. To demonstrate its practical feasibility on real robots, steer a publicly available robot policy in real task using DynaGuide ($4.5). We also conduct experiments on a toy 2D navigation environment, which can be found in Appendix A. Wherever relevant, we compare DynaGuide
representative set of baselines.
Base Policy. This is the diffusion policy [7] that we steer with DynaGuide. This policy offers a fairly uniform distribution across valid behaviors.
SynaGuide-Sampling (GPC) [39]. Instead of diffusion guidance, we sample from the base policy multiple times and pick the action sequence that best satisfies metric d (Fq 2), an idea demonstrated in GPC-Rank with engineered reward functions [39]. We re-implement this idea with our dynamics model and use this bascline to explicitly test the benefits of diffusion guidance.
5

---

Articulated
Movable Objjccts
Randomize Objjects
Inceract with Some Object but Not Others
Robot
Fully-Spacified Objective
Under-Specified Objective
Multiple Objectives
Figure 3: Experiment Setup. In the CALVIN simulator [30], we propose four experimental setups designed to showcase DynaGuide and its advantages over other stecring approaches. First, we test performance with high quality outcome observations guidance conditions (Fully-Specifet Objective). Next, we reduce the guidance condition quality by randomizing robot states and other states not relevant the target object (Underspecified Objective). Finally, we look at how we can guide the base policy in complex ways, including achieving multiple behaviors and avoiding behaviors (Multiple Objectives).
Goal Conditioning. We train a policy to take an additional visual observation as the goal. Past works in visual goal conditioning have used future observations as goals [2, 36, 41J. J. In our baseline. we take the common approach of using the last observation of each trajectory as the goal. We compare this model to DynaGuide by sampling from g* as our goal during inference-time. We also test an intermediate conditionn baseline Appendix A.2.
Position Guidance (TTPS) [50]. Past work has achieved diffusion policy guidance that steers the robot to particular location in 3D space supplied by human. For objects that do not move, replicate this guidance by finding the average location of the robot after accomplishing the desired behavior and using position guidance that location.
4.1 Experiments 1 & 2: Steering in Complex 3D Environments
Our first two experiments examine how DynaGuide can steer robots to particular behaviors in the CALVIN environment [30]. The CALVIN setup allows robot to interact with desks with drawers, switches, buttons, and cabinets (Fig. 3). It can articulate these objects and also rearrange set of three colored cubes that are placed randomly
the table. For these two experiments, focus on Guidance Conditions g" that represent desired outcomes, taken as the last observation of demonstration trajectories showing the target behavior. We use these same observations for the goal-conditioned baseline by sampling 9,t E gg for each rollout as the goal.
In this set of experiments, we are interested in the ability of DynaGuide to steer the base policy towards behaviors that interact with the articulated table elements (ArticulatedParts) and random- ized cube objects (MovableObjects). Overall, for every target behavior across both experiments, DynaGuide significantly increases the frequency of the target behavior over that of the base policy (Fig. 4, Top, Bottom-Left).
On ArticulatedParts, DynaGuide boosts the base policy's 's behavior by 8.7x and achieves an average target behavior success of 70%. Because the objects in ArticulatedParts. stay in place, we can run the ITPS baseline [50]. Using position guidance boosts the target behavior over the base policy, but the reasoning abilities of the dynamics model allow DynaGuide to outperform it across the tasks (Fig. Top). The location consistency of ArticulatedParts also means that the g outcomes closely match the true outcome of the current environment if the desired behavior is executed. Therefore, goal-conditioned baseline is in-distribution and performs near perfectly. This is expected, as the goal-conditioned model was trained for this exact setup.
The MovableObjects experiment poses a new of challenges. Instead of being fixed in location, the cube objects are small and randomly arranged every environment reset. All steering performances decrease in MovableObjects, but two notable differences arise. First, the GPC bascline performance drops to the base policy. Sampling approaches like GPC rely on actions drawn from the base policy, If these actions have high variance, then the selected actions may also have high variance. Compared to the steady guidance of DynaGuide, the increased variance leads to more erratic actions and impacts performance on precision tasks. The second difference is the drastic drop in goal conditioned performance. Unlike ArticulatedParts, the randomized cubes mean that not all outcomes in g are attainable the current setup. For example, some outcomes in g+ interact with the blue cube on the left side of the table, while test-time setup might have the blue cube on the right side. This

---

Articulated Parts
10
37
8.7x boost
0.4-
Succans Rato
Button
Buttun Ot!
Switch
Switch of!
Drawer
Drower close
Boor
Door
Avcrage
0.4
Movable Objects
0.41
Underspecified Objectives
2:1x boost
5:4x
32B
boost
0.3
0.0
Red Block Novs
Blus 5 Nove Prrk slock Move
Average
Button
Switchh
Drawer
Door
Average
DynaGuide
DynaGuide Sampling
Gool Conditioning
Position Guidance
Base Policy
Figure 4: Steering Ability and Robustness in the Calvin Environment DynaGuide enhances the target behavior (horizontal axis) significantly across all experiments (Section 4.1). The goal conditioning baseline performs very well on clean fixed articulated setup, but it drops steeply with lower goal quality while DynaGuide remains more robust (Section 4.2). For more precise tasks with movable cube objects, the active guidance in DynaGuide outperforms a sampling-based approach with the same dynamics model (Section 4.1)
mismatch means that the goal conditioned policy is out of distribution and is affected more than DynaGuide. We explore this phenomenon the next experiment.
4.2 Experiment 3: Robustness to Lower Quality Guidance Conditions
The cube randomization in MovableObjects created mismatch between desired outcome observa- tion and attainable outcome the current environment, a challenge that disproportionally impacted goal conditioning over DynaGuide. While MovableObjects had an inevitable mismatch, explore a deliberate mismatch in this experiment that models a more practical use case of inference-time steering.
The two prior experiments had used g drawn from trajectories where the robot achieves the desired behavior. For each guidance condition, the combined robot's 's position and the target object's position fully indicate the desired behavior. For practical deployment, easier to take pictures of the desired scene without collecting trajectory demonstrations placing the robot in the correct position. We mimic this requirement Underspeciffed to the desired pose and randomize all other objects, including the robot. We use images from this setup as g+
where we set the target articulated table part
As expected, the lower quality guidance conditions pushed the goal-conditioned policy out of distribution, leading to average success rates below 10% (Fig. 4, bottom right). contrast, the dynamics model guidance approaches (DynaGuide and DynaGuide DynaCuuieeSSmmpling) were still able to increase the target behavior over base policy rates and outperformed the goal conditioned policy by 5.4x average. DynaGuide also performed more consistently than its sampling counterpart, further illustrating the importance of active guidance. We attempt to strengthen the goal-conditioned baseline in Appendix A.2 by adding training data and implementing intermediate coal-conditioning algorithm, but none these approaches yielded consistent improvements over the tested goal- conditioned baseline here (Appendix A.2).
These under-specified objectives showcase the benefits of separating the policy and guidance. Al- though the guidance conditions are still mismatched for DynaGuide, the inputs (g or, a) were all individually in-distribution the parameterized encoder, base policy, and dynamics model of DynaGuide. The mismatch, therefore, occurs in the latent space of large visual encoder. With nothing out-of-distributionn we face an easier challenge extracting meaningful signal from these latent distances. Fortunately, DynaGuide also makes use of all the conditions in the latent space for the guidance metric (Eq. 2). While each latent distance d; might be noisy and inaccurate, the combined metric will cancel some of the noise, leading meaningful guidance.

---

Multiple Objectives
Underrepresented Behaviors
Positive Conditions
Negative Conditions
1.0
Dyuuuidu, Sampliig
1.0
0.5-
DynaGuide
Success Rate
0.4
8.-
Sampling
64
P
Base Policy
02
B2
Buthon
Deor
Saties DR
NG Drawer
NO boce
NO Door
20
60
50
Percent Data Included
behavior
Figure 5: Multiple Objectives and Underrepresented Behaviors. DynaGuide able to steer the base policy towards multiple behaviors while minimizing other behaviors and failures (Left). DynaGuide is also able to avoid undesired behaviors by performing other behaviors successfully (Middle). On these complicated objectives and in lower data regimes (right), DynaGuide performs better than sampling approaches.
4.3 Experiment 4: Steering with Multi-Objectives
Separating the guidance and the policy improves the robustness of DynaGuide lower quality guidance conditions, and it also allows the guidance signals to be combined, steering the policy towards a multimodal of desired outcomes and even away from set of negative outcomes. This capacity is not possible for an unmodified goal-conditioned policy that takes a single observation as the goal.
In MultiObjectives, we add multiple target behaviors to g* and populate with undesired be- haviors. Good stecring will allow all represented behaviors in g be executed while minimizing behaviors in g" and overall behavior failure rate (c.g., doing nothing). Indeed, DynaGuide steers the policy towards multiple target behaviors with fair representation of each behavior (Fig. 5, Left), achieving up an average of 80% success with nearly no full behavior failures. When faced with avoiding behaviors in g" DynaGuide achieves nearly perfect success in avoiding the undesired behavior while executing a variety of other behaviors (Fig. 5, Middle). For an additional comparison against Classifier-Frer Free Guidance (CFG) [20], refer Appendix A.5.
Because DynaGuide and GPC both use the same dynamics model for steering, they can both steer to multiple objectives. However, the sampling approach of GPC had lower overall success of desired behaviors and allowed more execution of undesired behaviors, especially when trying to avoid outcomes in g (Fig. 5, Middle). When avoiding g GPC also produces more behavior failures where nothing is executed during the trajectory horizon. Sampling approaches like GPC rely on the base policy to readily produce actions that satisfy the objective. However, especially when the robot is close to an object, the base policy might only sample actions for one behavior, making this behavior impossible avoid. contrast, active guidance during the diffusion process can seek rare modes in the action distribution, leading to higher chance of accomplishing complicated objectives like the ones this experiment.
4.4 Experiment 5: Enhancing Underrepresented Behaviors
In MultiObjectives, we observed that the GPC sampling approach struggles to satisfy complicated objectives because it relies on adequate samples from the base policy. In UnderrepresentedBe haviors, test this feature explicitly. Trained robot policies will offer behaviors based its representation in the training data. However, when we steer a policy, we also want to steer into behavior modes that are underrepresented in the training data.
To simulate data underrepresentation,, we reduce the presence of Switch- -nn behavior in the training set of the base policy. As expected, the base policy offers more Switch- On behavior as more data is included (Fig. 5, Right). GPC also increases with more data representation, as can boost the behavior by selecting the right action. Most importantly, DynaGuide consistently outperforms its sampling counterpart these lower data regimes, achieving 40% success rate with only 1% of the original Switch- On behavior included in the trained base policy (Fig. 5 Right). By directly influencing the

---

Cup Preference
Hidden Cup
Novel Behavior
70%
7096
10%
60%
10%
15%
Policy
Guide Red
Guide Gray
Policy
Guide
Base Policy
Guide
Grey Cup
Red Cup
Computer Mouse
Figure 6: Real World Experiments On a cup arrangement task (Left, Center), we show that DynaGuide can guide pretrained robot policy to express preference over cup color. Using the same policy, we also show that we create novel mouse-grabbing behavior (Right) by leveraging the additional knowledge inside the trained dynamics model.
action diffusion process, DynaGuide can leverage the knowledge of the dynamics model seek areas in the action distribution that satisfy an objective, even if they are underrepresented.
4.5 Experiment 6: Steering Off-The-Shelf Policies with a Real Robot
In the simulated Calvin environment, we used existing data train both the base policy and the dynamics model. DynaGuide works with this setup, but it should also work with any pretrained diffusion policy and it should also work on real robots. To test these two claims, we set up a real robot experiment leveraging a publicly available pretrained policy [6] that picks saucer. We train the dynamics model using open-source data from the UMI collection interface [6] and set of demonstrations collected on the experimental setup. For more details about the setup, refer to Appendix B.5.
cup and places it on
We conducted three experiments on this real-world setup. The first two, CupPreference and HiddenCup, present the robot with different-colored cups equidistant from the robot's starting gripper. The base policy will select cup at random and place it on the saucer. Applying DynaGuide in CupPreference creates preference for cup color (Fig. Left), leading an average target behavior success of 72.5%. In HiddenCup the red mug is hidden behind the grey mug. The base policy will typically go for the closer grey mug, but the active guidance DynaGuide enables the robot find the red cup 80% of the time (Fig. 6 Middle).
Inspired by the performance of DynaGuide with data representation 1 1 ($4.4), tested the ability of DynaGuide to steer the pretrained policy towards novel behavior: touching a computer mouse. In NovelBehavior, the dynamics model was trained various object manipulations, including mugs and computer mice from other open-source datasets [26]. We used the same off-the-shelf base policy that was only trained arrange mugs. Although the steered policy still expressed preference for the mug, DynaGuide was able to double the number of interactions with the novel object.
Conclusion and Discussion
In this paper, we proposed novel method of steering pretrained diffusion policies by using separately trained latent dynamics model. We demonstrated the ability of DynaGuide to enhance target behavior performance across simulated and real experiments, outperforming baselines like position guidance and goal conditioning in some or all setups. Finally, we demonstrated that DynaGuide can be stcered achieve multiple behaviors, avoid undesirable behaviors, and enhance behaviors that were underrepresented in the training of the base robot policy. Real- world robot deployment requires robots to be highly steerable their skills, and DynaGuide shows one potential avenue for this ability.
5.1 Limitations and Future Work
DynaGuide can successfully steer pretrained policies towards particular objectives, but currently difficult to specify the method of achieving this objective. This limitation is due to the current form of

---

guidance conditions as observations that represent desired/undesired outcomes. There are other, more complicated guidance modalities, including language and kinesthetic demonstrations, Future work includes enabling multi-modal guidance conditions, which can hopefully enable very fine-grained guidance for all parts of the trajectory. In the future, we also hope to add an ability for the base policy to remember past guidance, consolidate the knowledge, and apply it automatically to new tasks.
5.2 Acknowledgments
Maximilian Du is supported by the Knight-Hennessy Fellowship and the NSF Graduate Research Fellowships Program (GRFP). This work was supported in part by the NSF Award #2 143601, #2037101, and #2132519, and Toyota Research Institute. We would like to thank ARX for the robot hardware and Yihuai Gao for assisting with the policy deployment on the ARX arm. We appreciate Zhanyi Sun for discussions for their detailed feedback on paper drafts and experiment directions. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
classifier guidance and all members of the REAL lab at Stanford
The Calvin Experiments (Sections 4.1 -4.4) were made possible with the Calvin benchmark codebase [30]. The diffusion policy was adapted from the Robomimic repository [29]. The dynamics model was inspired by the Bino-WM implementation [56] and leverages representations from Dino- V2 [37].
References
[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? URL http: //rxxv.. arg/abs/2211... 15657.
[2] Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Picter Abbeel, and Wojeiech Zaremba. Hindsight expe- rience replay. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. URL htttps::/procceedddiiiee newrips www..ps.ccpppprrrrreeeeee. ttteeerffileeppapee/eaa http://SSaaabaaaaaaaaaaaaaaaaeee
Abstractt. html.
[3]
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke. Sergey Levine, Adrian Li- Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. Pi-0: vision-language-actioo flow model for general robot control, URL attp://arxix: org/ abs/ 2410.2411
[4] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models, URL http: http: httpp://axxvvvvrpeeee hwtgg//aa////../// 10639.
[5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang- Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems XIX. Robotics: Science and Systems Foundation. ISBN 978-0-9993747--9-2. doi: 10. 1.5607//SSS.22233XXXX. .df..
URL http:
//www www.wroboti.co www.rrooocccprcceeedinieeoeeeeee wwwpiiinnnggggggggreeoe
[6] Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In Robotics: Science and Systems XX. Robotics: Science and Systems Foundation. ISBN 779-8-9902848-0-- doi: 10. 026607/RSS.2024.00 XX.055.. URL
org/ wwrgrrss00p004 pdf.
10

---

[7] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin CM Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi: 10.15607/RSS. 2023.XIX.2226
[8] Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, and Dorsa Sadigh. No, the right: Online language corrections for robotic manipulation via shared autonomy. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction, HRI '23, pages 93-101. Association for Computing Machinery. ISBN 978-1- 4503-9964 doi: 10.1 00.1145/368162.33517 3568162.3578623
URL https: //tp./ddl.. /dddlaa acm. org/doi/ /t////oi11...0 1145/
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image syn- thesis. In Advances Neural Information Processing Systems, volume 34, pages 8780- 8794. Curran Associates, Inc. URL httttss:/prooeeednnggg
httpps/ccppppp//221/////// Abstrcfffaaaattat..a....acaaac.. html..
[10] Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imi- tation learning. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc. URL http:: httpp::/ppooceeedingg neurips cc/paper fuaper_ffless files/ wtt..fs/pppprr//11///////// heab://B5aa/eea3366666666602222 Abstractt tatmlaact..mll....... .html.
[11] Maximilian Du, Alexander Khazatsky, Tobias Gerstenberg, and Chelsea Finn. To err is robotic: Rapid value-based trial-and-error during deployment. URL http::///arxii /ttp:////axvvvv// wtg.//twwwii.....aaaa.ooo..o..ot atgg/bbb/2065..// 15917.
[12] Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model- based deep reinforcement learning for Assion-based robotic control. URL http://arxiv. org. 1112200081.....
[13] Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 7786-2793, doi: 10. 1100/CCRA.2017.799922. URL htttp::/iieexplorrt weeeeeeppooeeeeeeeeeeeeeeeeeeeee wwgg//ccmmnn/798922/...
[14] Danijar Hafner, Timothy Lillierap, Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent imagination. URL http: http:///aavvvv/aa org/ http///bs//11...... 01603.
[15] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models, URL http:://ta:tttt http://wwwwwoor.///////// 2210 abs/2010.02193. 02193.
[16] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse con- trol tasks through world models. 64008059))647-633, -41586-025-08744-2. URL https://wwwtt httpp::/www.nature.. nature. Publisher: Nature Publishing Group.
ISSN 1476-4687 doi: 10. 1038/
[17] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: /8/ 1.088/1/.455,,1 2018.
[18] Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In JCML, 2022.
[19] Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control, 2024.
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. URL htttt:ttp//xiiiii abs/2201 12598.
[21] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-z: Zero-shot task generalization with robotic imitation learning. In Proceedings of the 5th Conference on Robot Learning, pages 991-1002, PMIR. URI. https: / pttope//prooeeddigg mlr, wwwpp//eeee//// v164//ang222ee html. ISSN; 2640-3498.
11

---

[22] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning. 2022.
[23] Gi-Cheon Kang, Junghyun Kim, Kyuhwan Shim, Jun Ki Lee, and Byoung- Tak Zhang. CLIP- RT: Learning sunguage-conditioned robotic policies from natural language supervision. URL http://anze /ttp://///xivv//// org //s////////// htab///411 0tt///211...000000
[24] Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic la- tent actor-critic: Deep reinforcement learning with latent variable model. In Ad- vances in Neural Information Processing Systems, volume 33, pages 741-752. Cur- ran Associates, Inc. URL https://pppt attpp://prroeeedingg http://rrroednnnnnrrrrpppppppppp neurips. ctciiiss.ccpppeeee cttccc//ppppa ww/ppappr/222//aaeeeeeeeee 4805515002235200011188000000 Abstract.ttttttttt..e .ttll.
[25] Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. URL http:// //axxvv./ htt///axvv..ooraaaa// wtwg/bs////......... 00200.
[26] Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotic manipulation. arXiv preprint 1784472410.18647 2024.
[27] Huihan Liu, Yu Zhang, Vaarij Betala, Evan Zhang, James Liu, Crystal Ding, and Yuke Zhu. Multi-task interactive rohot fleet learning with visual world models. In 8th Annual Conference on Robot Learning (CoRL), 2024.
[28] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Proceedings of the Conference Robot Learning, pages 11 1113-1 132. PMLR. URL 1ynch20a. html. ISSN: 2640-3498.
mlr. wtt.pp/sss/////
[29] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martfn-Martín. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.0322 2021.
[30] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: benchmark for language-conditionee policy learning for Long-horizon robot manipulation tasks. URL
wttp/////.rro//////////o.bbbsass 03227.
[31] Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers sample-efficient world models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevie::
[32] Utkarsh Aashu Mishra, Shangjie Xue, Yongxin Chen, and Danfei Xu. Generative skill chaining: Long-horizon skill planning with diffusion models. In Proceedings of The 7th Conference Robot Learning, pages 2905-2925. PMLR. URL https: httpps://proceedingg mlr. press/v229/ mishra23a. html. ISSN: 2640-3498.
[33] Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. URL http: /attp://axxiv.. /arxiv.ivvvv org/ abss/100f 04742.
[34] Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. Conference Robot Learning (CoRL), 2024.
[35] Kensuke Nakamura, Lasse Peters, and Andrea Bajesy. Generalizing safety beyond collision- avoidance via latent-space reachability analysis. 2025.
[36] Iman Nematollahi, Branton DeMoss, Akshay L Chandra, Nick Hawes, Wolfram Burgard, and Ingmar Posner. Lumos: Language-condiiinned imitation learning with world models. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Atlanta, USA, 2025. URL http: //ais. informatio uni-freiburg http:rg.apppblpiiisssssssss/e/ nematollahi25icre. pdf.
12

---

[37] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. URL http::ttpax/xaaa /ttp///www..rrabaaa/////aaaaaa www//b// http://344..7111 07193.
[38] Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A. Efros, and Trevor Darrell. Zero-shot visual imitation. In ICLR, 2018.
[39] Han Qi, Haocheng Yin, Yilun Du, and Heng Yang. Strengthening generative robot policies through predictive world modeling, URL http: www.aawwwwww..aa.....a 20000000.0002.......
[40] Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, pages 1154-1168. PMLR. URL Utt.htts://prrcedding html. ISSN: 2640-3498.
mlr. www.pprrsssssse http////////aaaa///aa///
[41] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal conditioned imitation learning using score-based diffusion policies. In Robotics: Science and Systems, 2023,
[42] Jan Robine, Marc Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world models are happy with 100k interactions. In The Eleventh International Conference Learning Representations, 2023. URL https://openrere
[43] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. 588(7839):004
ISSN 1476-4687.
URL doi: 1511038/41556--00-0000000
https: / //www/wwww/o/.o///// nature. cwt. com/articles/////ooeooococoaosee ht////arrceeese116...... 0atp://4416600000000000000 03050--3051--00 4. Publisher: Nature Publish- ing Group.
[44] Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchical iision-lngguage-attoo action models. URL http: /tttp://arxivv org/
19417.
[45] Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z. Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell at your robot: Improving on-the-fly from language corrections. arXiv preprint arXiv: 2403. 1008.19900 12910, 2024.
[46] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. URL httt:://aree org/ abs at//a/bs/20000 2010.. .22002.
[47] E. Todorov and Weiwei Li. A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005., pages 300-306 vol. doi: 10. 10.1109/ACC.2005. 14..ACC00.0....949999 1469949. URL ISSN: 2378-5861.
ieee. www..eeeeeeeeeeeeeeeeee...eeee.e wtgg/document/14690
[48] Masatoshi Uehara, Yulai Zhao, Chenyu Wang, Xiner Li, Aviv Regev, Sergey Levine, and Tom- maso Biancalani. Inference- time aligmment in diffusion models with reward-guide generation: Tutorial and review. URL http: / www.aa
[49] Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Scohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning, 2025. URL https:
hrrg/abs/2506 15799.
[50] Yanwei Wang, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang. Yu-We Te-Wei Chao, Claudia Perez-D' Arpino, Dieter Fox, and Julie Shah. Inference-time policy steering through human interactions. URL http: /ttt:://arxis org/ abs/2411. 16627.
[51] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Day- dreamer: World models for physical robot learning. Conference on Robot Learning, 2022.
13

---

[52] Yilin Wu, Ran Tian, Gokul Swamy, and Andrea Bajesy. From foresight to forethought: Vim- in-the-loop policy steering via latent alignment, 2025. URL 01828.
[53] Xiaomeng Xu, Huy Ha, and Shuran Song. Dynamics-guided diffusion model for sensor-less robot manipulator design. In 8th Anmual Conference on Robot Learning.
[54] Ruihan Zhao, Ufuk Topeu, Sandeep Chinchali, and Mariano Phielipp. Learning sparse control tasks from pixels by latent nearest-neighhor-guue guided explorations. URL http::/tp/ arxxv org/ abs/2302. 14242.
[55] Tony Z. Zhao, Jonathan Tompson, Danny Driess, Pete Florence, Seyed Kamyar Seyed Ghasemipour, Chelsea Finn, and Ayzaan Wahid. ALOHA unleashed: simple recipe for robot dexterity. In Proceedings of The 8th Conference Robot Learning, pages 1910-1924. PMIR, URL. http:::tt////ttt httpss////ccceeeeeeeeeeee mlr. press/ seness/v270// v270/22ao2005 html. ISSN: 2640-3498.
[56] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO- WM: World models on pre-trained visual features enable sero-shot planning. URL http: org/ 04983.
[57] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wa Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopallakrishaan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: Vision Dision-language-aee action models transfer web knowledge to robotic control. In Proceedings of The 7th Conference on Robot Learning, pages 2165-2183, PMLR. html. ISSN: 2640-3498. mlr. prwss/ /229// zitkooicch22a... html,
URL http::::ttttcceeeeeiieee
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: average steering success on articulated task $4.1. Goal condition improve- ment: $4.2. All other results in the introduction already point to sections in the paper results:
1-.15.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: The limitations are explicitly discussed $5 and also mentioned in other parts of the paper, including the results 1-.1...5.
14

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Although brief mathematical derivation is shown in $3.1 and fully worked out in $B.3, they are only meant show the reasoning behind the guidance metric (Eq. 2). This paper does not contribute theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Experiment setups are described in the main paper $4 and in full detail in $B. For the simulated results, the data is publicly available and instructions will be provided on how to use it. Code is provided the supplemental and cleaned version will be released in the very near future.
Justification:
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Data in the simulated CALVIN environment was taken from a public dataset [30]. the real-world setup. the pretrained policy is publicly available as well as the data [6]. An additional dataset was collected in the experimental environment, which will be released publicly in the very near future. working copy of the codebase will be included in the supplemental material, and cleaned codebase implementation will be supplied in the very near future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ct cc/
Seaidddee/ooeeSumissssssee sssisssssicccc for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //ni http://wwwl.lgggeeeeeeedeeeiiii/
for more details.
16

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experimental setting is summarized in and detailed in B. The detailed training, testing, and model hyperparameters described in B.1. The code instructions will contain other practical details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All simulation bars and plots contain confidence intervals, with details these intervals found $B.4. As mentioned in B.4, the robot results are resource-intensive and therefore do not have intervals. This is very common for real robot experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
17

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Details compute provided in $B.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http://eeurpppp.pee cttpssccccpbbbiaee ceabllcc//hiiiee eeGuideliness
Answer: [Yes]
Justification: We confirm the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in $C
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
18

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our paper uses publically available datasets and publically available trained policies for real robots. The dynamics model predicts furture states in robot environment and has no easy way of misuse or dual use.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Credits for datasets and codebases are appropriately cited throughout the paper and detailed Appendix D.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: A pre-release version of the code the supplemental material is provided with documentation that will be fully fleshed out in the near future.
19

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing was used for this project.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No study participants were used, and therefore no IRB was needed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Apart from assisting with visualization code and grammar checks, LLMS were not used in this project.
Guidelines:
20

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non non-standard components.
Please refer to our LLM policy (https : /ttp///eu.ppppepeee ( /LLM)
for what should or should not be described.
21

---

A
Additional Experiments, Visualizations, and Ablations
A.1
Steering in a Simple 2D Block Task
Early Decision
Decision
Furthest Cube
Policy
Guidannc: Blue
Base Guidance:
Guidance
Policy
Agent
Example of Guidance During Diffusion
Figure 7: The BlockTouch Environment. This 2D environment requires the agent (black circle) to navigate to colored target (square). The base policy picks arbitrary target, and DynaGuide steers it towards a particular color (A). This steering works express square color preference (B). The dynamics model also enables the base policy accomplish novel tasks not explicitly trained on the base policy, including picking target from a tight cluster of squares (C) and navigating past three close cubes to a far target (D). In visuals B D, the distribution of squares are indicated by the shaded regions. The average square location represented by the solid square.
As an additional way of understanding how guidance works in DynaGuide, we look at toy Block- Touch environment. square. We train the vision-based base policy and dynamics model synthetic data that shows navigation to squares in any location of any color. Then, during test time, we use the dynamics model to guide the policy towards cubes of one particular color. In Figure 7A. visualize the creation of an action chunk under dynamics guidance. In the initial noisier steps, the guidance signal and the denoising policy signal compete for the action's 's direction. In the example, the agent is nearly touching the yellow square, so the base policy wants to denoise the action towards the yellow square. The guidance wants move the agent towards the blue square. The vector of these two forces pushes the actions towards the blue square. In the later stages of denoising, the guidance and the policy signals start work together to craft the final action sequence that points the blue square. This guidance works in situations where early decisions are important for hitting the correct color (Figure 7B) and in situations where late decisions are important (Figure 7C).
Touch, the agent navigates a 2D environment and touches a colored
Even though the dynamics and base policies are both trained random navigation behavior, we can lever- age the dynamics model to accomplish novel tasks not explicitly present in the base policy. The Late Decision setup (Figure 7C) contains squares that are closer together than the training data, requiring high navigational precision. The Go to Furthest Cube setup (Figure 7D) places the target much further away than three squares, requiring deliberate navigation around the closer squares. The base policy goes the closer squares, but the dynamics model was able reliably guide the policy between the squares and square distributions and the black dot at the yellow end of the trajectory shows the first and only contact with a square in the environment.
Red
Grey
DynaGuide SynaGuide-Smmling
75%
25%
60%
40%
Base Policy
55%
45%
Table 2: CupPreference Additional Result add a baseline to the GuideRed condition of CupPreference (Section 4.5. While DynaGuide- Sampling improves over bascline, DynaGuide achieves the best guidance.
the target. Note that the visuals in B-D show
A.2 Additional Goal-Conditioning Baselines
In Experiments 1-3, we use simple goal conditioning baseline by using the last trajectory state as the goal representation. To maintain fairness with the base policy, we also train the goal conditioned model on only the test environment's 's data in Calvin. However, we note that the dynamics model in
22

---

DynaGuide is trained with all parts of method, we train another con-ccondtio policy on the same multi-environment data. When evaluated on Underspeifie(Oheettivve some steering targets, although DynaGuide still outperformed this augmented goal conditioning baseline (GC w/ All Data, Table 3).
data. To maintain faimess with the total data exposed
this additional data improved performance across
Finally, we try to apply an algorithmic improvement on the goal conditioning bascline by implement- ing Hindsight Experience Replay [2]. Instead of conditioning the policy on only end states during training, we also sample intermediate future states to broaden the goal distribution. We find that this improves steering success across some tasks but not others (GC w/ HER, Table 3). Overall, we find that goal conditioning can be improved for select tasks with additional data or algorithmic changes, but these augmentations do not raise the performance of this baseline to that of DynaGuide.
switch
drawer ogen
drawer
door right
DynaCiuide
0.53-200031
0.362000332
0.24:0017
0.26=00024
0.26::0.115
0.27::00017
0.25
0.19:00.015
Original GC
0.023:10.0000
0.053:10.015
000031100018
0.007=000057
0.00:10.00
0.003:11000033
0.12:10:012
0.080:10.016
GC w/ All Data
0.054:1:00017
0.10:20012
0.0611:00015
0.047::0.015
0.00:10.00
0.00:100000
0.095 0.005:1.0006
0.005:10.021
GC w/ HIB
0.0057:1000042
0,087:00.030
0.027:10.011
0.020:00.007
0.00:10.00
0.00:100000
0.21 1:00023
0.13:10:021
Table 3: Stronger Goal Conditioned Methods. Strengthening the goal conditioning baselines with additional data or hindsight replay [2] shows some improvement in steering ability, but as tested on UnderspeecffedOOheec- tives (Exp 3), they still perform worse than DynaGuide.
A.3 Additional Real Robot Baseline
The real robot experiments (Section 4.5) demonstrated the ability for DynaGuide steer pretrained policies. To further demonstrate the performance of DynaGuide on real robots, we add DynaGuide- Sampling baseline for one real robot task. As seen in Table 2, influencing the action sampling with the diffusion model leads to an improved success rate of the target task over baseline, but the classifier guidance of DynaGuide still yields a higher success rate of the target task.
A.4 Ablations and Hyperparameter Investigation
To understand how various hyperparameters and components of DynaGuide contribute to the final success rate, we set of experiments on the In general, we observe that the two main hyperparameters a (Eq. 2) and s (Eq. 3) are robust to reasonable changes, meaning that DynaGuide is not difficult to tune. At very low values of s, success rates are lower because there is not enough guidance. Success rates increase with higher strength until the strength becomes too high and creates instability. Very low values of decreases guidance success by making the guidance conditions too far away for any meaningful signal. Increasing the number of stochastic sampling steps M generally increases the performance of guidance cost of computational expense. Surprisingly. DynaGuide is very robust to the number of guidance conditions G, achieving comparable guidance even with one guidance condition.
task in the ArticulatedParts experiment.
In practice, we use a guidance strength and a chosen from a gridsearch for each experiment. These parameters are often very similar across tasks the same environment (Table 4), which means that switching target tasks will not require extensive hy- perparameter search. We use stochastic sampling M = for our experiments balance of stability and computation efficiency. We use 20 guidance conditions per task, which is more than suf- ficient for Swit Pwrithh-On but is more important for underspecified goals ($4.2) and harder objectives.
Task
Scale
a
switch_on
1.5
30
switch_off off
1.5
30
drawer_ pewer__peen
1.0
40
drawer close
1.0
40
button_on
1.0
30
outton_off
1.0
30
door left
30
Critically, we discover that pretraining the dynamics model with a noised action is essential for performance (Fig. 9). This makes sense the noise would otherwise be out of distribution.
door door_right
2.0
15
Table 4: Sampling Hyperparame- ters per Task. Per environment, the optimal parameters are very similar across tasks.
In experiments 1-5, we leverage a more diverse dataset collected across four different Calvin environments to train the dynamics model. This data magnitude improves training stability and re- duces overfitting. However, because we test only single Calvin environment, we conduct
23

---

data ablation and train the dynamics model only on data from that environment. As seen in Table 5, reducing the dynamics training data also reduces the steering success, although the performance is still above baseline (Experiment 4. This ablation demonstrates that DynaGuide can transfer information from related environments to help steer a policy in target environment, thereby reducing the data requirement for training a dynamics model in new environment.
button_.on
button_off
switch_on
switch_off
drawer__ppes
drawer_close
door_left
door_right
All Emvironentts
0.705000051
0, 0.702:00038
0.74
0,76
0 00000001
0.66::0000 079
0,74200025
0.68000026
Target havironmen: Only
0.002
0.60+:00008
0.54+-0:43
0.58++00011
0.51 +0.022
0.53+00024
0.65:60.019
0.57:40.056
Table 5: Reducing Training Data in Dynamics Model. Ablation result from ArticulatedParts (Section 4.1) using DynaGuide with a dynamics model trained only on the D-split of Calvin.
A.5 Classifier-Free Guidance Baseline
An additional relevant baseline is the Classifier Free Guidance (CFG) [20] that leverages versions of the diffusion policy itself to guide the diffusion process. We implement and run CFG as a baseline for MultiObjectives. CFG can guide towards multiple objectives, feat that the goal-conditioned baseline was not able to accomplish. Table 6 presents these additional results for a CFG baseline, which should be compared to those of DynaGuide and SenaaGudde-Smplling in Figure 5. As expected, CFG can successfully guide the policy towards multiple objectives, including accomplishing multiple behaviors and avoiding other behaviors. However, the total success rate for positive guidance (first three rows in Table 6) is lower than that of DynaGuide (Figure 5).
The CFG is also able steer the policy away from behaviors, as seen in the last three rows of Table 6. The success rates of the undesired behaviors for CFG and DynaGuide are within one standard deviation. However, while DynaGuide nearly always offers valid alternative behavior (Figure 5), the CFG baseline fails to execute a behavior more than 10% of the time for all three tested conditions (No Behavior, Table 6). This is expected result, as the log-sum-exp latent selection method of DynaGuide allows the guidance to find an appropriate behavior and minimize the influence of other contradictory behaviors. In contrast, the CFG implementation gives equal weight to all guidance signals. The lack of a latent distance makes it difficult to select the most relevant guidance. Contradictory guidance signals can influence the diffusion process negatively, leading to poorly-formed actions and unsuccessful behaviors.
Button
Switch
Drawer
Door
Blocks
No Behavior
Button or Switch
0.18+0001
0.37+0.02
0.05=0.02
0.05:10.00
0.29:+0.02
0.05+10.01
Button or Door
0.37:10.01
0.09:1001
0.01=0.00
0.28
0.23++0.02
0.02++0.01
Switch or Button or Door
0.17-:0.02
0.29:00.44
0.03=0.00
0.23+00.2
0.25++0.11
0.03+00011
NO Drawer
0.16++0.03
0.23+00.2
0.01=0.00
0.09+00.00
0.35+00.2
0.17+0.02
NO Door
0.2311003
0.15
0.12=0.04
0.01110.00
0.3810.04
0.111:0.02
NO Drawer NO Door
0.1910002
0.25:10.3
0.02-0.01
0.00110.00
0.42:10.05
0.1210.00
Table 6: Additional Classifier-Free Guidance (CFG) Baseline. Behavior distribution for CFG baseline on the MultiObjectives Experiment shows that CFG can guide towards multiple objectives but struggles with avoiding specific behaviors without an increase in overall failures (lower three rows).
B Implementation Details
B.1 DynaGuide Implementation Details
Base Diffusion Policy The diffusion policy is trained to take a 2-step history stack of visual observa- tions and robot proprioception and predict a chunk of 16 actions. uses Resnet-18 image encoder that conditions a U-Net with 4 encoding and 4 decoding layers. This U-Net predicts the noise during training using the standard diffusion policy objective [7]. During inference, we use the noise predictor with the DDIM noise scheduler to craft the generated action chunk. Together, the diffusion policy has around 18 million parameters. We use the Adam optimizer with learning rate of le-4. We train the model for 200k gradient steps with a batch size of 16. All parts of the diffusion policy are trained
24

---

together using expert data. During execution, 14 actions are executed in the environment open-loop before the policy is queried again.
Hyperparameter Sweeps
8.00-
0-90-
Success Rate
0.79
0.30
0.00-
0.59
0.53
0.35
0.55
0.28
0.52
350
6.93
0.55
3.45
10
10
15
Guidance Strength
Parameter
Stochastic Sampling Steps per Denoising
Number of Guidance Conditions
Figure 8: Hyperparameter sweeps and ablations. We look at the impact of inference- time hyperparametters and noise pretraining final performance. For guidance strength, we hold o 40, M = 4, where M is the number of stochastic sampling steps. For a parameter, we hold For Stochastic Sampling, we hold 8 3, = 10. We use a higher to demonstrate the impact of stochastic sampling on stability. For Guidance Conditions we use s = 1.5, a 40, M = 4.
1.5, M where is guidance strength.
Dynamics Model. We use the V2 patch embeddings as the latent space, which provides 256 patches of embeddings with size 384. While this is large embedding size (256 384), past work has shown the effectiveness of this exact embedding in robot planning [56]. Empirically, both in past work and in our development, we discovered that the CLS token in D2 (1 x 384) does not carry as much spatial information needed for meaningful latent guidance. For the latent dynamics model, we use a 6-layer Transformer Encoder with heads that takes in the 256 patches, token for robot proprioception, and an action token. This action token is created by embedding each action in a using the same MLP and concatenating the embeddings into the token. We use a learned position embedding. The first 256 outputs of the transformer are taken as the predicted future state. We regress these patches to the D2 embeddings of the last (or later) observation of the trajectory. For the CALVIN experiment,
regressed to the final latent of the trajectory. For the real-world experiment, we regressed to the latent of the observation after executing the proposed chunk of actions in the environment. We use the Adam optimizer frozen, the transformer can be trained end-to-end using a regression objective. This transformer has approximately 16 million parameters. We train the model for 600k gradient steps with batch size 16, a point past model convergence. During development, we showed that validation performance depends strongly on the action's presence, indicating that the dynamics model has learned to listen to actions.
learning rate of le-4. Because the Dino V2 embedding
To account for noisy actions during guidance, add noise to the actions used to train the dynamics model. 50% of the fed actions are noiseless, and 50% are noised based on the same DDIM scheduler used during inference. We select the noising step using geometric random variable with an expected noise step of 20. The scheduler starts at step 100 with pure gaussian noise, so at step 20, it a noisy but still meaningful action that the dynamics model can use.
Inference-Time Steering. Before inference-time we take the guidance conditions and pre-compute their embeddings. Then, during inference-time, we compute the objective (Eq. 2). Empirically, the Euclidean distance yields more stable results in lieu of squared distance in Eq. 2. We backpropagate
this metric through the dynamics model and use the gradient with respect to the input actions as the guidance signal. Like the base policy, we use the DDIM sampler to reduce inference-time computation from 100 steps to 10 steps. To implement stochastic sampling, we repeat the same denoising step multiple times within the denoising loop.
Compute Hardware. All policies and dynamics models were trained on single RTX 3090 GPUs with 24GB VRAM, taking 24-48 hours to convergence. The dynamics model is = 15M trainable paramters, which takes up 44GB of GPU memory during training and inference. All experiments were conducted on single RTX 3090 GPUs taking 10-20 minutes per seed per task. Development of the method and experiments were also conducted on RTX 3090 GPUs, with the total compute-hours for development estimated at 10-20 times the results shown the paper. Most of this compute was spent developing the dynamics model.
25

---

B.2 Implementation of Baselines
Goal Conditioning. We use the provided conditionnn imple- mentation in Robomimic [29] and condition the diffusion policy on an additional input consisting of the final observation in the trajec- tory. We train this policy using the same hyperparameters as the base diffusion policy.
Noise Pretraining Ablation
0.
08
Success Rate
0.5
Position Guidance (ITPS) [50]. The ITPS algorithm excels at steering the policy towards target point in 3D specified by human. To compare ITPS to our approach that does not require human specification, we manually compute the average final position of the robot for set of 20 trajectories showing the desired behavior. We then use this position as the target point ITPS.
Ncbi Pretran
Nose Pretran
Figure 9: Ablating Noise Pre- training. Removing the dynamic model's exposure noised ac- tions greatly decreases its ability to steer the action diffusion pro- cess.
Sampling (GBC) [39]. Previous work that introduced GBC-Rank used objective function to rank action samples from policy select the best sample. We take that idea and apply it to our guidance metric (Eq. 2). We sample the base policy times per inference and pick the best action to execute based on the metric.
Classifier-Free Guidance [20]. We use the same goal-conditioned architecture as the goal- conditioned baseline. However, during training, we randomly zero the goal condition to repre- sent the unconditional input needed in dassifier-free guidance. During inference, compute the classifier- -rreffeer-feeeeeeeee guidance noise estimation using the same conditions used in the goal-conditiondd policy baseline.
B.3 Distance Metrie: Derivation and Design Choices
In this section, we will conduct a more detailed derivation of the metric shown Eq. 2. While other works [56] simply assume the metric to be Euclidian, we use probabilistic motivation to understand log-sum-exp, which we discovered empirically to very important for the performance of DynaGuide. For the first part of this derivation, we will only talk about the desired outcomes 2 and the undesired outcomes follow by negation.
We start from the very rough approximation that the latent space is Gaussian with diagonal variance Σ. It is an approximation that methods using squared latent distances already implicitly make. Here, the probability p(2 (2++)) = [x = bitH. K = =o). The log probability can be computed as follows:
1 [et 24411 -2 --- 2x+H1)
logp(2; 12+ )= log
exp
m/2 E
= - -2 - 2(+ - (21t - &+HH) + log (2m) n/2
Constant
(2 - 22+ = ((-H) +C
20
|| + 113
There are different ways of combining the influence of multiple 2,t in zt. One way that empirically does not work adding the log- proaabbbaillities together:
-| - 2a
d=
This is not a surprise, as adding rig-prbbabilities is equivalent to multiplying the probabilities together. As previously discussed, not all the 2 are achievable and some may be mutually exclusive. For example, in the multi-objective setup ($4.3), some = might correspond to pressing the button and
26

---

others to opening the drawer. The influence of any single 2 is too powerful in this setup because of the product.
In contrast, it is more sensible to add the probabilities. This way, changes that generally increase the likelihood of the 2t's will increase the metrie, even if it decreases the likelihood of some other competing a;" s. To add the probabilities,
do the following:
logp(z (2((((11+))) log Σ (22+(()))))))
(-2 || - 24tH111 + +C)
= log Σ exp
|| (-= - 2c- 1112
= log Σ exp(CC) exp
20
= CC + log
Σ exp
|| (- 20
Which creates an intuitive result, as the log- log-sum-cen is soft maximum. The metric focuses on the distances that are closer to the desired outcome. The value of a modulates the sharpness of the soft maximum.
We compute the same metric for z Because we want results in z to be mutually exclusive to z+, want to divide the probabilities:
d p(2*) p(z
Intuitively, it means that there is a strong gradient incentive to push p(=-) as small as possible, which use p(2+) p(2-). We observe the benefits of the division empirically. not strong if we subtracting the undesirable outcome metric from the desirable outcome This setup corresponds to metric. To create the final d, we absorb the constant 2 into and discard the constant C. This gives us the expression in Eq. 2:
-Ilo(0t) ho( a)]l
-1b((g7) - ho(o(o)),
d log Σ
log Σ exp
exp
σ
o
Empirically, we discover that Eq. 2 works best with the squared 1.2 distance substituted with Fuclidean distance. This deviation from the theoretical result is best attributed to the Gaussian latent assumption of D2noV2 not being fully true to the real embedding space.
In practice, g* can be either future state (t + H! or final state (T). Using a future state H/ is easier to train for the dynamics model but it is harder to find correct g+ during inference-time guidance, as must be state achievable steps from the current timestep. Alternatively, predicting final state T means that final states from demonstration trajectories can be used as g' making the choice of inference-ting time guidance examples straightforward. However, training dynamics model to predict final state can be difficult for long-horizon tasks. We find that the Calvin environment enables end state prediction (T), but the real world robot environment requires setting H steps into the future. We feed entire trajectories as g* and rely on the log-sum-exp latent selection to dynamically pick the most relevant guidance points. The choice of g is an important design choice and it depends the environment and the data.
B.4 Experimental Setup: Simulation
CALVIN Environment: Data. Although CALVIN is used for benchmarking, we just use the CALVIN tasks for the data and our own sets of experiments. The CALVIN data is provided
27

---

continuous set of transitions. We used privileged state information to segment these transitions into trajectories showing one behavior per trajectory: switch on, switch off, drawer open, drawer close, door left, door right, button on, button off, red touch, red displace, blue touch, blue displace, pink touch, pink displace. We did our own segmentations to be consistent with our evaluation criteria (See later paragraphs). The segmentations were also important to extract end observations for the coal-conditioning baseline and training the dynamics model.
CALVIN Environment: Base Policy. We use the CALVIN-D dataset to train the base policy. We third-person
use third- person observation, wrist camera observation, and full robot proprioception for the base diffusion policy. Because the demonstrations show a large variety of behavior, the trained base policy also offers wide variety of behaviors.
Dynamics Model. Because the dynamics model can take a wider range of data, we train the dynamics model the CALVIN-ABCD dataset, which is the full data split provided by the benchmark. We discover that adding the non-relevant environments improved convergence and reduced overfitting. We use the third herr-- observation and full robot proprioception as inputs to the dynamics model. The short tasks horizon of the CALVIN tasks meant that it was advantageous train the dynamics model to predict the final state observation of the single task trajectory. We use the trajectory segmentations to obtain the final observation target during training.
CALVIN Environment:
Evaluations. We segment the provided validation CALVIN dataset and randomly select 20 trajectories per desired behavior to extract g g using the last state observation experiment). For each evaluation, we perform 50 trials (except for the UnderspecifedOObeetiivy with a horizon of 400. We use privileged state information to monitor the object interactions and stop the rollout when object is sufficiently articulated moved. If no object is sufficiently moved or articulated after 400 steps, we count the trial as a failure (no behavior). For each target behavior (horizontal axis in Fig. 4), we compute the success rate by finding the frequency of trials that show the target behavior. We reset the robot randomly by sampling a starting pose in a validation set of trajectories. All error bars in simulation results (Figures 4, 5) are I-sigma error bars. The standard error is calculated by evaluating DynaGuide on six policies trained on individually separate validaaiiitiion splits of the Calvin dataset. Because each of the six success rates is already an average across 100 trials, we use the standard error of the mean as the sigma. We assumee germall--disribuue errors, so the standard error computed with the standard a/ //vn formula where a is calculated through Numpy function. For the line plot (Fig. 5 Right), the shaded regions were computed as standard error of the mean of six success rates the same as the bar graphs.
CALVIN Environment:
ArticulatedParts Experiment. To count as being sufficiently articulated, we require buttons and switches to be fully pressed/flipped such that the light changes state. For drawers and doors, they must be articulated past halfway from the starting location to their end location.
MovableObjects Experiment. In this task, we focused on the ability to steer the policy towards the colored cubes. To be successful, we required the robot to be touching the cube and displace a slight distance, which can be possible by lifting the cube nudging it.
UnderspecifeedObjeetiiss Experiment. In this task, we used the ArticulatedParts experiments but fed all the guidance approaches with a lower quality guidance condition. Instead of taking the observations from the final states of validation trajectories, we manually set the target object to the desired pose (e.g. drawer open, or door to the left). We sample states for all the other objects, and we randomize the robot by setting it to a start pose sampled from the validation trajectories. The critical difference between these g" and the g+ used in the other experiments is that the robot is no longer shown directly interacting with the object of interest. Even though the robot position is randomized, it is still possible to figure out the target behavior because it is the one object state that stays constant between conditions.
MultipleObjectives Experiment. This task, we extracted guidance conditions from validation trajectories that represented multiple types of behaviors. We are not interested in chaining the behaviors, but rather a steered policy that offer multiple desired behaviors with comparable frequency while avoiding the undesired behaviors. During evaluation, we still terminated the rollout after single behavior and we compute the success rates in the same way all previous experiments. However, instead of reporting a single success rate per task, we report the whole task distribution (Fig. 5, Left).
28

---

Unnderrppesentellellviinss Experiment. This task required retraining the base policies with modified training data. Using privileged state information to get trajectory behavior labels, we intentionally removed Switch-On behavior from the training set of the base diffusion policy to create sets where 1%, 2%, 5%, 10%, 20%, 40%, 60%, and 80% of the original Switch- On data was kept.
BlockTouch Experiments. We collect synthetic data of the agent navigating to randomly selected square by creating bezier curve with 0-2 intermediate points. This bezier curve means that the dynamics model can't 't infer the final destination with perfect accuracy based on the current direction of travel. Instead of using the metric d, we directly train the dynamics model classify the output square color as a 4-dimensional vector categorical distribution. During inference-ting we use cross entropy metrie between this distribution and one-hot vector representing the desired square color. During training and all data collection, we randomize the location of the squares. During test-time, we introduce specific square arrangements to test the properties of the guidance In the EarlyDecision test, we still randomize the squares but ensure that the cubes stay in their own regions, forcing the agent to make an early directional decision. In LateDecision, we keep the squares close together, and in Go to Furthest Cube, we always have the blue square on the opposite side of the environment as the starting agent location. This agent must move past the three other squares to find the blue square. We terminate the environment once any square has been touched.
B.5 Experimental Setup: Real Robot
Base Policy. We use a publicly available trained diffusion policy provided by the Universal Ma- nipulation Interface repository (github link). It was trained on =2k trajectories of cup grasping, reorientation (to move the handle to the left), and placement on a saucer. The policy takes images from gopro with a Max Lens mod and outputs 16 actions per generation. These relative actions represent the change in the robot from its current position [6]. This policy has mostly seen single cups and saucers in the environment, but when provided with multiple cups, it will choose a cup at random. To demonstrate DynaGuide on existing policies, we do not modify this policy. Although the policy offers random choice, also has a bias towards the left side of the environment. To account for this bias, we randomize the location of the desired cup in CupPreference. We also place the computer mouse in the area opposite to this bias in NovelBehavior to ensure that all improvements are due to steering and not base policy bias.
Data. We train the dynamics model on the open-source cup rearrangement data provided by the Universal Manipulation Interface repository (data link). Because we want to steer the policy towards one of two cups, we need train the dynamics model on these decisions. Using the Universal Manipulation Interface hardware, we collect 500 demonstrations that shows two cups and one saucer in the experiment scene. We pick a cup and place it on a saucer. We use different cups and saucers, with no correlation between cup pairs and saucers. We combine this data with the existing cup arrangement data for the dynamics model. For the NovelBehavior experiment, we additionally train the dynamics model on 3648 publicly available trajectories of computer mouse rearrangement [26), which gives it the experience needed to steer the base policy towards the computer mouse.
Setup. We use an ARX5 robot arm equipped with soft Fin Ray fingers and a gopro with Max Lens mod that mimics the setups used to collect base policy's 's training data. We use the hardware controller stack provided by the Universal Manipulation Intervace adaptation for ARX5 arms (Github Link) and a power supply with overcurrent protection for safety during deployment.
We conduct 20 trials for each base/guided policy. Real-world evaluations are very resource intensive, we did not create confidence intervals. This is very common for real robot results, even for large projects [5, 57].
Evaluations.
CupPreference Experiment. We arrange the red and grey cups such that the starting distance to either cup is roughly equidistant from the robot. We randomize this distance by sometimes placing the two cups close, and other times placing the two cups far away. We place the saucer randomly in the environment. For 50% of the trials, the red cup is on the left, and the other 50% the red cup is on right. We measure success picking up a cup and placing it on the saucer.
HiddenCup Experiment. We arrange the red cup in front of the grey cup, such that the grey cup is always closer to the robot than the red cup. We randomize the distance between the red and grey cup, as well as the overall location of the two cups from the robot. We randomly place a saucer in the environment, although count cup grasp as success without needing this saucer. We
29

---

discovered that the two cup setup in this configuration is out of distribution enough for the base policy increase the mistake cup placement. Because we are trying steer for cup color preference, the act of grabbing the cup is sufficient for this experiment. The previous experiment tested full behavior success.
NovelBehavior Experiment. Like CupPreference. we arrange the red cup and black computer mouse equidistant from the robot's starting location. As previously mentioned, place the mouse in the area opposite of its side bias to ensure that the observed effects are attributed to steering and not base policy bias. Like in HuddenCup, we count grasp attempt of the computer mouse as a success. This is because the computer mouse is very out of distribution for this base policy and it will not successfully grab the mouse. What matters this experiment is being able reach for novel object through dynamics guidance.
C Broader Impact
DynaGuide contributes to the field of robotics by improving the ability for pretrained policies to be steered without retraining, potentially reducing energy consumption otherwise needed during retraining. Adding steering onto pretrained policies also potentially increases accessibility for labs that would otherwise be unable to retrain large policies. This steering approach also be effective way of removing unwanted or problematic biases in trained robot policies after the training process. The dynamics model is trained data that is either publically available easy collect, and the data contains no private or sensitive information.
Because DynaGuide allows steering of existing policies, there is risk of bad actors steering off- the-shelf policies to do dangerous or malicious behaviors, risk also present for other generative models. To mitigate these risks, trainers of the base policy can ensure that malicious behaviors are not represented at all. Knowing the mechanism of DynaGuide, it may also be possible to train base policies to act adversarially to the guidance signal if asked to do malicious behaviors, making the guidance as difficult as retraining the policy from scratch.
D Code, Assets, and Licenses
All models trained publicly available data and leverages codebases and assets under these licenses:
CALVIN Environment and Data (MIT License) [30]
Robomimic Codebase (MIT License) [29]
Di Model (Apache license) [37]
Universal Manipulation Interface Codebase, pretrained model, and data (MIT License) [6, 26]
Our code will be publicly released under the MIT license in the near future. Provided in the supplemental material is barebones version the code.
30


Reviews:
[{"review_id": "0tjyO9hFEb", "reviewer": "Reviewer_RYQt", "summary": "The authors present DynaGuide: a method for steering a base diffusion policy to desired task outcomes using the gradient feedback from an additional predictive dynamics model which matches positive and negative goal visual references.\n\nThe overall approach is illustrated in Figure 2. An additional dynamics guidance model is included to predict future latent state outcomes (trained by equation 1, page 4). Using these predicted outcomes, a denoising guidance signal is derived (equation 3, page 5) based on the action gradient with respect to a guidance metric $\\mathbf{d}$ (equation 2, page 5). The guidance metric encodes positive $\\mathbf{g^+}$ and negative $\\mathbf{g^-}$ image references for visual scenarios to attain and avoid respectively. The authors hypothesize that their novel steering approach is well suited to finding under-represented samples and meeting multi-objective scenarios.\n\n__Novelty and Contributions:__\n- Novel guidance signal approach for diffusion models using gradient from a dynamics prediction model to steer the denoising process\n- Versus existing baselines, their method boasts improved performance in achieving multi-goal objectives and under-represented behaviours.\n\n__Experiments:__\n\nA number of experiments explore the performance and properties of DynaGuide using a robot arm in the CALVIN environment [30].\n\nIn experiment 1 (section 4.1, figure 4 top) their approach fails to beat a more conventional goal-conditioned policy when interacted with standard articulated parts, however in experiments 2 (section 4.1, figure 4 bottom left) and 3 (section 4.2, figure 4 bottom right), their method significantly outperforms other baselines for movable and under-specified objects. The authors assert that this is because their method is more robust to situations that would normally be out-of-distribution for a normal goal conditioned policy given that goal, observation, and action inputs are all individually in-distribution to their modular setup while also using additional guidance signals in the latent space (described on lines 278-285).\n\nIn experiments 4 (section 4.3) and 5 (section 4.4) they also illustrate how their method is suitable for multi-objectives and its ability to find under-represented scenarios versus a sample based DynaGuide variant (GPC) as the sample-based method is restricted by the frequency of the desired behavior in the dataset being rare in this case.\n\nFinally, in experiment 6, they test their method with a real robot versus the base policy. While the base policy is limited to the representative behaviors in the dataset, their policy effectively steers towards the conditioned goal.", "questions": "- Can the authors clarify why additional baselines were not included in the real-world experiment section (experiment 6, section 4.5)?", "limitations": "Yes. Limitations are listed in section 5.1 (lines 353-360).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "__Strengths:__\n\n- __Novelty and performance (originality and significance):__ Relatively novel and modern approach adapting recent diffusion models to a planning context using steering with primary advantages in executing multi-goal objectives and under-represented behaviours. Method outperforms state of the art baselines. See __Novelty and Contributions__ section above for more details.\n- __Pertinent use cases (significance):__ As discussed in the experimental section, the proposed method excels in multi-goal and under-represented behaviours. The usage for under-represented behaviour may be especially important for real world applications such as robotics where training data may be more difficult to gather and thus some behaviours may have limited representation in the dataset.\n- __Experimental rigour (quality):__ Experiments are done well for the most part (quality): Various aspects of the model were tested and modern relevant baselines compared against. Error bars are included. Through various experiments, they well illustrate the advantages of the proposed method compared to existing approaches for multi-goal objectives and under-represented behaviours (see __Experiments__ section above for more details). Ablation and parameter sweep results can be found in appendix A.2 with intuition concerning the parameter selection. However, the real-world experiments were somewhat lacking (see next section). \n- __Manuscript clarity:__ The manuscript is well written and contained sufficient detail.\n\n__Weaknesses:__\n\n- __Real-world experiments baselines (quality):__ The real-world experiments (experiment 6, section 4.5) only compared a base diffusion policy versus the full DynaGuide model. No other baselines were included (quality). This is a relatively weak baseline which does not seem to have any goal conditioning? I have asked for clarification regarding this issue in the questions section.\n- __Additional parameter tuning (significance):__ The method modulates between a base diffusion policy and task-based goal arrival (equation 3) weighted by a scaling guidance parameter. Selection of this scaling parameter might require additional tuning to achieve a reasonable policy versus other more direct optimization methods.\n\n__Current Assessment and Suggestions for Improvement:__\n\nI have currently marked this work as borderline accept. I think the submission is of good quality (based on the discussion related to strengths and weaknesses above) however I would of appreciated more baselines comparisons in the real-world experimentation section.", "was_revised": true, "final_justification": "After reading reviewing the authors' rebuttal - and absent any further modifications - I have decided to keep my overall score the same (`Borderline Accept`). Please see my response to the authors' review for more details. \n\nIn short, my primary criticism is that I would of liked to see more experimental baselines applied to the real-world experimentation section. More experimental rigor seems to have been applied to the simulation section which ideally could be carried over to real-world experiments. Nonetheless, this is a solid paper but I feel requires some additional experimental testing to push it to the `Accept` level. \n\nThe authors have also included more analysis into hyperparameter selection which eases some of my concerns for this previous weakness item.\n\nPlease see my original review for a summary of the strengths and weaknesses of the work.", "rebuttal": "Thank you for your detailed and thoughtful review of our paper. We appreciate your recognition that DynaGuide presents a novel approach for steering diffusion policies, and we appreciate your highlighting the performance boosts of DynaGuide in situations with under-represented behaviors and multi-guidance conditions. \n\nTwo main points are raised in the weaknesses / questions section, which we will address here, along with other minor points.\n\n# Real-World Experiments: Lack of other Baselines\n\n## Lack of goal-conditioned baseline \nThe primary concern mentioned in the weaknesses section is the **lack of a goal-conditioned baseline in the real robot experiments**. As mentioned in lines 325-326 and lines 71-74, one of the primary purposes of the real robot experiments is to test the hypothesis that DynaGuide can work on *any pretrained policy*. To do this, we use a publicly-accessible off-the-shelf policy [1], which unfortunately does not have a goal-conditioned version. Therefore, in that set of experiments, it was not possible to conduct a fair comparison against a goal-conditioned model, which we would have needed to train ourselves. \n\nOn the longer time scale, we can conduct an additional robot experiment set where we train our own policies and allow for goal conditioned comparisons. If we train our own policies with our environment-specific data, we anticipate the results to be stronger across the board. \n\n## Lack of other baselines (additional experiment)\nThe simulation environments provided a stabler and repeatable environment for head-to-head comparisons, which is why we chose the simulation setup to conduct extensive baseline comparisons (exp 1-5). DynaGuide does not make any assumptions that would cause a significant difference between real and simulated experiments. However, we acknowledge the reviewer’s concerns about the need for more baselines in a real-world setup. \n\nWe added the DynaGuide-Sampling (GPC) baseline on the real robot. Due to the short rebuttal turnaround time, we were only able to evaluate it on the first real robot experiment (pick red cup and place on black saucer). We got the following results (20 trials per experiment)\n\n| Method              | Red  (steering target) | Grey            |\n|---------------------|------|-----------------|\n| DynaGuide    [from paper]       | 75%  | 25%      |\n| DynaGuide-Sampling [NEW]  | 60%  | 40%             |\n| Base Policy   [from paper]      | 55%  | 45%             |\n\nAs noted in the paper, the pretrained base policy is biased towards one side of the environment (i.e., it will typically grab cups on the left side). We shuffle the cups equally for fairness, but to get above ~50%, the method must overcome this bias. Sampling was not effective in removing the bias, while DynaGuide was able to accomplish this. \n\n# Hyperparameter Tuning \n> Selection of this scaling parameter might require additional tuning to achieve a reasonable policy versus other more direct optimization methods.\n\nThis is a valid concern, as additional parameters mean additional hyperparameters to tune. However, as seen in Figure 8, the guidance strength parameter (the scaling parameter) is robust to large changes in scale (leftmost plot). DynaGuide is also relatively robust to the sigma parameter (second plot from right) and even the number of guidance conditions (rightmost plot). The stochastic sampling steps (second plot from left) generally increase performance with more steps, so they can be chosen based on resource constraints. \n\nTo get the results of DynaGuide shown in the paper, we only needed minimal tuning. Notably, these hyperparameters remain relatively stable across individual tasks within an environment. As an example, here are our optimal hyperparameters for the individual tasks:\n\n| Task         | Scale | Stochastic Sampling | Sigma |\n|--------------|-------|----|-------|\n| switch_on    | 1.5   | 4  | 30    |\n| switch_off   | 1.5   | 4  | 30    |\n| drawer_open  | 1     | 4  | 40    |\n| drawer_close | 1     | 4  | 40    |\n| button_on    | 1     | 4  | 30    |\n| button_off   | 1     | 4  | 30    |\n| door_left    | 1.5   | 4  | 30    |\n| door_right   | 2     | 4  | 15    |\n\nThe two most commonly changed parameters are Scale and Sigma. Generally, to deploy DynaGuide, we advise starting with guidance scale 1-2, and increasing if the policy does unwanted behavior. Choose a smaller Sigma if the guidance conditions have more variance and you suspect that only a few are relevant per environment, and vice versa. \n\n# Summary\nThank you for the comments and feedback. In our response, we detail why our current real-world experiment setup was unable to utilize a goal-conditioned baseline fairly. We conduct an experiment with a GPC baseline on the real robot setup. Finally, we discuss the sensitivity of DynaGuide to its hyperparameters and provide simple rules for tuning them. If any of your concerns were not addressed by this response, we welcome further discussion. \n\n[1] Chi et al. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. RSS."}, {"review_id": "7RVo6siNVU", "reviewer": "Reviewer_yJvQ", "summary": "DynaGuide proposes to guide diffusion based robot policies by learning an independent dynamics model which influences the gradient field of the diffusion model.\n\nThis approach is evaluated on three simulated tasks using CALVIN and a real-robot task and compared to goal-conditioning and position guidance approaches.\n\nIn addition to help guide achieving desired behavior, this approach works with pretrained, off-the-shelf policies, with many potential applications for diffusion based robot learning.", "questions": "## Main Question:\nRegarding CALVIN Datasplit (see Weaknesses). I do not believe running the full ABC-D ABCD-D CALVIN benchmark is reasonable to demand as a reviewer. I would welcome any results with the DynaGuide model trained only on D, as the other policies, if results are available. \n\nOverall I am already very impressed with the method and hope that answering these questions will lead to even higher impact in the research community.\n\n### Minor Questions to help improve my understanding:\n- I am confused by the CALVIN underspecified objective. I assume any highlighting in Figure 3 is just for illustrative purposes? So what does it mean to \"randomize robot states (and other states)\"? Because I am quite surprised by goal-conditioning's performance drop.\n    - Could DynaGuide guidance counteract BC's performance drop? Such an experiment would also more nicely highlight DynaGuide capabilities to work with different, \"off-the-shelf\" policies.\n- On multi-objective guidance: Am I understanding correctly, that each objective is used to produce a guidance signal independently, which is then combined across all objectives, for example by addition? Or does the Dynaguide model actually receive a set of objectives at once?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The paper is very well written and easy to understand.\n- Solid motivation and theoretical foundation.\n- Multiple experiments in simulation and on real robot.\n\n### Weaknesses\n**I am worried about the data split for the experiments**. The details are a bit hidden in Appendix B4:\n- The base policy is trained only on the CALVIN-D dataset, but the dynamics model is trained on ABCD. As the other baselines are not detailed, I have to assume they were also only trained on CALVIN-D.\n\n    I am not entirely convinced that this is a good basis for a fair comparison regarding the generalization capabilities of the different baselines, as the guiding model has seen more data. I would welcome some discussion on this issue.\n\n- Additionally, I would welcome an evaluation in the more common settings of ABC-D and ABCD-D (or D-D) on CALVIN. \nThis could help to estimate if the guiding model benefits from being trained once on a large dataset and guiding specialized policies, or alternatively, if both models basically require access to the same dataset anyway, to also be trained on the same data.", "was_revised": true, "final_justification": "My questions have been addressed and additional CALVIN D-D provided.\nI recommend to accept this paper.\n\nI believe investigating the transferability of the dynamics model by performing ABC-D experiments could further strengthen the potential impact.", "rebuttal": "Thank you for your positive response, detailed feedback about the CALVIN environment, and clarification questions about our experiments. We appreciate your recognition of DynaGuide’s versatility and our results in both simulation and real setups. Below we address your questions and add additional experiments inspired by your comments. \n\n# Dataset Splits: ABCD vs. D on Calvin (additional experiment)\nYes, you are correct that the base model is trained on `D` and the dynamics model is trained on `ABCD`, which contains more data. We used the ABCD split to showcase the performance of DynaGuide with a dynamics model trained as well as possible. This is how we ensured fair comparisons in our paper results: \n+ Comparing DynaGuide to DynaGuide-Sampling, we used the exact same base policy and dynamics model, varying only the steering method (sampling vs. guidance)\n+ Comparing DynaGuide to Position Guidance, we used the exact same base policy and steering method, varying only the steering signal (dynamics model vs position euclidian distance). \n+ Comparing DynaGuide to Goal Conditioning, we trained the base policy and goal-conditioned model on the same dataset and used the same guidance conditions. \n+ Comparing DynaGuide to base policy, we used the same base policy *inside* of DynaGuide, meaning that performance improvements are explained solely by the guidance. \n\nWe agree that training the dynamics model on the `D` split is a meaningful additional experiment because it completely isolates performance improvements of DynaGuide from data quality.  Below are the results on Exp. 1 with ABCD (paper results) and D splits for the dynamics model (requested) \n| Label         | button_on     | button_off    | switch_on     | switch_off    | drawer_open   | drawer_close  | door_left     | door_right    |\n|---------------|---------------|---------------|---------------|---------------|---------------|---------------|---------------|---------------|\n| ABCD  [in paper]        | 0.70±0.031    | 0.70±0.038    | 0.74±0.027    | 0.76±0.037    | 0.60±0.015    | 0.66±0.029    | 0.74±0.025    | 0.68±0.026    |\n| D [new] | 0.66±0.023    | 0.69±0.038    | 0.54±0.043    | 0.58±0.031    | 0.51±0.022    | 0.53±0.024    | 0.65±0.019    | 0.57±0.036    |\n\nAs expected, the dynamics model performs worse when trained on a smaller dataset. However, in practical implementation, the dynamics model may be trained on a more diverse dataset than the base policy because it requires fewer quality assumptions about data than the base policy. Therefore, we can ask another question: does DynaGuide make better use of a larger data split than a goal-conditioned baseline? To answer this question, we train the goal-conditioned baseline on the `ABCD` instead of just the `D` split shown in the paper. Below are the results on Exp. 3 (which originally shows a large impact in goal-conditioned performance.)\n\n| Label         | button_on     | button_off    | switch_on     | switch_off    | drawer_open   | drawer_close  | door_left     | door_right    |\n|---------------|---------------|---------------|----------------|----------------|----------------|----------------|----------------|----------------|\n| DynaGuide [in paper]     | 0.33±0.031    | 0.36±0.032    | 0.24±0.017    | 0.26±0.024    | 0.26±0.015    | 0.27±0.017    | 0.25±0.018    | 0.19±0.015    |\n| GC with D [in paper]   | 0.023±0.0080  | 0.053±0.015   | 0.073±0.018   | 0.047±0.0067  | 0.00±0.00     | 0.0033±0.0033 | 0.12±0.012    | 0.080±0.016   |\n| GC with ABCD [new]  | 0.054±0.017   | 0.10±0.012    | 0.064±0.015   | 0.047±0.015   | 0.00±0.00     | 0.00±0.00     | 0.095±0.026   | 0.075±0.021   |\n\nNotice how the goal-conditioned model benefits slightly from additional data but still is far away from DynaGuide’s performance on Exp. 3. This showcases that DynaGuide can make better use of this additional data split compared to the goal-conditioned baseline. \n\nConclusion from these two additional experiments: although the dynamics model degrades performance when trained only on D (DynaGuide w/ D exp), it leverages this additional data better than the goal-conditioned baseline (GC w/ ABCD exp). \n\n\n# Calvin Underspecified Objective Question\n> I am confused by the CALVIN underspecified objective. I assume any highlighting in Figure 3 is just for illustrative purposes? So what does it mean to \"randomize robot states (and other states)\"? Because I am quite surprised by goal-conditioning's performance drop.\n\nYes you are correct that the highlighting / greyscaling in Fig. 3 is purely for illustration. In reality, in Exp. 3 we supply the methods with guidance conditions (goal images) that are out of distribution by showing the desired *environment state* but not the correct *robot state*. \n\nConcretely, for all other experiments, the guidance condition shows the robot interacting with the target object because the guidance condition image is taken after the robot accomplishes the task. For exp. 3, we still show the target object in the desired configuration but we show the robot in a random position (randomly selected state in a validation dataset). As an example: normally for the `drawer-open` task, the guidance condition shows the robot holding onto the open drawer by the handle. In Exp. 3, the guidance condition shows the same open drawer, but the robot is not grabbing the handle. \n\nBecause all the models (dynamics, goal-conditioned) were trained with the robot on the target object, these guidance conditions in exp. 3 are generally out of distribution. This is why the goal-conditioned model's performance drops significantly. DynaGuide does not drop as much because these guidance conditions are still in-distribution to all individual components of the method; the guidance signal just becomes noisier. The guidance signals can still pull the policy in the direction of future states with open drawers, for example. \n\nWe conducted Exp. 3 to showcase how the DynaGuide process can handle these out-of-distribution guidance conditions better than a model trained end-to-end on the goal distribution. Especially when we are trying to steer an off-the-shelf policy in a new environment with our own guidance conditions, this robustness is especially important. We are happy to clarify further if this response does not fully answer your confusion. \n\n# Counteracting BC Performance drop \n\n> Could DynaGuide guidance counteract BC's performance drop? Such an experiment would also more nicely highlight DynaGuide capabilities to work with different, \"off-the-shelf\" policies.\n\nAssuming that the dynamics model is still in distribution, it would be reasonable to hypothesize that DynaGuide can improve BC performance when the base policy is out of distribution. We base this hypothesis on the results of Exp. 5, where DynaGuide can pull a very underrepresented behavior from a base policy. Executing this behavior is out of distribution for the base policy due to intentionally removing that part of its training distribution. DynaGuide is still able to coax the base policy to execute that behavior. \n\nWe also see an example of this on the real robot setup, where DynaGuide is able to get the robot to deliberately search for a red-colored cup when the base policy defaults to going towards the closest cup (Exp 6, Fig. 6 middle plot).\n\nIn general, if the dynamics model is robust, then DynaGuide could be a tool for counteracting BC performance drop. To say this with more certainty, however, more experiments are needed. \n\n# Multi-objective guidance \n\n> On multi-objective guidance: Am I understanding correctly, that each objective is used to produce a guidance signal independently, which is then combined across all objectives, for example by addition? Or does the Dynaguide model actually receive a set of objectives at once?\n\nSo DynaGuide actually takes a collection of guidance conditions and aggregates the distances into a single metric (Eq. 2). This seems like the latter approach in your question. This means that we only need to backpropagate through the dynamics model once per diffusion step. It technically is possible to compute guidance signals (action gradients) independently and aggregate them after backpropagation, although it is less computationally efficient. \n\n# Summary\nThanks for your comments and feedback. We add additional results using a dynamics model trained on the `D` CALVIN split as well as a goal-conditioned model trained on the `ABCD` CALVIN split to further isolate the impacts of individual DynaGuide components. We also address various questions about the method. If any of your concerns were not addressed by this response, we welcome further discussion."}, {"review_id": "snZHOC0FeO", "reviewer": "Reviewer_n1oB", "summary": "This work introduces DynaGuide, a method for guiding diffusion policies towards desirable goals by using classifier guidance. The guidance mechanism includes a learned latent dynamics model operating on DinoV2 embeddings. The dynamics model predicts a future state from the current state and action chunk, and the gradient of the distance from the desired goal embedding is used for guidance. This method allows decoupling guidance from policy, allowing dynamics models to be swapped out as needed, and can be used with an off-the-shelf policy. Experiments on CALVIN tasks show that DynaGuide can help in cases where the goal is misspecified or corresponds to an underrepresented behavior of the base policy.", "questions": "- The guidance metric in Eq. 2 compares the latent future state $z_{t+H}$ with desired goal state $g$. This would only make sense if the goal $g$ is achievable within $H$ steps from the current state. Since this is generally not true, I am not sure why this provides a meaningful guidance signal. In fact, it may actually be detrimental since the agent can try to reach some unrealizable state to minimize distance to goal within $H$ steps.\n- The value of $H$ is not explicitly mentioned, but appendix line 951 specifies that the policy produces action sequences of length 16 - does that mean that $H=16$ for all experiments? Why was this specific value chosen?\n- Why can’t position guidance be used for the moving objects experiments? In [3], the authors use block stacking and kitchen rearrangement tasks - what makes the cube moving experiment fundamentally different from these tasks?\n- The goal-conditioned baseline is trained by using the last observation of each trajectory as the goal. However, in lines 218-220, the authors mistakenly cite some papers which use goal relabeling techniques (such as hindsight experience replay [4]) to improve training of goal-conditioned policies. This has shown to work in visual domains as well [5,6]. Then, despite referencing such works, why were these techniques not used when training the goal-conditioned baseline?\n- The dynamics model in Eq. 1 predicts a future state $H$ steps away from the current state and action chunk, which is tractable only for deterministic environments. I wonder if the authors have any thoughts on extending this approach to the stochastic setting?\n- Typing error in Algorithm 1: the stochastic sampling loop does not correctly apply $M$ denoising steps at the same noise level $k$. See lines 6-9 in Algorithm 1 of [3].\n- Typing error in Algorithm 1: line 1 should be “positive guidance conditions $g^+$, negative guidance conditions $g^-$”\n\n*[3] Wang, Yanwei, Lirui Wang, Yilun Du, Balakumar Sundaralingam, Xuning Yang, Yu-Wei Chao, Claudia Perez-D'Arpino, Dieter Fox, and Julie Shah. \"Inference-Time Policy Steering through Human Interactions.\" arXiv preprint arXiv:2411.16627 (2024).*\n\n*[4] Andrychowicz, Marcin, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. \"Hindsight experience replay.\" Advances in neural information processing systems 30 (2017).*\n\n*[5] Nair, Ashvin V., Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. \"Visual reinforcement learning with imagined goals.\" Advances in neural information processing systems 31 (2018).*\n\n*[6] Florensa, Carlos, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Riedmiller. \"Self-supervised learning of image embedding for continuous control.\" arXiv preprint arXiv:1901.00943 (2019).*", "limitations": "The authors have briefly discussed the limitations of their work with respect to the type of guidance provided and lack of any mechanism to guide how to achieve the desired outcome. There are several important limitations that have been left out, including the computational cost of DynaGuide compared to baselines, the fact that it seems limited to deterministic environments, and that the base policy must already be able to solve the task in order for guidance to elicit the desired behavior.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The experiment design is done quite well. The authors study different settings designed to test different capabilities, such as working with low-quality goal conditions, multiple objectives, and testing underrepresented behaviors of the base policy.\n- It is not common to see experiments on real robots, and the fact that DynaGuide works in this setting is a strong point in its favour.\n- The authors have provided details of their implementation and baselines in the appendix, as well as provided their code in the supplementary material. While I did not study the code, the details in the appendix are quite thorough, and the authors’ efforts in promoting reproducibility are appreciated.\n- The paper is written quite clearly and is easy to follow.\n\n**Weaknesses**\n\n- DynaGuide unifies classifier guidance with learned dynamics, which is new for robotics, but inherits much from prior dynamics-distillation work (DINO-WM) and inference-time steering (ITPS). The advance is incremental yet practical; clearer articulation of conceptual novelty over these works would strengthen the contribution.\n- One primary motivation of DynaGuide is that typical goal-conditioned policies require anticipating the set of goals during training such that they cover the possible goals encountered during inference. DynaGuide does not seem to solve this problem, since the steering mechanism relies on the base policy already having the ability to solve the required task. Therefore, it seems that the claim in lines 34-42 that sets DynaGuide apart from existing methods should be reconsidered.\n- In practical settings, it is rarely feasible to specify the exact goal observation that the agent needs to accomplish. In line 103, the authors claim that “Other methods with fewer assumptions…using future observations directly as the goal”. Assuming access to goal observations for guidance is a much stronger assumption than relying on goal specification via natural language, for example.\n- The dynamics model uses much broader data than the policy (CALVIN-ABCD vs D). For truly off-the-shelf steering, users must still collect a large amount of unstructured interaction frames to fit a task-specific dynamics model.\n- One essential baseline not included in this work is classifier-free guidance, which has been proposed in prior work for goal-conditioned policies [1,2] and works quite well. Like DynaGuide, this also allows specifying multiple positive and negative goals.\n- DynaGuide’s stochastic sampling loop increases per-step computation by roughly $M\\times$ relative to a conventional goal-conditioned policy, resulting in significantly slower inference. This overhead should be discussed in the paper. By contrast, the sampling baseline’s need for multiple forward passes per step is not as serious a drawback since those passes can run in parallel, so its wall-clock latency should actually be much less than DynaGuide’s inherently sequential guidance routine. Apart from a discussion, including some wall clock time comparisons during inference would strengthen the experiment sections.\n\n*[1] Reuss, Moritz, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. \"Goal-Conditioned Imitation Learning using Score-based Diffusion Policies.\" In Robotics: Science and Systems. 2023.*\n\n*[2] Ajay, Anurag, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. \"Is Conditional Generative Modeling all you need for Decision Making?.\" In The Eleventh International Conference on Learning Representations.*", "was_revised": true, "final_justification": "During the rebuttal, the authors added some meaningful goal-conditioned baselines missing from the paper and some preliminary results for classifier-free guidance, with the assurance that full results will be added in the updated paper. They also added wall clock comparisons and results using the same amount of data (CALVIN-D only) as the baselines. Some additional clarifications around the experimental setting and wording of certain claims were also provided.\n\nThe main concern that came up during the discussion was the method used for guidance, particularly whether it is sensible to compare $t+H$ observation with the end goal when it may not be attainable within the horizon. The authors clarified that this is indeed a limitation of the method, and in the general case (like the real robot experiment), the user may need to provide guidance at intermediate states. In my opinion, this limits the applicability of the method since this guidance is quite difficult to provide in practice.\n\nTherefore, I maintain my rating leaning towards acceptance. I appreciate the fruitful discussion with the authors during the discussion period, which helped me better understand their method.", "rebuttal": "Thank you for the thorough comments, critiques, and questions. We appreciate your highlighting of the real robot experiments present in DynaGuide and the quality of experimental design. Below, we include additional experiments and discussion based on the points raised. \n\n# Addressing Main Limitation Remarks: Additional Experiments\n## Lack of GC baselines like HER\nIn the table below (additional results on exp. 3), we add `GC with HER`. To implement a HER-like algorithm, we train the GC policy on intermediate goals selected uniformly from a future observation in a trajectory. The results show some improvement on select tasks (notably the door task) but drops in other tasks. Similar results happen when the GC baseline is enriched with the `ABCD` data split (`GC with ABCD`). Overall, training on intermediate goals / enriched data had mixed results on performance. DynaGuide still consistently outperforms it. We will correct citations in the final publication. \n\n| Label         | button_on     | button_off    | switch_on     | switch_off    | drawer_open   | drawer_close  | door_left     | door_right    |\n|-|-|-|-|-|-|-|-|-|\n| DynaGuide [in paper]    | 0.33±0.031    | 0.36±0.032    | 0.24±0.017    | 0.26±0.024    | 0.26±0.015    | 0.27±0.017    | 0.25±0.018    | 0.19±0.015    |\n| Original GC [in paper]  | 0.023±0.0080  | 0.053±0.015   | 0.073±0.018   | 0.047±0.0067  | 0.00±0.00     | 0.0033±0.0033 | 0.12±0.012    | 0.080±0.016   |\n| GC with ABCD [new]  | 0.054±0.017   | 0.10±0.012    | 0.064±0.015   | 0.047±0.015   | 0.00±0.00     | 0.00±0.00     | 0.095±0.026   | 0.075±0.021   |\n| GC with HER [new]   | 0.0067±0.0042 | 0.087±0.030   | 0.027±0.011   | 0.020±0.0073  | 0.00±0.00     | 0.00±0.00     | 0.21±0.023    | 0.13±0.021    |\n\n## Wall Clock comparisons of DynaGuide\nWe have included some wall time analyses below on a 3090 GPU with no additional optimizations. \n+ DynaGuide. Setup: 4 stochastic sampling steps, 10 inference denoising steps (DDIM). Time for inference (for chunk of 16 actions): **1.2±0.08 seconds**\n+ DynaGuide-Sampling. Setup: 5 samples, batched. Time for inference (for chunk of 16 actions): **1.05±0.02** seconds \n+ Base policy / Goal Conditioned. Time for inference (for chunk of 16 actions): **0.09±0.01 seconds** \n\nWhen batched, DynaGuide-Sampling runs marginally faster than DynaGuide due to fewer runs through the dynamics model. However, for rare behaviors, the sampling baseline must run many samples to get a desired behavior. Scaling this through batching would require larger memory. In contrast, DynaGuide does not require any additional computation or memory scaling to search for underrepresented behaviors successfully (Exp. 5).\n\n## Dynamics Model Gets More Data\nThis is a concern shared by reviewer RYQt. In response, we have run DynaGuide using just the `D` split. Reducing the dynamics model dataset does reduce performance, as expected. However, as seen in the GC table above, adding the ABCD split to the goal-conditioned baseline does not improve performance much. Therefore, the dynamics model makes better use of the additional data compared to the GC baseline. \n\n| Label         | button_on     | button_off    | switch_on     | switch_off    | drawer_open   | drawer_close  | door_left     | door_right    |\n|-|-|-|-|-|-|-|-|-|\n| ABCD    [from paper]      | 0.70±0.031    | 0.70±0.038    | 0.74±0.027    | 0.76±0.037    | 0.60±0.015    | 0.66±0.029    | 0.74±0.025    | 0.68±0.026    |\n| D SPLIT ONLY  [new] | 0.66±0.023    | 0.69±0.038    | 0.54±0.043    | 0.58±0.031    | 0.51±0.022    | 0.53±0.024    | 0.65±0.019    | 0.57±0.036    |\n\n## Classifier-free Guidance\nThis is a reasonable baseline for the multi-objective setup. We implemented CFG and provide some basic results below for multi-objective, focusing on two tasks in Exp. 4. We tuned CFG’s guidance strength using a grid search to obtain these values. \n\n| Button On or Switch On    | Button | Switch |\n|-|-|-|\n| DynaGuide [paper]  | 0.54±0.02   | 0.26±0.04   |\n| CFG [new]  | 0.18±0.015   | 0.37±0.017   |\n\n\n| Door Left or Button On   | Door | Button |\n|-|-|-|\n| DynaGuide [new]  | 0.19±0.023   | 0.54±0.03  |\n| CFG [new]  | 0.28±0.032   | 0.37±0.0084  |\n\nAs you suggested, CFG can follow multiple objectives. However, its average per-task success (0.3) is lower than DynaGuide (0.38). One possible reason for this is the averaging of signals from the conditional models in the CFG implementation. In contrast, DynaGuide’s distance-based log-sum-exp prioritizes objectives that are the most reachable, leading to better performance with conflicting objectives. \n\n# Other Main Remarks and Questions \n## Novelty of Contribution\nDynaGuide uses classifier guidance like ITPS, but the guidance signal is propagated through a dynamics model instead of euclidian distance in the action space. DynaGuide uses a latent space like DINO-WM, but instead of running model-based planning, it modifies the behaviors of a base policy. DynaGuide also consolidates the latent distances of multiple objectives (eq. 2), which is not possible in DINO-WM. \n\nIn general, DynaGuide builds upon DINO-WM and ITPS, but combining these approaches required distinct innovations. The final result yielded a novel approach to guiding diffusion robot policies. We also provide real robot results, which were not provided by DINO-WM. \n\n## Improper comparison of future state to desired end state\nThis is a valid concern about the method. In the CALVIN environment, early task disambiguation allowed us to let $t+H$ be the last state of the trajectory (mentioned in Lines 149-150; this will be clarified in the final publication). Because the dynamics model predicted the final state, it was directly comparable to the desired final states. This trick was also used in DINO-WM [1]. \n\nOn the real robot experiments, the longer horizon and the wrist-mounted camera meant that $t+H$ could no longer be the final state. To account for the exact issue raised by your comment, we sampled states throughout desired behavior trajectories as the guidance conditions. The log-sum-exp acts like a soft minimum for the latent distances, so it automatically downweights states that are too far away / unreachable. \n\nThese details are somewhat hidden in the current paper version so we will add more details in the final publication.\n\n## Expanding DynaGuide to stochastic environments\nThe current dynamics model is best for deterministic environments, but Eq. 1 can also be interpreted as a stochastic log-probability loss with unit variance. To properly adapt the dynamics model to a stochastic environment, we would need to make the variance learnable, i.e. getting the model to provide a measurement of uncertainty through specialized architecture like a VAE. This is an interesting direction for future work. \n\n\n# Other Remarks\n## Lack of position guidance baseline in moving object experiment\nPosition Guidance requires knowing the exact 3d location of an object to define a euclidian distance objective for use during steering. In ITPS, a human provided this 3d location in every environment rollout, which enabled its use on movable objects. In all our baselines, we don’t assume privileged state knowledge. For fixed objects, we get their locations by using the final gripper positions in the training data. This heuristic is not available for movable objects. \n\n## Choice of action horizon value\nThis action prediction horizon value is 16, and while we never conducted extensive experiments for this value, we found that the guidance is weaker for shorter values like 8 and the diffusion policy performs worse for very long values like 24 or 32, so we picked 16 as a balancing point. \n\n## DynaGuide requires base policy to have ability to solve target task\nFor steering approaches, it’s generally assumed that the base policy has the skills to achieve a behavior. Otherwise, the base policy will not be of much use. The referenced claim in lines 34-42 focuses on assuming knowledge of the *steering objective*, which is necessary for goal-conditioning because it is trained on the steering signal. In contrast, DynaGuide can handle *novel steering objectives* that leverage existing skills. \n\nAn elucidating example: without using CFG, our goal conditioning won’t be able to *avoid* a behavior unless we train it specifically. However, we *can* avoid a behavior with DynaGuide by negating the guidance signal. This is an example of an *unanticipated steering objective* being applied to *known skills*. This should be clarified in the paper and we will do this in the final publication. \n\n## Goal Assumptions\n>  Assuming access to goal observations for guidance is a much stronger assumption than relying on goal specification via natural language, for example.\n\nYes, this is absolutely true. Although, the data needed to train a language-conditioned model requires more assumptions (task segmentation, good task labels, etc). We were having this interpretation in mind for line 103, although we see the ambiguity and will correct accordingly. \n\n## Typing Errors in Algorithm 1\nThanks for finding these typing errors. We will fix them in the final paper. \n\n## Formatting concern\nBased on our observation of past NeurIPS papers, our interpretation of the CFP remarks was that they referred to main paper margins / main text size. We are happy to change the caption formatting if it was the incorrect interpretation. \n\n# Summary\nThank you for the very detailed review and thoughtful questions. We have done our best to answer them by providing additional results including wall clock analysis, results with a different data split, stronger goal-conditioned baselines, and others. We also answer other conceptual questions and provide some discussion. We can provide further clarification if anything was not clear. \n\n[1] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning."}, {"review_id": "NAOT3r4RnL", "reviewer": "Reviewer_ma6B", "summary": "The authors propose DynaGuide to steer pre-trained diffusion policies at inference time without needing to retrain them. It uses a separately trained dynamics model to predict the future outcome of an action. The goal is specified through positive and negative goal images. By calculating the distance in a latent space between the predicted outcome and the goal images, the system generates a guidance signal. The gradient of this signal is used to directly modify the action denoising process of the diffusion policy, steering it towards desired behaviors and away from undesired ones.", "questions": "- **Negative Goal Selection:** How sensitive is the method to the choice of negative goal images? For high-precision tasks, a failure state might look very similar to a success state, which could make providing effective negative guidance difficult.\n    \n- **Alternative Guidance Modalities:** As noted in the paper's future work, can guidance be extended beyond static images to other forms like language (\"push the red block gently\") or kinesthetic demonstration to provide more fine-grained control over the trajectory?", "limitations": "- **Lack of Trajectory Control:** The guidance mechanism steers towards a final state but offers no direct control over the path taken to reach that state.\n    \n- **Computational Cost:** To use strong guidance without destabilizing the trajectory, the method relies on MCMC sampling (running each denoising step multiple times), which increases the computational overhead at inference time.\n    \n- **No Online Learning:** The guidance is applied only at inference time and is forgotten afterward. The base policy does not adapt or improve from the guidance it receives.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- **Plug-and-Play:** It can steer existing, off-the-shelf diffusion policies without any modification or fine-tuning.\n    \n- **Robustness to Low-Quality Goals:** By separating the policy and the dynamics model, the system is more robust to \"under-specified\" goal.\n    \n\n### Weakness\n\n- **Guidance is Outcome-Based:** The method can only specify the desired *final outcome* of a task via an image. It cannot control *how* the robot should perform the task (i.e., the specific trajectory or method of execution).\n    \n- **Reliance on Dynamics Model:** The performance is entirely dependent on the quality and data coverage of the dynamics model. If the dynamics model cannot accurately predict the consequence of an action, the guidance will be ineffective.\n    \n- **Limited Baselines:** The paper does not compare against methods that more tightly integrate a world model into the diffusion process itself, like the original diffuser paper + inpainting [1] , which could be a relevant alternative approach.\n\n[1] Janner, Michael, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. “Planning with Diffusion for Flexible Behavior Synthesis.” arXiv, December 20, 2022. http://arxiv.org/abs/2205.09991.", "was_revised": true, "final_justification": "The author makes a great effort to clearify my concerns. I believe this paper would be a great fit for the conference. Thus, I maintain my score.", "rebuttal": "Thank you for your positive feedback and thoughtful analyses of DynaGuide. We appreciate your highlighting of Dynaguide’s robustness to lower quality goal conditions that would otherwise cause goal-conditioned baselines to fail. Below, we answer the questions and address the limitations. \n\n# Answers to Questions \n## Sensitivity to negative goals\nEmpirically, we didn’t have to cherry-pick the negative goals to get DynaGuide working on the multi-objective setup. Sampling the negative states as the final states in a randomly selected set of validation trajectories was sufficient. \n\nThe performance of DynaGuide depends on the ability of the latent space to separate out desired and undesired outcomes. We use the DinoV2 latent space, as past work [1] had yielded successful results in the planning domain. If a particular task has similar success and failure states, then a special latent space should be constructed that separates them appropriately. This latent space will probably need to be trained in a task-specific way. The DinoV2 space was sufficient for our experiments, but improving the latent space is a meaningful direction for future work. \n\n## Alternative guidance modalities\nAs long as you can embed the guidance conditions into the latent space, any format of guidance condition should be valid. For example, if a CLIP embedding is used, then text can be used as guidance conditions as well as images. The latent space can also encode history and kinematic properties of the trajectory, if trained with the correct objectives. We leave the exploration of latent spaces to future work. The DynaGuide method is general and should work on these specialized latent spaces. \n\n# Discussion of Limitations and Weaknesses \n## Guidance is Outcome-Based / Lack of Trajectory Control\nIn the simplest formulation DynaGuide, we steer the policy with guidance conditions that represent outcomes. However, because Eq. 2 is a soft minimum, it is also possible to add intermediate guidance conditions that represent how a task should be done. We actually did this for the real robot setup due to the wrist camera’s end observation being not a good representation of the task. We found that these intermediate guidance conditions were critical in getting the desired performance improvement. DynaGuide as a framework supports guidance at any point(s) in the trajectory, as long as they can be represented as anchor points in a latent distribution. \n\n## Reliance on Dynamics model\nWe agree with this concern. By construction, DynaGuide relies on the dynamic model’s predictions for good guidance. However, the dynamics model can be trained on additional lower-quality and/or less-relevant data compared to an expert policy, making it easier to improve the robustness of the dynamics model compared to doing the same for the expert policy. In our Calvin experiments, we train the dynamics model on interactions in other domains (ABC split). In our real robot experiments, we add data from other object interactions. In both cases, the dynamics model benefited from this additional data. \n\n## Limited Baselines\nThere is indeed a line of work that integrates world models into the planning and diffusion process. Because we were focused on the ability to guide existing policies, we were less thorough in the planning-based baselines like DINO-WM and Diffuser. On the bigger picture, these approaches are still relevant and we will try to implement & test some of these approaches for the final publication. \nIn the meantime we added the following baselines in response to other reviews\n+ Adding DynaGuide-Sampling (GPC) to real world results (see: RYQt)\n+ Reducing the dynamics model training dataset to `D` from `ABCD` (see: yJvQ and n1oB) \n+ Increasing the data of the goal-conditioned policy to `ABCD` (see: yJvQ and n1oB) \n+ Adding HER-like training to goal-conditioned policy (see: n1oB) \n+ Testing classifier-free guidance on the multi-objective task (see: n1oB) \n\n## Computational Cost\nThe MCMC sampling is indeed adding to the computational overhead, although unlike sampling, DynaGuide’s computational cost is not dependent on the rarity of the target behavior or the difficulty of the task. When run on an RTX 3080 with a real robot, this inference process still runs in real-time due to the model predicting entire action chunks at once. For a profile of run times, see our response to Reviewer n1oB. \n\n## No online learning\nIt is true that the guidance is forgotten afterwards. However, the guidance is repeatable because the *guidance conditions are stored.* Guidance conditions can be added or removed dynamically based on past performance–and this is a notion of adaptation / persistence. This online learning may not be as expressive as finetuning, but it is more explainable, robust, and easily reversible if needed. It adds to the modular capabilities of DynaGuide. \n\n# Summary\nThank you for your thoughtful observations and questions. We have done our best to provide answers to questions about DynaGuide’s limitations and possibilities for other modalities and applications. We welcome any follow-up questions or comments. \n\n\n[1] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning."}]

Author rebuttal:

Meta-review: (a) This paper proposes DynaGuide, which combines action denoising gradients from a pre-trained policy with an inverse dynamics based guidance gradient. This guidance grad increases the probability of meeting guidance goals. The dynamics model is trained to predict future outcomes and compares them to guidance outcomes. The key hypothesis is that this helps the policy in reaching under represented parts of the manifold and also helps when there are multiple objectives to optimize for. 

(b) This paper presents a relatively new way of combining classifier style guidance with a learned dynamics model, and moreover it can leverage pre-trained diffusion policies. Multiple simulated tasks and real robot experiments were conducted, with added baselines during rebuttal (HER, classifier-free guidance, dynamics trained only on D split). The paper is also clearly written, includes detailed appendices, ablations, and released code.

(c) Initial submission lacked strong baselines in robot experiments (e.g., goal-conditioned, HER, classifier-free guidance). Some were added in rebuttal, but not fully comprehensively. DynaGuide requires access to goal states (images). In general this may be less feasible than language based specifications.

(d) The paper is technically sound, makes a meaningful/interesting contribution, and provides extensive experiments and rebuttal clarifications. I like that it exploits pre-trained policies and so it has the potential to be generally useful in practice. It is also nice to see this method being applied on real robotics problems. The authors also seemed to have addressed all the major points raised during the rebuttal process. 

(e) 
- RYQt was concerned about missing real world baselines. Authors added DynaGuide sampling baseline, explained lack of goal-conditioned real robot baseline due to pre-trained policy choice, and discussed robustness of hyper parameters. Reviewer maintained score but acknowledged improvements.

- n1oB raised issues around conceptual novelty, reliance on goal states, missing HER/CFG baselines, and wall-clock overhead. Authors added HER and CFG baselines, clarified guidance assumptions, provided timing comparisons, and discussed applicability to stochastic settings.

- ma6B generally positive, with concerns about reliance on dynamics model, lack of trajectory control, and missing baselines. Authors clarified sensitivity to negative goals, extended discussion on alternative modalities, and added new baselines.
Decision: Accept (poster)