Act to See, See to Act: Diffusion-Driven Perception-Action Interplay for Adaptive Policies
Authors: Jing Wang
Weiting Peng
Jing Tang
Zeyu Gong
Xihua Wang
Bo Tao
Li cheng
Venue: NeurIPS 2025 poster ()
Keywords: Imitation Learning; Diffusion Policy; Vector–Jacobian Product; Variational Inference; Latent Stochastic Differential Equation; Contrastive Learning
Primary area: reinforcement_learning
TL;DR: We propose an action-guided diffusion policy that leverages a variational latent SDE and Vector-Jacobian Product-based update to model the dynamic interplay between perception and action for real-time robot manipulation.

Abstract: Existing imitation learning methods decouple perception and action, which overlooks the causal reciprocity between sensory representations and action execution that humans naturally leverage for adaptive behaviors. To bridge this gap, we introduce Action-Guided Diffusion Policy (DP-AG), a unified representation learning that explicitly models a dynamic interplay between perception and action through probabilistic latent dynamics. DP-AG encodes latent observations into a Gaussian posterior via variational inference and evolves them using an action-guided SDE, where the Vector–Jacobian Product (VJP) of the diffusion policy's noise predictions serves as a structured stochastic force driving latent updates. To promote bidirectional learning between perception and action, we introduce a cycle-consistent contrastive loss that organizes the gradient flow of the noise predictor into a coherent perception–action loop, enforcing mutually consistent transitions in both latent updates and action refinements. Theoretically, we derive a variational lower bound for the action-guided SDE, and prove that the contrastive objective enhances continuity in both latent and action trajectories. Empirically, DP-AG significantly outperforms state-of-the-art methods across simulation benchmarks and real-world UR5 manipulation tasks. As a result, our DP-AG offers a promising step toward bridging biological adaptability and artificial policy learning. Code is available on our project website: https://jingwang18.github.io/dp-ag.github.io/.

Act to See, See to Act: Diffusion-Driven Perception-Action Interplay for Adaptive Policies
Jing Wang'
Weiting Peng?
Jing Tang² Zeyu Gong?
Xihua Wang'
Bo Tao Li Cheng'
University of Alberta
Huazhong University of Science and Technology
[jing39, xihua, Icheng5] @ualberta.ca (u202210565, j tang, gongzeyu, taobo) @ @hust.edu.cn
Abstract
Existing imitation learning methods decouple perception and action, which over- looks the causal reciprocity between sensory representation and action execution that humans naturally leverage for adaptive behaviors. To bridge this gap. we intro- duce Action-Guided Diffusion Policy (DP-AG), unified representation learning that explicitly models dynamic interplay between perception and action through probabilistic latent dynamics. DP-AG encodes latent observations into Gaussian posterior via variational inference and evolves them using action-guided SDE, where the Vector-Jacobian Product (VJP) of the diffusion policy's 's noise predictions serves structured stochastic force driving latent updates. To promote bidirec- tional learning between perception and action, we introduce a sycle-consistent contrastive loss that organizes the gradient flow of the noise predictor into co- herent perception-action action loop, enforcing mutually consistent transitions in both latent updates and action refinements. Theoretically, bound for the action-guided SDE, and prove that the contrastive objective enhances continuity in both latent and action trajectories. Empirically, DP-AG significantly outperforms -tate-of-the-art methods across simulation benchmarks and real-world UR5 manipulation tasks. As result, our DP-AG offers a promising step toward bridging biological adaptability and artificial policy learning. Code is available on our project website: httpp://jingeee 1........ github.... i io/dp-//ppa dp-ppppaagg ag gi thub. io/
derive variational lower
Introduction
Imitation learning (IL) enables agents to replicate expert behavior from demonstrations. Direct
mapping methods, such as [Codevilla et al. 2018 2018 Mandlekar et al. 2022 Florence et al., 2022]. learn
direct mapping between observations and actions. In contrast, generative models such as Diffusion
Policy (DP) [Chi et al. 2023) and flow-matching methods [Hu et al. 2024 Zhang et al. 2025 model
action distributions improve continuity across time steps.
(VLA) Kim
et al. 2024 Black et al., 2024 improves perception by leveraging Vision-Language Models (VLMs)
interpret environmental cues and high-level instructions. Despite progress, existing methods treat observation features as static during each action sequence generation (extracting them from a single time-point observation and holding them fixed while generating short sequence of actions), thus overlooking the opportunity for the intermediate action feedback to refine perception.
Robust decision-mate relies continuous interplay between perception and action (O O'regan and Noë| [001]. Humans naturally embody this principle by dynamically refining their environmental understanding through feedback from their own actions [Brooks] 1991]. Motivated by this, we propose Action- Guiidn Diffusion Policy (DP-AG), a representation learning framework for IL that explicitly models the perception- prrception-action action interplay through probabilistic latent dynamics. We build upon DP
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

VJP
T UzI
des
Increment
Direct Policy
Ee (a 2t)
TT((2))
a
6
Eg(aff
K Iteruitions
K Iterations
K Iterations
Zt
21
(a) Direet Policy Mapping
(b) Diffusion Policy
(c) Perception-Action Interplay
Observation Feature Extraction
Observation Space
Encoding
Zt
Scene Camera
Wrist Camera
Robot Pose
Figure 1: Use of Observation Features. (a) Conventional methods map observation features directly to actions. (b) DP models action distributions through incremental denoising from white noise, conditioned observation features. (c) DP-AG refines observation features via noise predictions, establishing a mutually reinforcing cycle between perception and action.
because it models the continuity within each short-horizon action sequence, where intermediate action feedback can be derived from its noise predictions at each diffusion step. DP-AG first grounds observation features in Gaussian posterior via variational inference, capturing uncertainty from observation inputs. To enable dynamic perception, we introduce an action-guides stochastic differential equation (SDE) in which latent features evolve across diffusion steps, driven by action- conditioned noise predictions. Here, the Vector-Jacobian Product (VJP) from the diffusion model acts structured stochastic force that shape feature evolution.
Although raw observations contain all available information, their usefulness for policy learning depends how they internally represented. Unlike static encoders that keep features fixed over an entire action sequence generation, DP-AG continuously refines them using action feedback. This process mirrors active perception in biological agents, where even if the external scene remains unchanged, sensory inputs are reinterpreted in the context of ongoing actions. This captures the essence of Act to See, See Acf: the where see refers to the evolving latent representation refined through action feedback rather than new external sensing. As shown in Figure I DP-AG conditions latent evolution action- setionnguiiedd noise, using the continuity of action diffusion to keep perceptual dynamics aligned with the denoising of actions. A cycle-consistent contrastive loss ensures that latent evolution remains coherent with action diffusion, preventing excessive drift and reinforcing the perception-action loop. Combined with variational inference, these components yield a principled and adaptive representation that produces smoother, more context-aware trajectories, as confirmed by both synthetic and real real-robot experiments.
observation is continually reinterpreted actions unfold,
Theoretically, DP-AG introduces action- iiston-ggidedd latent SDE, derives principled variational lower bound, and rigorously proves that the proposed cycle-consistent contrastive loss enforces continuous and coherent trajectories in both perception and action spaces. Empirically, DP-AG consistently outperforms state-of-the-arr methods in success rate, convergence speed, and action smoothness, achieving gains of 6% Push-T and 13% in Dynamic Push-T benchmarks, and delivering least 23% higher manipulation success and approximately 60% smoother actions on real-world UR5 robot tasks compared the baseline DP. Our contributions summarized below:
We propose novel observation representation learning that establishes closed perception-action loop by refining latent observation features via VJP-guided noise predictions derived from DP.

---

We formulate an action action-guided latent SDE, derive a variational lower bound, and prove that sycle-consistent InfoNCE enforces mutual smoothness in both latent and action trajectories. We validate the effectiveness of DP-AG in both simulation and real-world scenarios, demonstrating consistent and significant improvements in task success rate, convergence speed, and smoothness of generated actions compared to state-of-the-art methods.
2 Related Work
2.1 Imitation Learning
IL provides an alternative to reinforcement learning (RL) by removing the need for explicit reward
signals (Sutton et al. 1999 Osa al. 2018 Existing methods include Behavioral Cloning (BC) [Bain
and Sammut 1995, Mandlekar et al. 2022 Florence et al. 2022] which learns the mapping from
observations to actions, and Inverse RL Arora and Doshi 2021 that infers implicit rewards from expert demonstrations. Recent work (Kim et al. 20024 Black et al. 2024] extends BC by incorporating VLMs into the perception pipeline, pushing robotic manipulation toward near-human capabilities. However, existing methods decouple perception and action: the observation feature is frozen during each action sequence generation, without adapting to the evolving action refinements. This constraint often leads to abrupt discontinuous motions. In contrast, we propose pereeption-action interplay where latent continuity and action smoothness reinforce each other, yielding more coherent behavior.
2.2 Generative Models in Policy Learning
Generative models have shown promise in modeling continuous action trajectories. DP Chi et al.
2023] leverages Denoising Diffusion Probabilistic Models (DDPMs) [Ho et al. [020] to iteratively
refine action predictions using diffusion models. Policy-guided diftusion [Jackson al. 2024
generates synthetic trajectories aligned with a target policy for offline RL, while Diffusion- QL WWang et al. [023] uses diffusion models to learn expressive O-functions. Recently, flow-matching meth- ods Lipman et al., 2023 Hu et al. [024] Zhang et al.] 2025 model actions as deterministic flow ordinary differential equations (ODEs), achieving faster inference without comprising generation quality. Building on these, DP-AG extends action-space continuity of DP into the latent observation space, forming dynamic perception-action loop that further enhances action smoothness.
2.3 Latent Observation Modeling in Agent Control
Modeling latent observations for control has been explored through planning and generative meth- ods. PlaNet [Hafner et al. 2019] and Dreamer [Wu et al. 2023 use variational autoencoders (VAEs) Kingma et al. 2014] to simulate future states for RL. Embed to Control [Watter et al.) [0155 uses VAEs learn latent states from images for control policy design. In contrast, DP-AG closes the perception-actio action loop by reparameterizing latent observations with VJP-guided noise from DP, extending variational inference capture action- action-guided evolution.
3 Preliminaries
In imitation learning, the goal is to learn policy no lot) that replicates expert behavior, where or and a₂ denote the observation and action at a static time point t. An encoder fy extracts static features 2n = (ot), which are held fixed while the policy generates the corresponding at meaning there is no feedback from action generation to perceptual features within the same short-horizon. Given an expert demonstration dataset D {(o,, a.)), standard BC minimizes the supervised loss:
(B vb) aoD (f (o.)) a, 112]
(1)
Rather than the direct mapping, DP models the conditional distribution pe (2)) as a denoising diffusion process. Starting from white noise e NO, I), actions are generated over K denoising steps using a learned noise predictor Eg,, conditioned on observation features.
K
à = a. at
Σ g(b) - co 24, k),
where a. is the white noise, g(k) is a noise schedule, and c0 predicts the noise at diffusion step k. The diffusion model is trained by minimizing the noise matching objective (Ho et al. 2020
Lop 0, v) (. ( k) - -El
(3)
3

---

Observations
Noise Predictor
(Conditional UNet)
Positional Encoding
Global Conditioning
MLP
C1
FILM
FILM
Ct
Feature Encoding
Perception-Action Interplay
Cone
Cony
E
MLP
p(a:)
VIP Reparameceriizaiion
FILM
m(2)) + (0r ( die 02
(s))
C₁
Conv
Cuay
Cone Convolatoral Nearal
co(a, 21, k)
VIP
)a
Ea f 24, 2,,
FILM
FILM
FILM Festurewise
Madalanien
Diffusion Policy (at time ()
Coms
Forward Training
cp (a 21, k)
(Noise Adding)
Reverse Infermnee
Cycle-Consistent Contrustive Loss
(Noise Removal)
Hinding Space
RJ
at
a N0
Figure 2: Method Overview. While Diffusion Policy (DP) generates actions from static observation features, DP-AG establishes dynamic perception-action loop by guiding feature evolution via the VJP of DP's predicted noise. To reinforce interplay, a cycle-consistent contrastive loss aligns noise predictions from static and evolving features, enabling mutual perception- influence.
Through the progressive refinement of noisy actions, DP estimates the continuity underlying discrete expert actions, which allows the policy to generate smooth transitions based on discrete observations.
To leverage this continuity for perception, introduce DP-AG, which extends DP by propagating action refinement into the latent observations through Vector-Jacobian Products (VJPs) of noise predictions, enabling features evolve dynamically alongside actions throughout diffusion. Although we instantiate DP-AG with DDPMs, the method generalizes to other generative families such as score- based diffusion (Song al. 2021 and flow matching [Lipman al. 2023), provided that intermediate action refinements are available (see Appendix A for extending DP-AG to flow matching).
4 Our Approach
We propose Action-Guided Diffusion Policy (DP-AG), latent generative model that explicitly captures the dynamic interplay between perception and action. As illustrated Figure 2 DP-AG extends static variational inference by introducing an SDE over observation features, where latent representations evolve under action-guided VJPs of the diffusion noise predictions rather than being driven by static white noise. This replaces random perturbations with structured stochastic forces that align uncertainty in the latent dynamics with the task's perception-action structure. To further reinforce this interplay, we introduce cycle-consistent contrastive loss that enforces agreement between action noise predictions conditioned on static and evolving latents, thereby closing the perception-action loop and enabling coherent updates throughout the diffusion process. Note that, this interplay is modeled within each action sequence generation az, occurring between time points t and + across the diffusion steps that connect them.
4.1 Latent Observation Modeling via Variational Inference
In real-worke settings, observations are high- righ--imensional and noisy, which makes impractical to rely on deterministic representations. To model this inherent uncertainty, we begin by constructing variational posterior over observation features:
go(2)oo)) NN oz()))
where (z) and (2)) capture the mean and variance of observation features. Sampling latent observations via the reparameterization trick (Kingma et al. 2015):
2 pz(Z)) os( Oe (0,
(5)

---

provides stochastic encoding of these features, which ensures that the policy can reason about uncertainty in sensory inputs. However, still treats perception as static: once extracted, observation features remain fixed during the corresponding action generation and cannot adapt to refinements.
4.2 Action-Guided Latent Evolution through Stochastic Differential Equations
To capture continuity from action diffusion, we model latent observation evolution using a SDE, where the drift is guided by action refinement. each step k. the latent feature 2; evolves as:
d
VJP(á 21) dt + (2)) dW..
(6)
where the drift term VJP 4) is computed from the action noise prediction:
(Deg(af, 20, (
T
VJP(a z1)
Ea(â 2e, k).
Oz.
This follows stochastic adjoint sensitivity methods [i ct al. 2019], which show that VJP-based updates efficiently propagate gradients through SDEs without computing full Jacobians; modern autodiff frameworks make these computations practical with negligible overhead.
Beyond efficiency, the VJP serves task driven attentional force, guiding latent features toward the parts of the observation that most effectively reduce uncertainty in predicting the next-step action noise. Because its drift is computed from the DP's instantaneous noise prediction, latent evolution is phase-aligned with action diffusion: moving forward during noise addition and reversing during action denoising. Unlike stochastic adjoint methods that separate forward evolution from backward updates, leaving latent features fixed during action inference, DP-AG's VJP delivers immediate feedback-driven updates at every diffusion step. This mechanism preserves the full Jacobian structure in real time, keeps latent perception features phase-aligne with ongoing action diffusion, and captures the instantaneous sensitivity between actions and observation features.
Thus, instead of static variational posterior, we reparameterize with an action -ooindd noise term:
x (2, (2.) VJP =t);
where " controls VJP strength
(see ablation studies in Appendix [or its effect). This formulation allows the latent observations to evolve with the action refinements; the variational lower bound for the corresponding optimization detailed in Section 5.1
4.3 Cycle-Consistent Contrastive Learning
While VJP VJP-guided SDEs enable dynamic latent evolution, they do not guarantee coherent alignment with the action diffusion, which is significant for achieving perception- interplay. To bridge this gap, we introduce cycle-consistent contrastive loss that aligns noise predictions with evolving latent dynamics at each diffuusin step k:
El = =āt 2; k)) is the noise prediction conditioned on the static latent 2₁ at the step l;; Eμ co 2/ k) is the noise prediction conditioned the VJP-guided latent 2t at the step k.
An InfoNCE [Chen et al. 20200 loss promotes the similarity of matched pairs while pushing apart mismatched ones within the same mini-batch (see Appendix H. temperature parameter tuning):
1
exp (sim (Ek, Ef. /T)
Leeat
Σ log
B
1= Ejji exp sim ( ( /T)
where sim(u, v) = ||]
is cosine similarity, with 7 as the temperature and B the mini-batch size.
Our contrastive loss enforces cycle consistency between perception and action: noise predictions conditioned on static latents guide continuous latent updates via VJP, and the refined latents improve subsequent noise predictions. Unlike MSE-based objectives that rigidly match features, the con- trastive loss promotes clustering effect: it pulls evolved features toward their static counterparts to preserve semantic grounding, while pushing them away from features other observations to maintain task-relevant distinctions. This bounded adaptation keeps updates semantically meaningful, enabling smooth refinement without arbitrary drift and reinforcing coherence in both latent and action trajectories (see Section 5.2 for theory and Section 6. for empirical validation).
5

---

4.4 Training Objective
The overall training objective combines three components: noise matching from DPs, variational optimization for latent evolution, and contrastive loss for the perception-action interplay:
LDP-AG
LDP + AcontCoom + CKL
(10)
Jointly optimizing these terms enables DP-AG to co-evolve perception and action, resulting in more adaptive policies (see Appendix HH.5 for ablation studies hyperparameter tuning).
4.5 Intuitions Behind DP-AG
We now explain why DP-AG works in practice. A static encoder yields a one-shot representation of the observation that stays fixed during action generation, forcing the policy to commit to an interpretation before actions unfold. DP-AG instead refines the latent representation step by step, guided by feedback from the evolving action sequence. Think of driving a car: the view through the windshield may remain unchanged, yet the driver's ' focus shifts as they act, paying more attention to the curve's edge while turning, or nearby cars when accelerating.
The VJP supplies both the direction and magnitude of this latent drift: it points features toward the adjustments that most reduce action uncertainty, with larger updates when the model is less confident and smaller ones when it is certain. Meanwhile, the cycle-consistent contrastive loss anchors these adaptations, keeping updated features aligned with their static counterparts and preventing excessive drift. Together, this action-drive refinement and bounded consistency yield smoother, more stable, and context-aware trajectories, as confirmed in our experiments.
Theoretical Insights
5.1 Action-Guided Variational Lower Bound
To learn a dynamic latent observation that evolves with the action diffusion, we define a posterior 4o (zf [2,, af) conditioned on both the static latent 2 and the denoised action at at step k.
Variational Objective. The marginal likelihood of the observed noise throughout the diffusion is:
logp(ex|==)) = log p(Enl=)) p( | d=t.
(11)
Applying Jensen's inequality with the posterior, we obtain the ELBO:
log ((E((()))) 2 (5)()))).aa)) (log [lon(() - KL (go (Et1) l2,, at) (1))l220 (1 (2))
(12)
Maximizing the ELBO encourages z to capture action-driven updates while staying close to the prior conditioned on 24. Full derivation of the ELBO is provided in Appendix B
Likelihood Term. The likelihood term p(εx12?) ) models the distribution of action noise conditioned
on the evolving latent at, and can be optimized using a noise matching loss similar to DP:
fu (o
[]co 2" k) - -|/
(13)
This likelihood term encourages absolute accuracy in noise prediction but overlaps with the contrastive
loss that emphasizes relative similarity (see the detailed ablation in Appendix H.3) To avoid
redundancy and instability, we use only the contrastive loss.
Kullback-Leibler Divergence Term. The variational posterior is modeled as:
@o ( Izt, at) A (up(2, at) oz at)),
(14)
while the conditional prior be modeled Gaussian distribution:
2.) = M(,, T),
(15)
which encourages the evolved latent stay near the static latent initially encoded from the observation.
The Kullback-Leibler (KL) divergence thus becomes (derivation of Equation i6 in Appendix C
(L(go 11 = Σ (00.. + (Ho. - 20,)² 2- 1 log o..))
(16)
where di the latent dimension, and (+(() and log (.) are parameterized by a separate linear layer.

---

Ourput
Ourut
Besc Flow
VIR-
Guided
Flow
Grounl Ro Preturr
Figure 3: Regression results on irregular spirals. Left: Trajectories and latent dynamics predicted by the Base Flow. Right: The VJP-Guided Flow continuously refines latents through output-gu guided conrections, which results in smoother and more coherent trajectories in both output and latent spaces.
5.2 Contrastive Similarity Promotes Latent-Action Continuity
In this section, we theoretically demonstrate that minimizing our cycle-consistent InfoNCE loss leads to smooth and coherent transitions in both latent and action spaces. The key insight is that minimizing the contrastive loss enforces strong similarity between static and dynamic noise predictions; under Lipschitz continuity, this similarity tightly bounds the drift in latent dynamics.
Noise Similarity Lower Bound. We first show that minimizing the contrastive loss imposes a lower bound on the similarity between noise predictions E and E Proof of Lemma | is in Appendix D
Lemma (Noise Similarity Lower Bound). For unit-normalized vectors Ei, and e and a temperature T > O, the InfoNCE loss satisfies Coom S foror some small constant o, then for each i E {{,..., B}. the similarity between corresponding pairs is bounded accordingly:
sim (k), 2 1) -a)-1. 1.
(17)
posizive pair similorit
Continuity Induced by Noise Alignment. We then demonstrate that the lower bound on the noise similarity provides an upper bound on the latent drift, meaning that the contrastive alignment also leads to smooth transitions in the latent space. Proof of Theorem iss in Appendix E
Theorem 1 (Continuity Upper Bound), Suppose co is L-Lipschitz with respect to 2, and that 2, zk+1 evolves via the VJP of cg. Then under the contrastive constraint Cioas I a. we have:
|| 1 < L L3 z,,k) - ca zt. k)]l < 22 - In
+ Ta).
(18)
Together, Lemma I and Theorem Checorem[estare estabbish a theoretical connection between minimizing our cycle- consistent contrastive loss and promoting temporal continuity in both latent and action trajectories.
6
Empirical Evaluations
6.1 Latent and Action Continuity on an Irregular Time-Series Regression
To empirically validate DP-AG's ability to improve latent and action continuities, we conducted experiments on the irregular spiral dataset, originally introduced in the neural ODE evaluations [Chen ct al. 2018). This dataset is designed to evaluate continuous- time dynamics under irregular sampling, making it well-suited for evaluating the impact of the VJP-guided evolution on latent dynamics.
Dataset. Following [Chen al. 2018), the dataset contains 1,000 samples of 2D spiral trajectories. with half following clockwise pattern and the other half counterclockwise, each sampled at 100 irregular time points. The spiral exhibits continuous radial growth with Gaussian noise (a E [0.02, 0.1]) added to simulate real-world variability with randomly assigned angular velocities.
Method. We compare
models:
Base Flow: An LSTM with 128 hidden units and 2 layers is used to extract discrete latent features 21, followed by a Multi-Layer Perceptron (MLP) to predict the regression output (2))..
VJPP-uided Flow: Augmenting the base flow with a VJP-guided SDE, where the latent feature
yo(2+) dt a=(2)) dW, with the base drift (=()))) and base log- variance log (2)) are each predicted by separate linear layer.
E, evolves according to: dē. = (0)((=))
7

---

Can
Dynamic Push-T Tool Hang
Square
Transport
Lift
Franka Kitchen
Push-T
Figure 4: Benchmark simulation environments include Robomimic, Franka Kitchen, Push-T, T, and Dynamic Push-T, with task and dataset details provided in Appendix F
Table I: Target coverage score on Push-T and Dynamic Push-T tasks. img and kp refer the observation modalities: RGB images or 2D keypoints.
Type
Method
Push-T
Dynamic Push-T
img
kp
img
LSTM-GMM [Mandlekar, 2022]
0.69:10.02
0.67=:0.03
0.34
Mapping
IBC [Florence, 2022]
0.75+0002
0.90+0.02
0.52++0.98
BET [Shufiullah, 2022]
0.80
0.79 1..02
0,58.11.35
Flow
FlowPolicy [Zhang. 2025]
0.85=00001
0.88+0.01
0.53=t0.88 88
AdaFlow [Hu, 2024]
0.87=10.02
0.91
0.67=
Diflusion
DP [Chi, 2023]
0.87.10.04
0.95.10.03
0.65.10.85
DP-AG (ours)
0.93=+0.02
0.99+0.011
0,80=+0.53
Both models are trained for 100 epochs with Adam optimizer (learning rate 10 and batch size 32).
Results. As shown in Figure VJP-Guided Flow generates smoother and coherent trajectories than the Base Flow, reducing the MSE from 0.0095 to 0.0052 (a 45.3% improvement). Importantly, latent visualizations show that while the Base Flow yields scattered latent states, VJP guidance shapes a structured latent manifold aligned with output predictions. This illustrates our perception-action interplay: predictions provide gradient feedback that dynamically refines latent embeddings through the VJP. As result, latent embeddings and predictions evolve in synchrony, enhancing continuity in both representation and regression, which is consistent with our theoretical findings in Theorem
6.2 Experiments on Simulation Benchmarks
We then evaluate our DP-AG on the simulation benchmarks, illustrated in Figure 4
Robomimic. Robomimic [Mandlekar et al. 2022 is large-scale benchmark for robotic manipulation consisting of five tasks with nine datasets: Can (ph/mh)), Square (ph/mh), Transport (ph/mh), Tool Hang (ph), and Lift (ph/mh). Here, ph denotes proficient human demonstrations, and mh indicates a mix of proficient and profiiific demonstrations.
Franka Kitchen. Franka Kitchen [Gupta et al., 2020 is a simulation benchmark with a 9-DoF Franka arm performing four household tasks (tl. 12, 13, and t4) per trajectory, using 566 human demonstrations across 7 interactive objects. Available through platforms like Gymnasium-Robottccs
Push-T. Push-T (Chi et al. 2023 is a manipulation task adapted from IBC [Florence et al. 2022]. where a circular end-effector pushes a T-shaped block to a target location. Both the block and end-effector start at random positions. Observations consist of either RGB images (img) or set of nine 2D keypoints (kp) outlining the block's shape, along with the position of the end-effector.
Dynamic Push-T (Ours). Many IL benchmarks are nearly saturated: they are deterministic, scripted, and lack diversity, often solvable without closed-loop feedback (Jia al. 2024). To evaluate real-time
https://robotics.c farama. wtt.://wwww...eeeeeeeeaaaaaaaaaa

---

Table 2: Success rates across Robomimic and Franka Kitchen tasks. ph: proficient human demos; mh: mixed-quality demos; t1 to t4 denote task IDs in Franka Kitchen.
Type
Method
Lift
Can
Square
Transport
ToolHang
Franka Kitchen
ph
mh
ph
mh
ph
mh
ph
mh
ph
1
12
13
14
LSTM-GMM [Mandlekar, 2022]
1.00
1.00
0.98
0.82
0.64
0.88
0.44
1.00
0.90
0.74
0.34
Mapping
IBC [Florence, 2022]
0.94
0.39
0.08
0.00
0.03
0.00
0.00
0.00
0.00
0.99
0.87
0.61
0.24
BET (Shafiullah, 2022]
1.00
1.00
1,00
1,00
0.76
0.68
0.38
0.21
0.58
0.99
0,93
0.71
0.44
Flow
FlowPolicy [Zhang. 2025]
0,98
0.95
0.98
0.98
0.86
0.90
0.88
0.82
0.85
0.96
0.86
0.95
0.87
Adalloow [Hu, 2024]
1,00
1,00
1,00
0.96
0.98
0,96
0.92
0.80
0.88
0.99
0.89
0.92
0.83
Diffusion
DP [Chi, 2023]
1,00
1,00
1.00
1,00
0.98
0,98
1.00
0.89
0.95
1,00
1,00
1,00
0.99
DP (ours)
1,00
1.00
1.00
1.00
1.00)
1.00
1.00
0.94
0.98
1.00
1.00
1.00
1.00
Comparison Teiring Convergence DP vs. BP (Push-T)
Comparison Training Convergence: vs AS (Can)
DP
CP
DR
DP (Ours)
5933
0.00
0300
0.00
k000
0.44
2000
Froes
Fpoch
Figure 5: Convergence Plots. Training action MSE over epochs on Push-T and Robomimic Can.
adaptability, we propose Dynamic Push-T, which augments Push-T with moving ball that bounces unpredictably and intermittently interferes with manipulation. Since the ball's trajectory varies in each episode, the agent must adapt online, blocking or avoiding the ball while pursuing the main objective. This develops a dynamic and unscripted challenge that evaluates true adaptability beyond replaying demonstrations. Details of this benchmark are provided in Appendix F
Baselines and Evaluation Metries. We compare DP-AG against IL baselines that include direct mapping (LSTM-GMM [Mandlekar et al., 2022]. IBC (Florence et al., 2022]. BET (Shafiullah et al. ).22),, diffusion-baasee (DP [Chi et al. (023]) ), and flow-matching methods (AdaFlow [Hu et al. 2024]. FlowPolicy (Zhang et al. 2025 Results are from either our reimplementation or the original papers, averaged over 5 traiming seeds and multiple evaluation seeds (50 for Push- Mish-T/Kitchen, 22 for Robomimic, 50 for Dynamic Push-T), We use success rate as the main metric, with target area coverage for Push-T variants. Additional implementation details are in Appendix
Results. Tables I and 2 show that DP-AG consistently outperforms baselines across Robomimic, Franka Kitchen, Push- T, and Dynamic Push-T, achieving near perrect success on static tasks and the highest coverage on dynamic ones. These gains come from enforcing smooth latent and action trajectories. which are especially important in Dynamic Push-T. DP-AG also converges faster than DP 5 with additional plots in Appendix M). Ablation results are provided in Appendix H and computational analysis is detailed in Appendix H.I
6.3 World Evaluation on UR5 Robot Arm
To evaluate the URS robotic arm in three visuomotor manipulation tasks: Painting, Candy Push, and Peg-In-Hole Insertion (Figure( (6). These tasks are designed to evaluate different aspects of perception-action interplay and the policy's 's ability to generalize from partially observable inputs. The details (dataset, training, etc.) of the real-world manipulation experiments are provided Appendix !
performance of our DP-AG, we deploy DP and DP-AG policies onto a
Painting (Planar Precision). The robot is tasked with tracing shart- and prccularshapped paths using paintbrush. This task requires smooth and continuous motion to avoid over-painting or streaking. Our DP-AG improves both path fidelity and smoothness compared to DP.
Candy Push (Object-aware Adaptation). The end-effector pushes small candies into designated goal area. Object positions are randomized across trials. DP-AG's latent evolution allows it to adapt to variations in candy layouts and locations, which results in fewer collisions and smoother trajectories.
Peg-in-Hole (3D) Visual Reasoning). The robot must insert a circular peg into a vertical hole using only RGB inputs from scene and wrist cameras, without explicit depth sensing, requiring the policy

---

Candy Push
Painting
Peg-In-Hole Insertion
Figure 6: wealdworld evaluation on UR5 robot arm across three manipulation tasks.
Table 3: Performance Real-World URS Tasks. Mean and standard deviation are reported.
Task
Method
Success Rate (%)
Smoothness (Avg. Jerk)
IoU (%)
Time to Complete (s)
Painting
DP
0.083 = 0.014
68.9 = 5.2
49.5 = 4.1
DP-AG
0.032 0.009
92.1 + 3.4
18.0 + 3.2
Candy Push
DP
65.0 8.4
0.107 0.016
24.0 - 3.9
DP-AG
90.0 + 5.5
0.039 + 0.011
9.5 = 2.6
Peg-in-Hole
DP
0.0 + 0.0
0.096 0.017
DP-AG
85.0
0.036 0.008
13.0 2.1
to infer 3D geometry from indirect cues. Baseline DP offers no mechanism to adapt when the hole is slightly misaligned. Upon contact, it repeatedly executes the same ineffective motion, failing in all trials. In contrast, DP-AG leverages action feedback: each blocked step triggers a VJP-guided latent update that sharpens sensitivity contact-region the same RGB inputs are reinterpreted to expose geometry cues sufficient for correction, enabling successful insertions under occlusion and unseen perturbations.
(e.g., rim alignment). Over a few refinements,
Evaluation. We evaluate our DP-AG against DP on the three tasks, each repeated over trials. For Painting, report ToU between the painted and target shapes; for Candy Push and Peg-In-Hole, we report success rate. Moreover, we measure trajectory smoothness via average jerk and task completion time. As shown in Table 3 DP-AG consistently outperforms DP across all metrics. Notably, the largest gain occurs in the peg-in-hole task, where DP-AG's dynamic latent updates help infer 3D information from 2D observations, which is an ability DP lacks due to its static features.
7 Conclusion
We introduce DP-AG, a representation learning framework that closes the perception- action loop in diffusion policies. By evolving latent observations through an action-guided SDE, DP-AG transforms diffusion noise gradients into structured perceptual updates via VJPs. A cycle-consistent contrastive loss aligns static and evolving latents, enabling continuous and bidirectional coupling between perception and action throughout diffusion. We derive principled ELBBO and prove that contrastive alignment enforces mutual continuity in latent and action trajectories. Empirically, DP-AG achieves sate-of-the-art performance, especially under partial observability and dynamic conditions, demonstrating that perception- action interplay is significant for effective and adaptive policy learning.
Broader Impacts. DP-AG advances imitation learning by modeling dynamic perception-actee acction interplay, enabling smoother and more context-aware robotic manipulation and reducing the risk of failure or unintended motions. This capability is particularly valuable in crittccli domains such as automation, manufacturing, and assistive robotics, where robust decision-maker is significant for effective human-robot collaboration. However, relying expert demonstrations can introduce biases or suboptimal behaviors into the policy. To mitigate these risks, is important to rigorously validate the quality and diversity of demonstration data minimize unintended actions.
10

---

Acknowledgments and Disclosure of Funding
This work was partly supported by NSERC Discovery, CFI-JELF, NSERC Alliance, Alberta Innovates and PrairiesCan grants.
References
S. Arora and P. Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
M. Bain and C. Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pages 103-129, 1995.
K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. lchter, Jakubezak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch,
L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky. no: A vision-language- action flow model for general robot control. arXiv preprint arXiv:2410.24164 2024.
R. A. Brooks. Intelligence without representation. Artificial intelligence, 47(1-3):139-159 1991.
C. Chen, F. Deng, K. Kawaguchi, C. Gulcehre, and S. Ahn. Simple hierarchical planning with diffusion. International Conference on Learning Representations (ICLR), 2024.
R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pages 1597-1607. PMLR, 2020.
C. Chi, Z. Xu, Feng, F. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273666
2023.
F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 4693-4700. IEFF, 2018.
P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning (CoRL), pages 158-168. PMIR, 2022.
A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving morgzoorzzon tasks via imitation and reinforcement learning. L. Kaelbling, D. Kragic, and K. Sugiura, editors, Conference Robot Learning (CoRL), volume 100 of Proceedings of Machine Learning Research, pages 1025-1037, 30 Oct-01 Nov 2020.
D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning (ICML), pages 2555-2565. PMLR, 2019.
D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse control tasks through world models. Nature, pages 1 1-7, 2025.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and Lerch- ner. beta-VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations (ICLR), 2017.
Ho, Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33:6840-6851, 2020.

---

X. Hu, Q. Liu, X. Liu, and B. Liu. Adaflow: Imitation learning with variance-adaptive flw-based policies. Advances in Neural Information Processing Systems (NeurIPS), 37:138836-133 138858, 2024.
M. T. Jackson, M. T. Matthews, C. Lu, B. Ellis, S. Whiteson, and Foerster. Policy-guided diffusion. arXiv preprint D.Xiv:24044063555 2024.
M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning (ICML), 2022.
X. Jia, D. Blessing, X. Jiang, M. Reuss, A. Donat, R. Lioutikov, and G. Neumann. Towards diverse behaviors: A benchmark for imitation learning with human demonstrations. International Conference on Leurning Representations (ICLR), 2024.
M. J. Kim, K. Pertsch, S. Karamcheti, Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Open An open-source iision-language-action model. In 8th Annual Conference on Robot Learning (CoRL), 2024.
D. P. Kingma, M. Welling, al. Auto-encoding variational bayes. The Second International Conference on Learning Representations (ICLR), 2014.
D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterization trick. Advances in Neural Information Processing Systems (NeurIPS), 28, 2015.
S. Li, R. Krohn, T. Chen, A. Ajay. P. Agrawal, and G. Chalvatzaki, Learning multimodal behaviors from scratch with diffusion policy gradient. Advances in Neural Information Processing Systems (NeurIPS), 37:38456-38479, 2024.
X. Li, T.-K. L. Wong, R. T. Q Chen, and D. K. Duvenaud. Scalable gradients and variational inference for stochastic differential equations. In Second Symposium on Advances in Approximate Bayesian Inference, 2019,
Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
H. Ma, T. Chen, K. Wang, N. Li, and B. Dai. Efficient online reinforcement learning for diffusion policy. In International Conference on Machine Learning (ICML), 2025.
A. Mandlekar, D. Xu, Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL). PMLR, 2022.
A. Q. Nichol and P. Dhariwal, Improved denoising diffusion probabilistic models. In International Conference Machine Learning (ICML), pages 8162-8171. PMLR, 2021.
J. K. O'regan and A. Noë. A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24(5):939-973) 2001.
Osa, J. Pajarinen, G. Neumann, A. Bagnell, P. Abbeel, J. Peters, al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 1(1-2):1-179, 2018
A. Parulekar, L. Collins, K. Shanmugam, A. Mokhtari, and S. Shakkottai. Infonce loss provably learns cluster-preserving representations. In The Thirty Sixth Annual Conference on Learning Theory, pages 1914-1961. PMLR, 2023.
N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning & modes with stone. Advances in Neural Information Processing Systems (NeurIPS), 35:22955-22268. 2022.
Y. Song. J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
12

---

R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances Neural Information Processing Systems (NeurIPS), 12, 1999.
Tan and A. Xie. An extension of the am Society, 46:245-200 251, 2020.
am-gm gm -m-hmmmhmmmm-mmm inequality. Bulletin of the Iranian Mathematical
Wang, W. Bae, J. Chen, Zhang, L. Sigal, and C. W. de Silva. What has been overlooked in contrastive source-free domain adaptation: Leveraging source-informed latent augmentation within neighborhood context. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. Featured Certification.
Z. Wang. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
M. Watter, J. Springenberg. J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in Neural Information Processing Systems (NeurIPS), 28, 2015.
Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg. Daydreamer: World models for physical robot learning. In Conference on Robot Learning (CoRL), pages 2226-2240, PMLR, 2023.
Q Zhang, 7. Liu, H. Fan, G. Liu, B. Zeng, and S. Liu. Flowpolicy: Enabling fast and robust 3d flow-based policy via consistency flow matching for robot manipulation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 14754-14766 14762, 2025.
C. Zhu, R. Yu, S. Feng, B. Burchfiel, P. Shah, and A. Gupta. Unified world models: Coupling video and action diffusion for pretraining large robotic datasets. In Proceedings Roboties: Science and Systems (RSS), 2025.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper discusses the limitations of the proposed work in the Broader Impacts section.
Guidelines:
13

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Yes, the paper provides the full of assumptions and a complete and correct proof for each theoretical result in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper fully discloses all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper. More importantly, the paper provides the sufficient details of the conducted real-world experiments including dataset collections and model implementation, The code is also provided in the supplement materials.
Justification:
14

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, the paper provides open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in sup- plemental material. The code and data are available on our project website: https //j /jingwangg8. github. io/dp-ag ag. wwt.ggittubbii. io/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/ gablic/guides http://uuddos//ooeuuuuiiissss
Poliiioooocyy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /niiss cc/ wwwpcc/ppccc....c hoooll/cgguides//oeeeubbiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
15

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, the paper specifies all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results for all experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes. the paper reports error bars suitably and correctly defined or other appropriate information about the statistical significance for all experiments including the real-world experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
16

---

Answer: [Yes]
Justification: Yes, the paper provides sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e-g-. preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the paper discusses both potential positive societal impacts and negative societal impacts of the work performed in the Broader Impacts section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
17

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
18

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components. LLMs are used solely for language refinement.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, mon-standarr components.
Please refer to LLM policy for what should or should not be described.
19

---

Appendix
A Extension to Flow Matching Models
While demonstrate DP-AG built upon DDPMs (as in DP), the proposed perception action interplay can be extended to alternative generative models that produce smooth action trajectories, such as flow matching models [Lipman al. 2023 Black al. 2024 Zhang al. 2025
Background on Flow Matching. Flow matching formulates action generation as solving an ordinary differential equation (ODE) that deterministically transforms sample from a known base distribution (e.g., Gaussian) to the target data distribution. Given a continuous time variable E [0, 1]. the action a(t) evolves according to:
da(t) vo((((), zt. #,,4),
(19)
dt
where ve denotes learned velocity field conditioned on observation features 24. The goal match the score of the intermediate marginal distribution pr(a) at time t by minimizing a regression loss. Unlike diffusion models, flow matching generates continuous evolution directly without stochastic perturbations.
Action-Guided Latent Evolution with Flow Matching. To extend DP-AG to flow matching models, we leverage the structure of the learned velocity field Up to guide the latent dynamics. To be specific, each time I, define the evolution of the observation feature through a VJP-guided ODE:
dž, dl. - ( (Ovola(t), z4.t)
vo vo(a(t), Z,,),
(20)
Oz.
where (a()((), 2z, i) serves as the action-conditioned driving force, and the VJP propagates this force back to update the observation feature ε, This setup follows our DP-AG formulation for DDPMs but replaces the stochastic increments from noise predictions with deterministic updates guided by flow matching dynamics.
Training Objective. Similar to the DP-AG framework, we introduce a variational posterior go 16(2/20 2e, att) over the evolved latents and formulate ELBO:
LELBO E4 [log))(v((a(();, E,, KL
(21)
where p(ze|ze) is the base distribution, e.g., isotropic Gaussian centered at Moreover, the cycle-consistent contrastive loss can be formulated between the velocity fields vs (a((), z., 1) and vg(a((), to promote the consistent evolution between the action and the latent trajectories.
Discussion. The flow matching extension has several benefits:
Deterministic evolution improves sample efficiency compared to stochastic DDPM training.
The continuous latent trajectories can be directly controlled via va. which simplifies the interpreta- tion.
The perception-action interplay remains: smoother latent evolution produces more coherent actions, and better action flow, in turn, improves the latent dynamics.
Thus, DP-AG extends beyond DDPMs, which provide flexible framework for integrating with generative-model-bsse policy learning under both stochastic and deterministic dynamics.
B Derivation of DP-AG Variational Lower Bound
In this section, we show the full derivation of ELBO using Jensen's inequality:
log p(Ex(2)) 2 E (=. ( ((x+)())1))) KL (go( |2, at) I| ()))))))))
(22)
where qs ( Iee a is the variational posterior approximating the true posterior. We start from the exact marginal:
p(Eulz)) =
dzt.
(23)
20

---

Inserting the variational posterior ( af) via importance sampling:
p(Exl2))= = 1 golet ( 1e, a/)
[((E) ( (2))
džt = Bg
(24)
a ( 2e, o.)
aolEt 2er a
Thus, we have:
plEkl pl ( q (2)) Re, a.)
logp(en(=x) = log Ens
(25)
Since log(-) is concave function, by Jensen's inequality:
log E Egg [X] 2 Egg [log X),
(26)
for any non-negative random variable X. Here, plex 18, (e()) clearly non-negative because all densities are non-negative.
(2) 2,,07)
Thus, applying Jensen's inequality to case gives:
p(( go 2 12: a.))
log p(Exl= )({ pp(+)=)))(+11))))
log Bgg
2 Eugs
(x( 2r, a
-Eng [logp(Ex(<() + log logg(;
log at))
= Egg
[logp(Ex|2]] [+])] + Ega [logp(((1<0] Egg [Log (0 at]]
Eng [logp(Ex| KI. (go(= at) -1)()=
(27)
Thus, we have proven:
logp(ex/z1) 2 E (2t [logp(Ex|=+) KL (us( a |I)())=)))
(28)
C Derivation and Intuition for the KL Divergence Term in Equation 16
To regularize the latent evolution af and preserve semantic consistency with the observation-encodee latent 2g, introduce KL divergence between a variational posterior and conditional prior. This section provides a step-by-step derivation and intuition for the expression in Equation I6
Formulation. The variational posterior is defined Gaussian distribution:
q ( at) (pp(zz at), diag(o at))),
(29)
where both the mean and larvanriance are predicted by linear layers. The conditional prior is set to:
p x.) = .Nz I),
(30)
which enforces that the evolved latent remains close to its observation-drivel anchor 2, unless strongly influenced by the action.
KL Between Diagonal Gaussians. The KL divergence between two diagonal Gaussians N(1g Egg)
and N (jip E,p) is given by:
KL(g))p)
= 2
d Σ 1=1
og +
(Mg.i -Mg.i-ppp 11pp)) log
q.
(31)
p.i
p..
where d is the dimensionality of samples drawn from Guassians. Substituting My = Ho(2e, at).
og = az(2,, at), Ho Z1, and 2p 1, we obtain:
KL(go 11 (1
Σ (0 (uo. 21,1)2 - log o2 2..4)
(32)
where d now the latent dimension, and all quantities are computed element-wise. Intuitions. Each term in the KL expression serves distinct regularization role:
21

---

(us.1 =x,,); latent.
penalizes deviation of the evolved latent mean from the observation-encoded static
a penalizes over-dispersion and encourages confident latent representations.
logor o rewards expressive uncertainty when needed, balancing the previous term.
This KL. divergence term acts as a geometric constraint to preserve alignment with the observation- anchored latent space, while still allowing dynamic evolution based on the action. Combined with our contrastive loss (Equation 9) it stabilizes training by maintaining local continuity and global discriminability.
D Proof of Lemma 1
Following the theoretical work on the InfoNCE loss [Parulekar et al. 2023 Wang et al. 2024], we derive the lower bound for the positive key alignment:
Lemma (Noise Similarity Lower Bound). For unit-normalized vectors ej and E and temperature
> 0, if the InfoNCE loss satisfies Leom I or some small constant o, then for each i E {l,..., B)....,,,, B}. the similarity between corresponding pairs is bounded accordingly:
sim (Ek
2 2 - -a-
(33)
positive pair similarity
Proof. Recall that the InfoNCE loss is defined as:
B
exp (sim (ct, /T)
Ccomt
log
(34)
B
Σ Ejgt exp sim ck & /T)
To simplify the notations, let s/ = sim ( E and sij = sim = for j + i. We define:
exp(s1/TT)
Pi
Ejjt exp(st/7)
(35)
The loss constraint Zcomt I a implies:
1
logn: S = Σ
log 2 Ba.
(36)
B
Taking exponential at both sides, have:
B
I p. 2 exp(-Ba).
(37)
To derive lower bound on sü, need the denominator EjØs exp(si: (tg//) be as small as possible. Since Sij -1 (unit vectors), the denominator is minimized when all siy -1. Thus, we have:
Σ exp(8t//1) /T) 2 (B e)xxx(11//)
(38)
and
exp(sii/T)
Sü + 1 r
Pi <
B 1) exp( (1/7) B exp
(39)
Assume that p: is equal across all i to maximize the product under the constraint by Arithmetic deam-Geemettic Mean (AM-GM) inequality [Tan and Xiel 2020]:
1/B
P
Pi
2 exp((-Ba) exp(-ar.
(40)
22

---

Substituting into Inequality 39 we have:
1
8i +
exp(-a) S S
exp
(41)
B-1
7
Focusing on the inequality:
Sei + 1
exp(-a) S
13 1 exp
(42)
Multiply both sides by B 1:
Syi 8i+tl + 1
(B - 1) eexp(ca) S exp
(43)
Take the natural logarithm of both sides:
In ((B 1) exp((aa) < Sai +
(44)
Simplify the left-hand side:
In(B - 1) + In(exp(-aa)) = -In(B - 1) - a.
(45)
Thus:
In(B 1) - 5 8ü +1
(46)
Multiply through by T:
Tlnn(B - - a) S S/4 +1
(47)
Finally, we have the lower bound for 8;:
8a T - 1) - a) - 1.
(48)
Thus, we have proved:
sim (t), 2 - 1) a) 1.
(49)
E Proof of Theorem 1
Theorem (Continuity Upper Bound). Suppose col-) is L-Lipschitz with respect to 2, and that 2/++1 evolves via the VJP of eg. Then under the contrastive constraint Leoms S a we have:
|| 1 < L Le 2,, k) c0 cāt 2 k)1l3 s 22 In(B 1) Ta).
(50)
Proof. We split the proof into two steps:
Step 1: Lipschitz continuity of ey relates latent updates and noise difference. Taking the vector- Jacobian product (VJP) approximation for the drift of the SDE evolution, the update of 2 at each step satisfies:
|| - 112 VJP 24),
(51)
where, under the Lipschitz continuity assumption, the VJP depends linearly on the local noise gradient structure.
Assuming proper normalization of the step size in the SDE discretization (absorbed into the L.- Lipschitz constant), we have:
|
= 1| S Llleo(at 2z, k) - ca sk k:)l
(52)
Squaring both sides yields:
|| 112 S I [ 2,, k) ca 2 1.)
(53)
Step 2: Contrastive loss bounds the noise difference. From Lemma [ when Loomt I a, the cosine similarity between et and & is bounded below:
sim # 2 (In(B - -1) -a) - 1.
(54)
23

---

Recall that cosine similarity between two unit-norm vectors satisfies:
1 2 = 2(1 - sim (u, u)).
1011-si((ee
(55)
Thus, applying this relation to εk and ck. we obtain:
|| ze, k) ca z A - 2 (1 sim (Ea Rez Fe, co (at; #t, k)))
(56)
By plugging in the lower bound on similarity from Lemma 1 we get:
|loo(āf Ze, k) co = 1)11 < 2 In + T()-
(57)
Thus, combining Step and Step 2, we can conclude that:
|| - 1 S r,2 (|ca(aff Ze. k) co(a a A) s 21,2 - T rln(B (B - 1) ra).
(58)
This completes the proof.
F
Robotic Manipulation Benchmarks
This ablation section details the tasks and corresponding datasets for four robotic manipulation benchmarks for simulation: Robomimic, Franka Kitchen, Push-T, and Dynamic Push-T, T. Each benchmark contributes unique tasks and datasets evaluate various aspects of robotic manipulation, from precision and coordination to adaptability in dynamic environments. Below, we describe each task, its objectives, and the associated datasets, highlighting their roles in our empirical evaluations. Robomimic Benchmark. Robomimic is comprehensive benchmark for robotic manipulation with five tasks: Can, Square, Transport, Tool Hang, and Lift. These tasks are designed to evaluate range of manipulation skills that include pick-and-place, precision assembly, multi-arm coordination, and basic object manipulation. Each task includes expert demonstrations collected via proficient human (PH) teleoperation, with additional mixed human (MH) datasets, resulting in nine datasets total. The datasets are structured in HDF5 format, containing observations, actions, rewards, and other metadata, and are available online hhttps: https://github. com ARISE- Inwtiative/roooniieeiii
Table 4: Summary of Robomimic Tasks and Datasets
Task
PH Demos
MH Demos
Key Skills Tested
Can
200
300
Pick-and-place
Square
200
300
Precision manipulation
Transport
200 (2 operators)
300
Multi-arm coordination
Tool Hang
200
300
Precision grasping, insertion
Lift
200
300
Basic object manipulation
Can. The Can task involves picking up a and placing it in the correct bin, testing pick-and-plas skills. The dataset includes two variants:
Proficient Human (PH): 200 demonstrations collected by single proficient operator using the RoboTTurk platform (https:// stanf ord. edu/).
Mixed Human (MH): 300 demonstrations from six operators of varying proficiency, introducing variability in demonstration quality.
The PH dataset provides high-quality demonstrations, while MH introduces real-world variability,
Square. The Square task, also known as Square Nut Assembly, requires fitting square nut onto square peg (and potentially a round nut onto round peg). The scene includes two colored pegs (square and round) and two nuts on a tabletop, with randomized nut locations at episode start (https://roo /tttps:/rrobosuite ai/docs/aiaeececceeeeeeeea irtoossmodduleg moduless//////cooc.oo www.eennnntntss...mmm htmll)) This task evaluates precision manipulation. The dataset includes two variants:
Proficient Human (PH): 200 demonstrations collected by proficient operator via RoboTurk.
Mixed Human (MH): 300 demonstrations from operators of varying proficiency.

---

These datasets test algorithms on precise manipulation with both expert and varied human inputs.
Transport. The Transport task involves transporting object using robotic arms, which requires multi-arm coordination. is noted for its complexity, with observation spaces including shoulder and wrist views per arm, indicating a dual-arm setup (https: ftttps://robommiic. github. ttt::o//stuu stody/. The dataset includes two variants:
Proficient Human (PH): 200 demonstrations collected by two proficient operators working together (https://A http://wwwwttt.rrf..ee
tensorf wwtp:oo///aatttts/ccaaaeeeae http://aaaaoooooooooioioooo
Mixed Human (MH): 300 demonstrations from varied operators.
The datasets evaluate coordination and robustness in multi-arm cooperation tasks.
Tool Hang. The Tool Hang task requires hanging a tool (c.g., hook), which involves precise grasping and insertion. The dataset includes two variants:
Proficient Human (PH): 200 demonstrations collected by a proficient operator via RoboTurr.
Mixed Human (MH): 300 demonstrations from operators of varying proficiency.
These datasets evaluate on high-precision tasks.
Lift. The Lift task involves lifting an object, simple manipulation task that benefits less from large datasets compared to complex tasks. The dataset includes two variants:
Proficient Human (PH): 200 demonstrations collected by a proficient operator via RoboTurk.
Mixed Human (MH): 300 demonstrations from operators of varying proficiency.
The datasets provide baseline for evaluating basic manipulation skills.
Table 4 provides summary of the Robomimic tasks and their corresponding datasets.
Franka Kitchen Benchmark. Franka Kitchen is a simulation benchmark where 9-DoF Franka arm operates in kitchen environment, performing four household tasks per trajectory. The tasks involve manipulating objects achieve a desired goal configuration, which include:
Open the Microwave: The robot opens the microwave door.
Move the Kettle: The robot repositions the kettle to a target location.
Flip the Light Switch: The robot toggles light switch to turn on a light.
Slide Open the Cabinet Door: The robot slides open a cabinet door.
The benchmark is hosted on platforms like Gymnasium- (nttps://robotccs
farama. org wwtp:////////aaaaaa/./a/aaa kitchen/) and uses datasets from D4RL (https://github.
Farann--Foundaiioo//DRLL
Table 5: Summary of Franka Kitchen Tasks and Datasets
Task
Description
Dataset
Key Skills Tested
Open Microwave Open the microwave door
566 trajectories
Object interaction, door manipulation
Move Kettle
Reposition the kettle
566 trajectories
Object repositioning Switch manipulation Sliding mechanism inter- action
Flip Light Switch
Toggle the light switch
566 trajectories
Slide Cabinet Door
Slide open the cabinet door
566 trajectories
Datasets. The Franka Kitchen dataset consists of 566 trajectories, each completing all four tasks, collected via human teleoperation across seven interactive objects (a microwave, a kettle, an overhead light switch, a sliding cabinet, a hinged cabinet, a top burner, and a bottom burner) (https:/ html). The dataset evaluates the ability
minari.. farama. org/ to sequence and execute multiple subtasks realistic kitchen environment. Table 5 providee summary of the Franka Kitchen tasks and their corresponding datasets.
25

---

Push-T. Push-T is a manipulation task adapted from Implicit Broadcast Communication (IBC), where a circular end-effector pushes a T-shaped block to target location. The block and end-effector start at random positions, which can be defined by different evaluation seeds. Observations are either RGB images (img) or nine 2D keypoints (kp) outlining the 'T block's s shape and the target location, as well as the end-effector's position (https: /ppperss//ppwwww//////aa httpeerwwwihhhooeeeeeee thcode. com wwwaa/kkkbbboooo www.rrooooooopppppeeeeeee manipulation
Dataset: The Push-T dataset includes 62,500 push interactions across 200 evaluation seeds, designed to evaluate the pushing dynamics and perception in a controlled setting.
We extend the Push-T T task by introducing moving orange ball that travels at 100 units per second within a 500-unit square. The ball bounces off both the walls and the end- effector. occasionally disrupting the T-shaped block. The agent must simultaneously block these disturbances while pushing the block to the target, requiring it to maintain stable control under dynamic interference. This setting explicitly evaluates latent continuity and adaptability: abrupt latent shifts can destabilize interactions, whereas smooth latent evolution enables the agent to adapt effectively to changing dynamics.
Dynamic Push-T.
Dataset: Similar to the Push-T dataset, we include 62,500 push interactions across 200 evaluation seeds, specifically designed to evaluate robustness and adaptability in dynamic and unpredictable environments. Table [ provides a summary of the Push-T and Dynamic Push-T T tasks and their corresponding datasets.
Table 6: Summary of Push-T and Dynamic Push-T Tasks and Datasets
Task
Description
Dataset
Key Skills Tested
Push-T
Push T-shaped block to target
62,500 pushes
Pushing dynamics and per- ception
Dynamic Push-T
Push T-shaped block with 62,500 pushes
Adaptability and disturbance handling
moving ball disturbance
G Implementation Details
Our DP-AG builds on the Diffusion Policy (Ch al., 2023), with two key modifications: (1) we add two linear heads for predicting the base drift (6(2)) and log-variane variance log 03 from the static latent features, and (2) we incorporate VJP-guided SDE for latent evolution without introducing extra learnable parameters.
For observation encoding, we use a ResNet-18 [He et al. 2016) backbone for image-based tasks. We adopt the same conditional U-Net architecture from DP [Chi et al. 2023 to predict diffusion noise. During training, we apply random cropping with task-specific sizes as in DP, while a center crop is used at inference. To maintain dynamic consistency, color jitter random flipping is applied. An MLP is used to encode the agent's 's proprioceptive inputs. For key-point-basee tasks, we follow the original DP setup and use fully connected networks.
We also apply action normalization seales each action dimension independently to [-1, 1] to ensure compatibility with the DDPM denoising process, where predictions are clipped within this range at each step. For positional control tasks, actions use 6D rotation representations. Velocity control tasks use 3D axis-angle representations, which follow standard practice.
We train the model using the iDDPM algorithm NNchol and Dhariwal 2021 with 100 diffusion steps. All models are trained for 300 epochs on vision-based based tasks and 200 epochs for key-point- based tasks. For learning rate scheduling, we use cosine annealing schedule with linear warmup of 500 steps. Batch sizes are set to 64 for image- based tasks and 256 for wer-ponnt-basee tasks. Optimization uses Adam W with a learning rate of 1 x 10-4 all experiments. All hyperparameters not directly related to DP-AG extensions (e.g., diffusion step count, augmentation, action normalization) are kept identical DP for controlled and fair comparison. For inference, we maintain the same number of diffusion denoising steps training to avoid introducing distributional shift.
26

---

H Ablation Studies
H.1
Latency and Computational Cost Analysis
We evaluate the computational overhead introduced by our DP-AG compared to the original DP by measuring both training time per epoch, the inference latency, and the total training time to convergence. The computation analysis is conducted on the Push-T benchmark. Experiments are conducted on an Nvidia RTX 4090 GPU with 24GB VRAM. Table 7 summarizes the detailed computational cost comparison between the DP baseline and our DP-AG.
Training Time per Epoch. On the Push-T benchmark, the average training time per epoch increases slightly from 114.2 seconds for DP to 119.5 seconds for DP-AG, corresponding to an overhead of 4.6%. This increase is expected because VJP computation requires additional backward- mode automatic differentiation through the noise predictor. However, the extra cost remains moderate, largely due to the relatively small dimensionality of the latent observation space.
Inference Latency. At inference time, the VJP computations are omitted entirely. Our DP-AG simply operates with the latent features extracted from the observation encoder at inference. As a result, inference latency remains virtually unchanged: 145.3 milliseconds per action sequence generation for DP versus 146.5 milliseconds for our DP-AG, marginal 0.8% difference. This negligible overhead ensures that DP-AG maintains real time responsiveness for sighffreuency robotic control while improving the smoothness and consistency of action generation.
Training Efficiency and Total Time to Convergence. Although our DP-AG introduces a minor increase in per-epoch training time, it significantly accelerates convergence. On the Push-T bench- mark, DP requires approximately 200 epochs to converge, whereas DP-AG achieves comparable performance within only around 100 epochs. This effectively reduces the number of required training epochs and the total training time by nearly 50%. To be specific, DP completes training in about 6.2 hours, while DP-AG completes training in approximately 3.5 hours, which results in net saving of 2.7 hours. Thus, despite the slight per-epoch overhead, our DP-AG achieves faster overall training and improved sample efficiency.
Summary. Therefore, while DP-AG introduces additional VJP computations, modern autodiff frameworks (e.g.. PyTorch execute them efficiently, leading to negligible runtime overhead. As shown in Table 7 DP-AG incurs only minor 4.6% increase per-epoch training time but delivers nearly 50% higher training efficiency. Inference latency remains virtually unchanged: DP-AG sustains real-time control on the UR5 robot while producing smoother trajectories, lower jerk, and faster task completion. These results demonstrate that the added computation does not hinder deployment or responsiveness, and that our perception-action interplay both improves DP and substantially accelerates training, making DP-AG well suited for real-world applications where rapid retraining and real-time control are important.
Table 7: Computational cost comparison between DP and DP-AG on Nvidia RTX 4090 (Push-T).
Model
One Epoch Time (s) Epochs Converge
Total Training Time (h)
Inference Latency (ms)
DP
114.2
~200
662
145.3
DP-AG (Ours)
119.5
~~00
3.55
146.5
H.2
Effect of Cycle-Consistent Contrastive Loss
To evaluate the importance of the cycle-consistent contrastive loss in DP-AG, conduct an ablation study by removing this component while keeping the rest of the architecture unchanged. This allows us to isolate the role of cycle consistency in enforcing mutual smoothness between latent evolution and action refinement during training.
Setup. We compare two variants on the Push-T benchmark:
DP-AG (full model): Includes the cycle-consistent contrastive loss between static and VJP- VJP-guided noise predictions.
DP-AG w/o Contrastive: Removes the contrastive term from the training objective, relying only on the diffusion loss and KL regularization.
27

---

Training Convergence with/witout contrastive on Push-T
DP-AG (full)
5000
DP-AG w/o Contrastive
Train Action MSE
4000
3000
2000
1000
0
50
100
150 Fpoch
200
250
300
Figure 7: Comparison of training convergence on the Push-T benchmark for DP-AG with and without
cycle-consistent contrastive loss.
Both variants are trained under the same settings; and the action MSE is evaluated across training epochs. We repeat the experiments with different initialization seeds for training.
Results. As shown in Figure and Table removing the cycle-consisten loss leads to slower convergence, higher final training action MSE, and reduced target coverage scores. To be specific, the model without contrastive loss requires approximately 3 times more epochs to converge compared to the full DP-AG; the final success rate drops from 93% to 85% the Push-T benchmark; and the training action MSE remains consistently higher throughout training, which indicates less accurate action generation.
Analysis. Without the contrastive alignment between evolving and static latents, VJP-guided per- turbations can drift away from the optimal action refinement trajectory, which degrades both latent continuity and action smoothness. The cycle-consistent loss important in closing the perception- action loop, which can ensure that latent evolution remains tightly coupled with action denoising across diffusion steps.
Table 8: Effect of cycle-consistent contrastive loss on Push-T benchmark.
Model
Epochs Converge
Converged Train Action MSE
Score
DP-AG (full)
100
65.8
0.93
DP-AG w/o Contrastive
300
183.5
0.85
H.3 Likelihood Supervision vs. Contrastive Loss
derive noise regression objective from the variational lower bound, which serves a similar role to our cycle-consisten contrastive loss. To evaluate the necessity of the contrastive loss, we replace it with the likelihood-baic noise regression objective defined in Equation [3]
In Section 5.1
LI B(DD, ksu(1.K) (a) at k)
(59)
Experimental Setup. We compare three variants:
Cycle-Consisteny Contrastive: Uses the proposed cycle-consistent contrastive loss Loont only.
MSE-Only: Replaces Loomt with the noise regression loss LLH.
Combined: Combines both objectives: L Leeut + LIH-
All models are trained with random seeds on the Push-T benchmark. We report the mean and standard deviation of validation accuracy over 100 epochs.
Results. As shown in Figure g the contrastive-only variant not only converges faster but also achieves the best validation accuracy. Although MSE yields smoother convergence, it lacks the structural
28

---

Training Dynamics: Contrastive vs, Regression Noise Supervision
0.95
0.90
Validation Accuracy
0.85
0.60
0.75
0.70
Contrastive Loss
MSE Noise Regression
0.65
Combined (Contrastive + MSE)
20
40
60
80
100
Epoch
Figure 8: Validation accuracy during training with contrastive loss vs. MSE noise regression on the Push-T benchmark. Contrastive loss achieves faster convergence and better performance but with slightly more variance during training.
benefits of noise alignment that contrastive learning offers. The combined objective does not improve performance over the contrastive-only variant and sometimes results in unstable training, likely due to conflicting optimization signals between absolute and relative supervision. These results validate that the contrastive alignment provides stronger inductive bias for learning dynamic consistency between static and VJP- VJP-guided latents, while the likelihood term, though informative, introduces redundancy. thus omit the likelihood objective from our final model.
H.4 Effect of VJP Strength
In this section, we study how the strength of the VJP-guided perturbations affects the performance of our DP-AG. Recall that the VJP acts as a stochastic "force" that shapes the evolution of latent observation features based on the diffusion process that refines the action generation. While moderate VJP guidance helps structure latent trajectories coherently, overly weak strong guidance may destabilize training or restrict flexibility.
Setup. We introduce scaling factor applied to the VJP term during latent updates in Equation g
dæ, = 7 VJP( =.) di + (z)) dW.
(60)
We evaluate the following settings: ? E (0.0, 0.5, 1.0, 2.0, 5.0), where 1.0 is our default setting. We repeat the experiments with 5 different initialization seeds for training.
Results. The results are presented Figure 9 and Table 9 We observe that:
Without VJP guidance, latent features remain static during diffusion, effectively reducing our DP-AG to standard DP.
Moderate VJP strength (? 0.5, 1.0) achieves the best results, which tradeoff balance between encouraging latent evolution and maintaining stability.
High VJP strength (y 2.0, 5.0) leads to unstable latent evolution, where the updated trajectories deviate excessively from the static latent features, ultimately degrading policy performance.
H.S Effect of KL Loss Hyperparameter
In this section, we investigate the effect of varying KL divergence coefficient Åςι. on the per- formance of our DP-AG using the Push-T benchmark. While 3-VAE [Higgins et al. 2017 tunes KI. regularization to promote disentanglement,
our case, the KI. term stabilizes the action-guided latent evolution without affecting its ability to adaptively track action refinements.
29

---

Effect of VJP Strength on Push-T Target Coverage Score
0.94
Target Coverage Score
Target Coverage Score
0.92
0.88
0.86
0.84
0
2
3
VJP Strength Y
Figure 9: Effect of the VJP strength y on the Push-T benchmark.
Table 9: Effect VJP strength y on Push-T benchmark.
Final Train Action MSE
Score
Latent Behavior
0.0
75.3
0.87
static smooth
0.5
58.3
0.93
1.0
65.8
0.93
smooth
2,0
71.2
0,90
over-reactive
5,0
241.5
0.85
unstable
Setup. We train DP-AG on the Push-T T benchmark with the following λKI. values: (0, 0.5, 1.0, 2.0, 4.0, 5.0, 10.0). All other training configurations remain unchanged across experiments. We repeat the experiments with 5 different initialization seeds for training.
Results. The results presented in Figure 10 and summarized in Table 10 following trends:
We observe the
Without KI. regularization (ΛKL. 0), latent evolution becomes less stable, causing updates to drift excessively from the static latent and resulting in lower target coverage scores.
Effect of KL Loss Weight Push-T Score
0.94
Target Coverage Score
0.92
Target Coverage Score
0.90
0.88
0.86
0.84
0.82
0.80
10
KL Weight AKL
Figure 10: Effect of KL loss weighting ÁRI. on the target coverage score for the Push-T benchmark.
30

---

Table 10: Effect of KL loss hyperparameter AKL on Push-T benchmark.
AKL
Converged Train Action MSE
Score
Latent Stability
0
123.1
0.89
slightl unstable
0.5
86,6
0.91
stable
1.0
65.8
0.93
stable
2.0
91.3
0.91
slightly over-constrained
4.0
78.3
0.92
over-constrained
5.0
168.8
0.88
over-constrained
10.0
324.5
0.82
severely over-constrained
Effect of Temperature on Contrastive Loss (Push-T)
0.94
0.92
Target Coverage Score
0.90
0.88
0.86
0.84
0.82
Target Coverage Score
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Temperature
Figure 11: Effect of temperature in the cycle-consistent contrastive loss on Push-T benchmark performance.
Small KL values (AKL. = 0.5, 1.0) achieve the best trade-off between flexibility and stability, which leads to the best performance.
Moderate to large KL values (ARı. = 2.0, 4.0, 5.0) begin to over-regularize the latent space, which limits its ability adapt action refinements.
Strong KI. regularization (Aęı. 10.0) severely restricts latent evolution, which causes underfitting and significantly lower target coverage score.
Analysis. These results suggest that the KL regularization our DP-AG should not be viewed through the lens of promoting disentanglement, in 8-VAE. Instead, it acts to anchor the VJP-guided latent dynamics toward stable yet adaptive evolution.
H.6 Efffet of Contrastive Loss Temperature Parameter
In this section, investigate the influence of the temperature parameter in the cycle-consistent contrastive loss on the performance of our DP-AG. As in self-supervised learning settings (Chen) al. 22200. the temperature parameter controls the sharpness of similarity scores between query and its positive pairs in the contrastive learning. A low enforces highly sharp alignment, while high leads smoother matching across samples.
In our DP-AG. unlike its usage for self-supervised learning, our contrastive loss is designed to enforce cycle consistency between perception and action noise predictions during latent evolution. To evaluate the impact of the temperature parameter on this novel priceptiio-actionn interplay, we vary across a broad range: (0.01, 0.1, 0.2, 0.5, 0.8, 1.0, 1.5). We repeat the experiments with different initialization seeds for training.
Results. Figure 11 presents the results, with summary provided in Table
II
We observe:
31

---

Very low temperature (r = 0.01, 0.1) could overly sharp alignment that causes unstable training and brittle latent evolution.
Moderate temperatures (r 0.2, 0.5) improve stability compared to 0.01, but still do fully leverage latent adaptability.
High temperature (r 0.8) achieves the best performance, which yields the highest success rates and smoothest latent evolution.
Very high temperature (T = 1.5) could weaken cycle consistency, which leads to performance degradation.
Table 11: Effect of temperature r on cycle-consistent contrastive loss (Push-T benchmark).
Converged Train Action MSE
Score
Stability
0.01
433.8
0,82
brittle
0.1
243.4
0.85
unstable
0.2
186.9
0,90
improved
0,5
103.1
0.92
stable
0.8
65.8
0.93
best stability
1.0
96.3
0.92
slightly diffuse
105.5
0.89
diffuse
Analysis. In DP-AG, higher temperature (e.g., 0.8) provides better flexibilityy for aligning noise predictions during latent evolution, allowing dynamic perception refinement while maintaining cycle consistency. Extremely low values over-constrain the model, while excessively high values (T > 1.0) can weaken the noise prediction alignment.
H.7
Comparison with Input Perturbation Smoothness
To evaluate whether the stability achieved by DP-AG could be reproduced by alternative smoothness regularization, we compared against baselines that enforce consistency under input perturbations. For each observation, MSE or cosine similarity between their predicted action noise scores. Table 12 reports results on the Push-T benchmark. Action smoothness is quantified normalized inverse jerk (higher is smoother).
generated a perturbed version by adding Gaussian noise, then minimized
Results. DP-AG clearly outperforms perturbation- based baselines, achieving higher success rates and smoother action sequences. This demonstrates that contrastive regularization not only enforces smoothness but also preserves semantic alignment, leading to superior policy performance.
Table 12: Comparison of DP-AG with input perturbation smoothness baselines on Push-T.
Method
SR (img)
SR (kp)
Smoothness (img
Smoothness (kp t
Perturbation MSE
0.85
0.92
0.83
0.87
Perturbation Cosine
0.88
0.95
0.86
0.92
DP-AG (Contrastive Loss)
0.93
0.99
0.91
0.95
I More Comparisons of Training Convergence
In this section, we compare the training convergence behavior between the baseline DP and our DP-AG in Robomimic Lift and Square tasks. Training convergence is evaluated by measuring the MSE between predicted and ground-truth actions over training epochs.
Setup. Both the baseline DP and our DP-AG models are trained under the same settings, which use the same dataset splits. optimizer configurations, and diffusion schedules, etc. Training is conducted on an Nvidia RTX 4090 GPU with a batch size of 64, and convergence is measured by tracking the action MSE 300 epochs for each method.
Results. The convergence curve for the Rohomimic Square task shown Figure 12 and the curve These results demonstrate that the action-guide
for the Robomimic Lift task is shown in Figure B
32

---

Comparison of Training Convergence: DP vs. DP-AG (Square)
0.07
DP
DP-AG (Ours)
0.06
Train Action MSE
0.05
0.04
0.03
0.02
0.01
0.00
50
100
150
200
250
300
350
400
Epoch
Figure 12: Training action MSE over epochs on Robomimic Square.
Comparison of Training Convergence: DP DP AG (Lift)
DP
DP-AG (Ours)
0.5
Irain Action MSE
0.4
0.2
0.1
25
50
75
100
125
150
175
200
Epoch
Figure 13: Training action MSE over epochs on Robomimic Lift.
perception updates in our DP-AG not only improve the policy performance but also enhance the training dynamics. By dynamically refining latent features through VJP-guided evolution, DP-AG accelerates the denoising process, which leads to more sample-efficient and stable training compared to the static latent used in the baseline DP.
Real-Worte Manipulation Experiments on UR5 Robotic Arm
We conduct real-world manipulation experiments using the Universal URS robotic arm (https //www, universal-row -robotss...o........oooooo.a.s.s com/prodduts//rrbe/1
6-degree-of-freeddmm platform widely adopted in both industrial and research contexts for its versatility and reliability. In this section, compare the performance our proposed DP-AG against the baseline Diffusion Policy (DP) across three visuomotor tasks: Painting, Candy Push, and Reg-in-Hole Insertion. These tasks specifically designed to evaluate the perception- aaciiionnnaiionnnn interplay, testing each policy's 's ability to generalize from limited sensory inputs (e.g., RGB images) precise motor control commands. All evaluations conducted in a controlled real-world setting, allowing us to evaluate each method's s effectiveness, robustness, and potential for the real-world deployment in practical robotic applications.
Dataset Collection. We prepared task-specific datasets for Painting, Candy Push, and Peg-in-Hole Insertion support real-world experiments using the UR5 robotic arm. Each dataset was collected
33

---

via kinesthetic teaching using a teach pendant. Human operators manually guided the robot through the desired trajectories, while synchronized visual and proprioceptive data were recorded in real time,
RGB images were captured at 30 frames per second (FPS) using two calibrated cameras: one fixed in front of the workspace and another mounted on the robot wrist to provide egocentric views. Simultaneously, robot joint states and Cartesian end-effector poses were recorded at 100 Hz via the URS's internal API. To ensure alignment between visual and action modalities, timestamps from both the camera streams and robot sensors were synchronized using a shared system clock. Each trajectory thus consisted of time-aligned sequences of images and robot poses.
For the Painting task, we collected 100 trajectories by having the operator trace either heart-shaped or circular paths horizontal plane. Ground- truth 2D target contours were predefined, and the demonstrations were guided to closely follow these shapes. The Candy Push task involved 80 trials where small candy objects were randomly placed within 20cm x 20cm area. The human operator manually guided the robot to push each candy toward a designated target area marked on the table surface. Success or failure labels were assigned based on whether all candies reached the target area within a tolerance threshold. In the Peg-in-Hole Insertion task, 50 demonstrations were gathered by inserting pegs into variable hole positions, with occlusion and visual ambiguity intentionally introduced. Success was defined as full insertion without slippage or bounce-back. All demonstrations were segmented and temporally synchronized with the image streams using trajectory timestamps. Each dataset was then split into 80% of the trajectories for training and 20% for evaluation.
Training and Implmentation Details. The training procedure and implementation closely follow those used in the simulation benchmark experiments. We build our training pipeline on top of the DP framework, incorporating two core changes in our DP-AG: (1) two separate linear projection heads are added to map the static latent features 2; to the drift term Ho (2,) and log-variance log o (ze), and (2) latent evolution is guided using a Vector- Jacobian Product ( [VJ))-rriven stochastic differential equation (SDE), implemented without introducing new trainable parameters.
We extract visual features from RGB frames using a pre-trained ResNet-152 encoder. Proprioceptive inputs, including joint angles and velocities, are processed through a two-layer MI.. For diffusion noise prediction, we use the same conditional U-Net architecture as in the simulation experiments, where white noise serves as input and observation features act as conditioning signals. During training, we apply random cropping with sizes tuned per task; at evaluation time, we apply center cropping for consistency. No color jitter or random horizontal flipping is used, as such augmentations may disrupt the temporal consistency critical for dynamics modeling.
Actions are normalized to the [-1, 1] range per dimension to align with the denoising diffusion process, and outputs are clipped to stay within bounds. Peg-in- tasks use 6D continuous rotation representations to avoid gimbal lock, while Candy Push and Painting use 3D axis- annll vectors for velocity control. Training follows the iDDPM objective with 100 diffusion steps. Each policy is trained for 200 epochs with a batch size of 64, using Adam optimization and a cosine learning rate schedule with a 500-step warmup and a base learning rate of 1 x 10
Evaluation Metrics. We evaluated each policy using 20% held-out trajectories per task, with each trial repeated 20 times. Four task-specific metries were used to evaluate performance: success rate, smoothness (average jerk), IoU (for Painting), and time to complete.
Success Rate (%): Measures how often the robot completes the task correctly among all trials. In Candy Push, a trial is successful if all candies are pushed into the target zone. For Peg-in-Holds success means the peg is fully inserted. We report mean and standard deviation over 20 trials to demonstrate consistency under variations like random object positions or occlusions.
Smoothness (Average Jerk): Captures the quality of motion based on jerk (the rate of change of acceleration). It is derived from the end-effector's 's position data (recorded at 100 Hz) using finite difference approximations. For a position sequence {21, Fn at uniform time intervals At. average jerk is computed as: Velocity (first derivative):
Ti+1 T.
U/
(61)
At
Acceleration (second derivative):
U/+1 1 Vi
2x + Zt
a:
(62)
At
At²
34

---

Jerk (third derivative):
a++1 at
2443 32,42 + 3r 24
(63)
At
Ar3
Average jerk across the trajectory:
Average Jerk
1 71
Σ
Iilll
(64)
1=1
Lower jerk values indicate smoother movements, which are especially important tasks like Painting and insertion.
Intersection over Union (IoU in %): Used only in Painting, IoU evaluates how well the painted shape matches the ground-truth contour (i.e., heart, circle), which is calculated as:
loU Intersection Area
(6.5)
Union Area
A higher IoU means better adherence to the intended shape.
Time to Complete (s): Records how long it takes to finish the task, which evaluates the efficiency alongside accuracy and smoothness. Faster times, particularly in Candy Push, suggest better adaptation to object dynamics.
These tasks evaluate perception-actio action interplay and generalization from partially observable inputs. Painting demands smooth motion, where jerk and IoU are important. Candy Push tests adaptation to varying layouts, with success rate and time as the key indicators. Peg-in-Hole requires 3D reasoning from 2D data, where success rate and jerk highlight stability. The dataset split ensures robust training and evaluation, consistent with the standard practice. In future work, we plan to explore sim-to-real transfer for our DP-AG, enabling real-world deployment with minimal or additional data collection or model retraining.
K
Visualizations of Action-Guided Latent Evolution
To clarify the semantic meaning of the latent drift in DP-AG and why it matters for action predic- tion, we visualize how observation latents evolve under action guidance on the Push-T benchmark. The key question is: "If the end-effector must move toward its desired future state, how should the latens representation of the current scene be adjusted?" In DP-AG, the answer is given by the VJP, which nudges the latent toward features that most predictive of the next action. This latent drift is not arbitrary. It is shaped by the policy's uncertainty: when the action head is confident, updates small; when the action is ambiguous, the latent pulled toward features that disambiguate the correct motion. Thus, drift expected to improve prediction by refining perception precisely where it is action-relevant.
To make these refinements interpretable, we decode the evolving latents with a lightweight VAE, This choice directly addresses the concern of whether latent drift has semantic meaning: decoded frames expose what the policy is implicitly "re-seeing" as actions unfold. If latent evolution were meaningless noise, decoded frames would be incoherent; instead, we observe structured changes that align with the intended push trajectory.
K.1 Decoded Reconstructions and Policy Attention.
Implementation. Let zo be the static latent from the encoder. At each VJP step kt, update zq with an action-aligned perturbation normalized to fixed step size. A lightweight VAE, trained only on static latents and then frozen, decodes each z to an image ž5. Gradients are never propagated back through this decoder. To reveal the policy's 's evidence, we also compute Grad-CAM on the (æ, y) end-effector head and overlay it on the decoded frames.
Visualization. Figure 14 shows that the decoded VJP-evolved latents yield localized and coherent changes: the end-effector (blue gripper) shifts step by step toward its actual future position, while the background remains stable. Grad-CAM overlays (Figure 15) confirm that these same regions receive the strongest policy attention. Notably, the static latent (k 0) contains future cue,
35

---

Frame
Frame 2
Frame
Frame
Frame 5
Frame 6
Frame
Frame 8
Static ground truth
VJP
Step
Recon.
VJP
Step
A
Recon.
VJP Step Recon.
Figure 14: Decoded VJe-leellee observation latents. Columns independent episodes. The top row (k=0) decodes the static latent. Rows k=l,..., 5 decode the latent after each VJP step.
Static
VJP step
VJP step
VJP step
VJP step
VJP step
Figure 15: Policy evidence on decoded frames. Grad-CAM from the end-effector (x., y) head the policy is overlaid on the same reconstructions Figure 14
36

---

Figure 16: Grad-CAM (control view). Spatial evidence of the policy without reconstructions.
whereas the evolved latents (k= 1) highlight the precise spatial displacements the policy will execute. This demonstrates that the action- sirion- latent drift has both semantic meaning (visible decoded images) and functional value (aligned with the policy's attention).
Together, the two figures demonstrate that VJP-induced latent drift produces interpretable image- space refinements and that these refinements are spatially aligned with the policy's 's own predictions. This coupling connects perceptual updates directly to the end-effector objective, demonstrating how action-guider latent evolution sharpens the visual grounding of control.
K.2 Attention-Only Policy View.
As control, we project Grad-CAM CAM directly onto the raw input frames without decoding (Figure These heatmaps still trace the end-effector's 's trajectory, confirming that the decoder does not hallucinate the effect. Instead, the VAE reconstructions provide semantic lens: they make explicit how the evolving latent imagines the scene differently after each update.
16].
K.3 t-SNE of Action-Guided Latent Evolution.
While decoded frames and Grad-CAM demonstrate where VJP modifies the observation in pixel space, they do not reveal how these updates are organized in latent space. t-SNE embeddings provide complementary view by projecting both static and VJP-evolved latents into dimensions, allowing us to inspect whether action-guided updates follow structured and task-aligned trajectories rather than random drift.
The t-SNE embeddings in Figure 17 show that VJP does not cause latents to seatter randomly. Instead, each static latent (k=0,, bluc) serves as anchor from which the evolved latents (k=1,... 5) trace short, smooth, and consistently oriented trajectories. Across episodes, these trajectories align form clusters that correspond to motion modes, such push direction, indicating that VJP structures the latent space around action-relevant dynamics rather than noise. Importantly, the cycle-consistent contrastive loss further reinforces this structure: it pulls each evolved latent back toward its static anchor while pushing it away from unrelated samples, preserving semantic grounding and preventing arbitrary drift. As result, the trajectories remain compact and discriminative, ensuring that latent evolution not only tracks action dynamics but also maintains clear and auskveeevant separation.
Visualization.
In summary, across decoded frames, attention overlays, and t-SNE latent embeddings, coherent picture emerges:
VJP-driven drift produces semantically interpretable refinements (decoded end-effector positional shifts).
37

---

VJP-guided latent evolution (5 steps)
Static latent (DP)
DP-AG step DP-AG step
DP-AG step DP-AG step 5
DP-AG step
40
L-SNE (perpl=6) dim
20
-20
400
60
20
40
L-SNE (perpl=6) dim-1
Figure 17: t-SNE of static vs. evolved latents. Each static latent (blue) is paired with its VJP trajectory over five steps (orange > green - red + purple > brown).
2. These refinements highlight exactly the regions that the policy deems most informative for predicting the next action.
In latent space, drift follows smooth and task-aligned trajectories rather than arbitrary noise.
Together, these results demonstrate that action-guided latent evolution is not only mathematically principled but also semantically and functionally meaningful.
L Extension to Online Mode
Although our main experiments conducted in offline imitation learning, DP-AG is inherently well-suited for online training. this section, we present a streaming variant of DP-AG that remains fully consistent with the original formulation. Its key mechanism, action-guided latent evolution via VJP, naturally supports the incremental refinement of perceptual features as new action-observation pairs arrive. Below, we describe the specific modifications that enable DP-AG to function effectively the online regime:
Online latent evolution. At each diffusion step A: {l,..., K} between and [+1, the latent observation updated using the same VJP-guided rule as the main model:
Dea (eollt.20 21, Oz,
ck
(o(2)) + (00(=
(a 2,, k),
(66)
where " controls the step size of action action-guided latent refinement. Noise predictions conditioned on static and evolved latents are
Ek c0 k),
Es co(af zt k),
38

---

and are aligned through the cycle-consistent InfoNCE loss
1
exp exp(sim(et,) k//T))
Coont
log
(67)
iml Ejft exp expp(sim(s), 3/T))
The overall loss remains unchanged:
LDP-AG
LDP
Acont Lcont + XKL LKL.
(68)
Streaming update rule. Unlike the offline variant, which optimizes {0, Φ over a fixed dataset, the online version continuously updates the model
observations arrive. At each environmental time I, DP-AG both executes an action and uses the resulting transition as fresh training data:
1. Sample latent: Draw ~ (0(2))o)) from the variational posterior of the current observation.
Within-step diffusion (for k = .). K):
(a) Compute the noise prediction & eo (á Zt; k).
(b) Update the latent via Eq. (66) to obtain z/.
(c) Compute & and accumulate the contrastive loss Leont-
(d) Accumulate the diffusion loss LDP.
3. Action execution: After K denoising steps, decode and execute the action dr in the environment.
4. Immediate update: Use the freshly collected pair (0,,4), optionally together with a short replay buffer containing the last transitions, to perform lightweight gradient update of 0 and Φ on CDP-AG using online RMSProp
Ma et al. 2025
This schedule makes AP-AG online: model parameters updated continually from streaming interaction data, rather than only once on a static offline dataset. As result, the policy can adapt on the fly to distributional shifts novel dynamics while acting.
Experimental setup. We evaluated both the offline and online variants of DP-AG on Push-T (both image- and keypoint-based settings) and Dynamic Push-T benchmarks. Performance was measured as the average success rate over 50 online evaluation episodes. Unless otherwise noted, hyperparameters matched those of the offline configuration.
Table 13: Online streaming results of DP-AG compared to baselines (success rates, mean
Method
Push-1 (img)
Push-T (kp)
Dynamic Push-1 (img)
Diffusion Policy (DP)
0.87
0,95
0.65+00.85 0.80+0.53
DP-AG (offline)
0.93+0.00
0.9910.01
DP-AG (online)
0.90+0.03
0.96+0.02
0.76+10.89
Results. As shown in Table 13 the streaming variant of DP-AG yields slightly lower scores than its offline counterpart, which is expected given the limited replay and noisier updates inherent to online training. Nevertheless, it consistently surpasses the standard DP baseline by clear margin, demonstrating that DP-AG retains effective even under the more challenging online setting with continual real-time updates.
In summary, the online variant DP-AG remains fully consistent with the offline model. It employs the same VJP definition and latent update rule, with the time discretization constant absorbed into 7. Cycle-consistent contrastive alignment is still applied every diffusion step, ensuring the perception perception-action action loop is preserved within each [t, ++1] horizon. Moreover, both the KL regularizer on ds and the overall training loss LDP-AG remain unchanged; only the optimization schedule differs, shifting from offline minibatch training to lightweight streaming updates.
M Multimodal Decision-Making in DP-AG
DPs are inherently capable of modeling multimodal action distributions, making them well-suited for capturing diverse strategies in complex tasks. By extending DPs with action-guided latent evolution, DP-AG not only preserves this multimodality but amplifies DP-AG enhances the representation of multiple effective strategies through two complementary components:
39

---

oction-conditione latent evolution. DP-AG's VJP-guided latent updates let the model refine features in response to action feedback, so each sampled trajectory adapts its latent state uniquely. This supports parallel exploration of multiple strategies by leveraging stochasticity from both the diffusion policy and latent SDE.
Cycle-consistent contrastive alignment. The contrastive loss keeps features for the same action close and pushes different actions apart, organizing the latent-action space to separate and preserve multiple plausible strategies for each task.
Experimental setup. We evaluated multimodal capability on the Franka Kitchen benchmark, which naturally admits multiple strategies for tasks such as opening drawer or flipping a switch. 40 successful trajectories per task were collected and encoded as key end-effector waypoints. Distances between trajectories were computed via Dynamic Time Warping (DTW), followed by unsupervised clustering to identify distinct modes. This procedure follows the existing work on multimodal diffu- sion policies [Li et al. 2024), which demonstrated that clustering trajectories provides principled way to quantify behavioral diversity. We then measured (a) the number of discovered modes, (b) inter- cluster distance (strategy distinctiveness), (c) intra-cluster variance (consistency within a strategy), and (d) success rates across all clusters.
Table 14:
Multiple strategy discovery and diversity on Franka Kitchen. Higher #Modes (1) and inter- cluster distance (f), and lower intra-cluster variance (4), indicate better multimodal representation.
Method
# Modes SR (t1) SR (12)
SR (13) SR (14) Inter-Cluster Dist. Intra-Cluster Var.
FlowPolicy
1.1
0.96
0.86
0.95
0.87
2.0
1.2
DP (Baseline)
2.8
1.00
1.00
1.00
0.99
7.1
1.2
DP-AG (Ours)
3.2
1.00
1.00
1.00
1.00
9.5
1.3
Table 15: Cluster-specific analysis of strategies discovered by DP-AG Franka Kitchen.
Mode ID
Coverage (%)
SR (t1)
SR (12)
SR (13)
SR (t4)
Strategy Description
41
1.00
1.00
1.00
1.00
Left-handed drawer pull
32
1.00
1.00
1.00
1.00
Right-handed drawer pull
3
27
1.00
1.00
1.00
1.00
Two-step approach / mixed arm
Table 16: Robustness to distribution shifts in Franka Kitchen (task t4).
Method
Modes
SR (t4)
Switch Rate (%)
Comment
FlowPolicy
1.1
0.64
12
Rarely adapts, prone to failure Sometimes adapts, less reliable
DP (Baseline)
2.2
0.81
35
DP-AG (Ours)
2.9
0.95
58
Switches to alternatives if blocked
Results. Table 14 presents the overall diversity analysis, which demonstrates that our DP-AG uncovers and maintains on average more than three distinct strategies per task, significantly more than FlowPolicy (1.1) and slightly more than baseline DP (2.8). These strategies are not minor variants: the inter-cluster distance is significantly higher, confirming that the modes correspond to distinct behaviors. At the same time, intra-cluster variance remains low, ensuring compact and consistent execution. Table 13 provides cluster-specific breakdown, highlighting interpretable strategies such as left-handed drawer pulls, right-handed pulls, and two-step mixed-ami -ame approaches, each achieving near eeerfctfe success. Moreover, Table 16 evaluates robustness under distribution shifts. When trajectories were blocked, DP-AG preserved an average of 2.9 modes with a 58% switch rate alternative strategies, while DP and FlowPolicy degraded significantly.
These findings demonstrate that our DP-AG not only retains the multimodal decision-making ability of DPs but actively encourages the emergence of more rich, diverse, and interpretable strategies.
40

---

Its robustness to distribution shifts highlights its potential for real- robotic applications where flexibility and adaptability are important.
N More Experiments on Trajectory Planning
To situate our DP-AG within the broader family of trajectory generation methods, we additionally compare it against Diffuser [Janner et al. [202] and Hierarchical Diffuser [Chen et al. [024]]
Following the experimental protocol of [Cen et al. 2024), we adopt two benchmark familis: (1) Maze2D and Multi2D tasks from D4RL, which evaluate long-horizon navigation in continuous control environments, and (ii) the sulti-stage FrankaKitchen benchmark, which evaluates robotic manipulation requiring both skill composition and generalization to unseen states. In Maze2D and Multi2D, an agent must reach goals varied layouts (U-Maze, Medium, Large), with performance measured by average return. In FrankaKitcher policies are rolled out from diverse initial states and evaluated by the number of sub-tasks completed within long-horizon episodes.
Experimental setup. We match the evaluation protocol of the baselines, adopting the same trajectory segmentation and planning horizons (e.g.. .. K 15 for Maze2D and Multi2D, K for Gym- MuJoCo, and horizon lengths H 120 for U-Maze and H = 255 for Maze2D Medium). All models are evaluated over 100 random seeds, reporting the average return or goal completion rate under identical training, validation, and testing splits.
Table 17:
Trajectory planning on Maze2D and Multi2D (long-horizon), Performance measured by average return (higher is better).
Environment
Diffuser
Hierarchical Diffuser
DP-AG (Ours)
Maze2D U-Maze
113,9 + 3.1
128.4 + 3.6
142.7 + 2.9
Maze2D Medium
121.5 2.7
135.6 + 3.0
150.2 2.6
Maze2D Large
123.0 + 6,4
155.8 + 2.5
172.1 + 2.2
Multi2D U-Maze
128.9 + 1.8
144.1 = 1.2
168.2 + 1.2
Multi2D Medium
127.2 + 3.4
140.2 + 1.6
153.7 + 1.3
Multi2D Large
132.1
+ 5.8
165.5
+ 0.6
182.2 0.5
Average
124.4
145.0
161.5
Table 18: Multi-stage robotic manipulation on FrankaKitchen. Performance measured by average number of completed sub-tasks.
Task
Diffuser
Hicrarchical Diffuser
DP-AG (Ours)
Partial Kitchen
56.2 + 5.4 50.0 + 8.8
73.3 + 1.4
82.2 = 1.3
Mixed Kitchen
71.7 + 2.7
78.5 2.1
Average
53.1
72.5
80.4
Results. Across all benchmarks, DP-AG achieves the highest returns, with average improvements of +16.5 over the Hierarchical Diffuser and +37.1 over the Diffuser in Maze2D and Multi2D (Table These performance gains indicate that our DP-AG generates longer and higher-quality trajectories, es- pecially in large and complex environments. In the multi-stage FrankaKitchen benchmark (Table DP-AG completes on average 80.4 sub-tasks, significantly outperforming both Hierarchical Diffuser (72.5) and Diffuser (53.1), which highlights DP-AG's effectiveness in composing diverse skills and generalizing them to extended horizons. Together, these results indicate that the action action-guided perception action loop in DP-AG provides clear advantages for both navigation and manipulation tasks, beyond what trajectory diffusion alone can achieve.
Extending DP-AG with World Models
World models such as DreamerV3 [afner et al. 2025) and Unified World Models (UWM) (Zhu et al. 2025 learn to "imagine" latent trajectories by predicting future states and rewards. While
41

---

fixed observation is these predictions enable long- morgzooizzon planning, the latent trajectories are typically static within each rollout: once generated, they do not adapt to evolving action feedback. In contrast, DP-AG continually updates latent features within each action diffusion step, guided by action-covt cetion-conditioned VJPs. By combining with world models, our DP-AG gains long-horizon foresight while retaining its intra-step refinement.
Hybrid Architecture. We integrate DP-AG into UWM by adding an intra-step feedback mechanism on top of UWM's diffusion transformer backbone:
UWM Backbone. UWM jointly learns to predict actions and future visual observations by treating them as parallel diffusion processes. This unified design allows the same model to serve flexibly as a policy, forward dynamics predictor, an inverse dynamics model, or video generator, simply by adjusting which modalities are denoised. The architecture uses ResNet- based encoders for observations, spatiotemporal patching for latent images, shallow MLPs for actions, and transformer layers conditioned through adaptive normalization with additional register tokens to promote information exchange across modalities.
DP-AG Intra-step Feedback. During action generation, DP-AG introduces a feedback signal into UWM's latent representations. At each denoising step, the current action prediction influences how features are refined, ensuring that perception and action remain aligned as the sequence unfolds. This intra-step refinement captures the core principle of DP-AG, where reinterpreted in light of evolving actions.
Bounded Consistency. To prevent excessive feature drift, apply lightweight penalty that encourages smooth updates across steps, complemented by DP-AG's cycle-consistent contrastive anchor. This balance preserves stability while still enabling aition-driven adaptability.
Implementation Details.
Observation and Conditioning. Visual inputs are processed by frozen VAEs into compact latent grids, which are patchified and combined with action tokens. Diffusson time steps for both action and observation branches are encoded as embeddings and injected into each transformer block.
Training Objective. UWM is trained with coupled denoising loss over action and future-image predictions. For action-free videos, the action branch is masked out, allowing the same objective to learn from both robot trajectories and pure video data.
DP-AG Integration. The action-guided feedhack mechanism is applied during both imagination and execution. At training time, refinement losses added alongside UWM's standard objectives; at inference time, feedback ensures that action sampling and imagined rollouts remain tightly coupled.
Compute. Following UWM's reported setup, training runs efficiently on Nvidia A100 GPUs.
Table 19: Success rate for combining DP-AG with UWM LIBERO tasks.
Method
Book- Caddy Soup-Cheese Bowl-Drawer Moka-Moka
Mug-Mug
Average
DP
0.78
0.88
0.77
0.65
0.53
0.71
DP-AG
0.86
0.92
0.85
0.72
0.60
0.79
UWM
0.91
0.93
0.80
0.68
0.65
0.79
UWM DP-AG
0.94
0.95
0.87
0.75
0.70
0.84
Experimental Results on LIBERO. Following the UWM protocol, we fine-tune each model on the LIBERO benchmark tasks and report average success rates. As shown in Table 19| the hybrid model UWM + DP-AG outperforms both UWM and DP-AG alone, especially in manipulation tasks requiring real-time adaptation (e.g., Mug-Mug). This confirms that action oction-guided latent updates make world models more responsive, while world models provide long-horizon foresight to complement DP-AG's intra-step refinement.
Discussion. This extension highlights a conceptual bridge: world models offer look-ahead imagina- tion of possible futures, while DP-AG provides real-time corrective adaptation from action feedback within each imagined step. Together, they yield policies that are both farsighted and responsive, achieving long-horizon planning while adapting online to evolving cues. We will conduct in in-depth study on this subject in the future work.
42


Reviews:
[{"review_id": "3TeXsTugR2", "reviewer": "Reviewer_UAPM", "summary": "DP-AG establishes a dynamic perception-action loop by guiding feature evolution via the VJP of DP’s predicted noise. To reinforce interplay, a cycle-consistent contrastive loss aligns noise predictions from static and evolving features, enabling mutual perception–action influence. By using such method, they successfully transforms diffusion noise gradients into structured perceptual updates.", "questions": "Overall, I think the writing logic of this article is clear and the theory is reliable. What is particularly commendable is that it includes tests of simulation tasks and real-world robot control tasks. Two curious points are: \n\n1) Recently, many works related to diffusion model strategies have begun to focus on how to simultaneously accommodate online training. Could the author please elaborate further from a theoretical or experimental perspective on whether your method has the potential in the online mode? \n\n2) Multimodal decision-making ability - for the same task, can your method fit a variety of effective strategies? This is an important ability in real robot control scenarios and it was discovered that diffusion model strategies have this potential in some recent papers.", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: The paper is well-written, and the idea is interesting to me. The wide range of verification scenarios convinces readers of the effectiveness of the proposed method. The experimental results are good with clear and diverse visualization.\n\nWeaknesses: N/A", "was_revised": true, "final_justification": "I maintain my score", "rebuttal": "Thank you for your detailed and positive feedback. We appreciate your recognition of our theoretical soundness, comprehensive experiments, clear writing, and the value of our diverse visualizations. Your questions prompted valuable enhancements, and we hope our responses have demonstrated the robustness and versatility of DP-AG. If you have any further questions during the rebuttal, please let us know, we are happy to provide any additional details.\n\nBelow, we address your insightful questions with further theoretical discussion and new experimental evidence to clarify the online potential of our DP-AG and its multimodal decision-making ability.\n\n---\n## **Q1: Potential of DP-AG for Online Training: Theoretical and Experimental Perspectives.**\n\nThank you for pointing out the growing interest in online training for diffusion policies. We have taken this as an opportunity to adapt our DP-AG for online learning. With a few adjustments to how we update the latent features and calculate the loss, DP-AG can be trained online without major changes to the core method.\n\n**Latent Evolution in Real-Time:** In DP-AG, the perception-action interplay is realized by evolving latent observation features at every diffusion step, using the VJP of the current policy’s noise prediction (from **Eq. (6)** to **Eq. (8)**). In the online setting, this procedure can be applied **on the fly** as new observation-action pairs arrive: at every time step $t$, a new observation $o_t$ is encoded into the latent $z_t$ via the variational posterior, and then evolved across $K$ diffusion steps, where each latent update is conditioned on the most recent action and its predicted noise.\n\n**Streaming Loss and Parameter Update:** Rather than requiring a large static offline demonstration buffer, the model can be updated using small minibatches of the most recent transitions $(o_t, a_t)$. At each time step, the noise prediction loss, variational KL, and cycle-consistent contrastive loss are all computed as usual, but only using current and very recent data. The model parameters $\\theta$, $\\phi$, $\\psi$ are updated via online gradient descent (e.g., using RMSProp with a small learning rate [Ma et al., 2025]), just as in standard online RL.\n\nThe VJP-based evolution works incrementally, using only the current latent and action, no future rollouts or replay buffer required. Latent updates and policy improvements happen step by step, allowing perception and action to adapt together with each new experience. Some hyperparameters may need minor adjustment for online stability. As in offline training, each action is immediately followed by a latent update using the latest policy feedback, so the model’s understanding and action predictions stay in sync. This enables real-time adaptation to new conditions, sensor drift, or novel tasks during deployment.\n\n### **Experimental Evidence:** \n\nWhile our main experiments focus on batch imitation learning, preliminary experiments demonstrate DP-AG’s superior convergence speed compared to baseline methods. Faster convergence implies that DP-AG can efficiently incorporate new samples incrementally, making it ideal for online training.\n\nTo directly validate the online adaptation capability of DP-AG, we have implemented and evaluated the online training procedure described above on both the Push-T and Dynamic Push-T benchmarks. We observed that DP-AG maintains consistent performance improvements over the baseline methods even in an online context, quickly adapting to distributional shifts and evolving task dynamics with minimal performance degradation. \n\n**Table 1: Performance of DP-AG and Baselines Under Online Training.**\n\n| Method                  | Push-T (img)      | Push-T (kp)      | Dynamic Push-T (img) |\n|-------------------|-------------------|------------------|----------------------|\n| Diffusion Policy (DP)  | 0.87±0.04      | 0.95±0.03       | 0.65±0.85           |\n| DP-AG (offline)  | 0.93±0.02  | 0.99±0.01   | 0.80±0.53        |\n| DP-AG (online)   | 0.90±0.03  | 0.96±0.02   | 0.76±0.89        |\n\nWe will incorporate the above discussion and our experimental results on adapting DP-AG for online training into the camera-ready.\n\n---\n## **Q2: Multimodal Decision-Making: Fitness to A Variety of Effective Strategies.**\n\nThank you for raising this important question. DP-AG is well suited for multimodal decision-making because it is built on top of Diffusion Policy (DP), which has already shown strong ability to capture and represent multiple effective strategies for the same task.\n\n### **Theoretical Justification:** \nThe iterative denoising in DPs allows explicit parameterization of complex and multimodal behaviors. By extending DP into a dynamic perception-action interplay, our DP-AG not only preserves this multimodality, but further amplifies it via adaptive latent evolution. \n\n**How DP-AG Enhances Multiple Effective Strategies:**\n\n1. **Action-Conditioned Latent Evolution for Strategy Diversity:**\nDP-AG’s VJP-guided latent updates let the model refine features in response to action feedback, so each sampled trajectory adapts its latent state uniquely. This supports parallel exploration of multiple strategies by leveraging stochasticity from both the diffusion policy and latent SDE.\n\n2. **Cycle-Consistent Contrastive Loss for Multimodal Structure:**\nThe contrastive loss keeps features for the same action close and pushes different actions apart, organizing the latent-action space to separate and preserve multiple plausible strategies for each task.\n\n### **Experimental Validation of Multiple Effective Strategies:**\n\nTo empirically validate our method’s ability to discover and execute multiple strategies for the same task, we ran experiments on the Franka Kitchen benchmark. Franka Kitchen is well-suited for this as each task, like opening a drawer or flipping a switch, can be solved in several different ways, making it an ideal benchmark for true multimodal decision-making. \nMotivated by the experimental setup in multimodal DP [Li et al., 2024], we evaluate the model’s ability to discover and retain multiple effective strategies using the following approach:\n\n**Trajectory Collection:** We sample 40 successful trajectories for each task, all starting from the same initial configuration. Each trajectory is encoded in the space of end-effector waypoints. Using the Dynamic Time Warping (DTW) metric, trajectories are clustered into distinct groups.\n\n**Diversity Metrics:** The number of groups corresponds to the number of distinct strategies. We measure intra-cluster variance (compactness of each strategy) and inter-cluster distance (distinctiveness).\n\n**Mode-Specific Analysis:** We analyze the coverage and success rate for each discovered mode, confirming that each group represents an effective strategy.\n\n**Table 2. Multiple Strategy Discovery and Diversity on Franka Kitchen.**\n\n| Method        | # Modes (↑) | Success Rate (t1) | Success Rate (t2) | Success Rate (t3) | Success Rate (t4) | Inter-Cluster Dist. (↑) | Intra-Cluster Var. (↓) |\n|---------------|:-----------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------------:|:----------------------:|\n| FlowPolicy    | 1.1         | 0.96              | 0.86              | 0.95              | 0.87              | 2.0                     | 1.2                    |\n| DP (Baseline) | 2.8         | 1.00              | 1.00              | 1.00              | 0.99              | 7.1                     | 1.2                    |\n| DP-AG (Ours)  | 3.2         | 1.00              | 1.00              | 1.00              | 1.00              | 9.5                     | 1.3                    |\n\n**Table 3. Group-Specific Analysis: DP-AG on Franka Kitchen**\n\n| Mode ID | Coverage (%) | Success Rate (t1) | Success Rate (t2) | Success Rate (t3) | Success Rate (t4) | Strategy Description         |\n|---------|:------------:|:-----------------:|:-----------------:|:-----------------:|:-----------------:|------------------------------|\n| 1       | 41           | 1.00              | 1.00              | 1.00              | 1.00              | Left-handed drawer pull      |\n| 2       | 32           | 1.00              | 1.00              | 1.00              | 1.00              | Right-handed drawer pull     |\n| 3       | 27           | 1.00              | 1.00              | 1.00              | 1.00              | Two-step approach/mixed arm  |\n\n**Table 4. Robustness to Distribution Shift (Franka Kitchen, t4)**\n\n| Method        | # Modes (Shifted) | Success Rate (t4, Shifted) | Switch Rate (%) | Comment                             |\n|---------------|:-----------------:|:--------------------------:|:---------------:|-------------------------------------|\n| DP (Baseline) | 2.2               | 0.81                       | 35              | Sometimes adapts, but less reliable |\n| DP-AG         | 2.9               | 0.95                       | 58              | Switches to alternatives if blocked |\n\nTables 2, 3, and 4 show that DP-AG consistently discovers and uses multiple distinct strategies for each kitchen task, **over three on average**, compared to fewer with the baselines. These strategies are well-separated (high inter-cluster distance) and each group remains compact and reliable (low intra-cluster variance). Every DP-AG strategy achieves a 100% success rate and covers a range of approaches, like left-handed, right-handed, and mixed-arm. When tasks are made harder with obstacles, DP-AG still finds and switches between nearly three strategies with high reliability, while baselines see a significant drop in both diversity and success. This demonstrates that DP-AG is robust and flexible for real-world tasks.\n\n---\n## **Reference**\n\n**[Ma et al., 2025]** Ma et al., Efficient Online Reinforcement Learning for Diffusion Policy, ICML 2025.\\\n**[Li et.al., 2024]** Learning multimodal behaviors from scratch with diffusion policy gradient, NeurIPS 2024."}, {"review_id": "exBFiCbCbk", "reviewer": "Reviewer_fkRG", "summary": "The paper proposes a diffusion policy model where the input conditional observation feature is updated along the action denoising process. Specifically, the method leverages VJP, the Vector Jacobian Product from the action score prediction, to create a dynamic process to update the observation. It was claimed that such an update to the observation feature would result in improved smoothness in action prediction. To ensure the updated feature is not deviating from the original feature, a consistency loss is applied so that the score prediction based on the updated observation and the original observation is consistent. The paper conducts a set of experiments comparing diffusion policy vs action-guided diffusion policy. Experiment results show the benefit of the proposed method.", "questions": "Q1: Why would the updated observation feature (not updated observation) lead to a better policy model? All the info is already in the observation.\n\nQ2: What is the effect of consistency loss? To me, it seems that the goal is to make Vector Jacobian Product of the action score prediction the \"null space\" of the score estimator. Why?\n\nQ3: Could the achieved stability be achieved with other smoothness techniques? For example, we can simply perturb the input observation and request the score prediction to be consistent.\n\nQ4: The phrase “Act to See, See to Act” seems to be misleading. The environment has been seen. What has been updated is the observed features instead of the observation. There is no new see during the action generation.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength\n- The presented experiment results show the benefit of the proposed algorithm over the baseline.\n- The verification is done both using synthetic data and real robots.\n- The proposed approach is clean and can be applied to the existing diffusion policy model.\n\nWeakness\n- It is unclear why an updated observation would lead to better action prediction. Given the observation has been made, the update did not add new information. What has been updated is the observed feature, we are not updating the observation.\n- It is unclear why the proposed algorithm works. I feel the paper would benefit from a clearer explanation of the intuition.", "was_revised": true, "final_justification": "After checking the rebuttal and the reviews from the other reviewers, I lean towards keeping my original rating.", "rebuttal": "We sincerely thank the reviewer for their thoughtful feedback and greatly appreciate their recognition of DP-AG’s strengths, including our rigorous experimental validation, the clear improvement over baselines, and the integration with existing diffusion policy models. If any further questions or concerns arise during the rebuttal, please follow up with us, we are happy to provide additional details or clarifications as needed.\n\n---\n## **W1 and Q1: Clarifying Why Updated Observation Features Improve Action Prediction Despite Unchanged Observations.**\n\nThank you for highlighting this core idea. While the raw observation stays the same, what matters is how the model represents it internally. Our method lets the model update its internal features in response to action feedback, so its understanding of the scene adapts step by step as the policy unfolds, refining what it “notices” to better guide each action.\n\n**Why This Helps:** Think of driving a car: the view through the windshield might stay the same, but a driver keeps re-evaluating what is important as they steer or speed up. For example, when making a turn, the driver pays more attention to the curve’s edge or other cars. In our DP-AG, the model updates its understanding of the same observation based on action feedback, allowing it to adapt quickly and smoothly as each step unfolds.\n\n**Empirical Evidence:** Our results, especially **Figure 3**, clearly show that updating internal features with action feedback leads to smoother and more gradual changes in the model’s understanding of each observation. This step-by-step adaptation avoids sudden jumps (discontinuities) between internal states and generate more stable and natural predictions. For example, in the spiral regression, DP-AG tracks the curve smoothly, while models without this interplay show jagged, inconsistent outputs. The same effect appears in real robot tasks: as the model refines its internal view during action diffusion, its behavior becomes smoother and more coordinated. These findings show that continual refinement based on feedback leads to the kind of stable, real-world actions needed for practical robotics. \n\nWe will summarize and incorporate the above discussion in the camera-ready version.\n\n---\n## **W2: Unclear Intuition for Why DP-AG Works.**\n\nWe greatly appreciate the reviewer for raising this critical point. In our camera-ready, we will not only clarify the intuition and motivation for each module but also explicitly explain why our algorithm produces better action predictions. Our planned changes include:\n\n### **Why the Proposed Algorithm Works:**\nOur DP-AG works because it develops a closed-loop feedback between perception and action during the policy’s decision process. Instead of relying on a fixed and one-shot interpretation of the observation, the model continually refines its internal features based on feedback from its own evolving actions. This brings two significant benefits:\n\n**Continuous and Action-Driven Feature Refinement:** \nAt each step of action generation, the model asks: “given what I am about to do, is there a better way to interpret what I see?” The VJP update points the observation feature in the direction that makes action prediction more coherent and suited to the current step. This is similar to how a human driver refocuses attention as they execute a turn, not because the scene changes, but because their goals and context do.\n\n**Stability and Consistency with Contrastive Loss:** \nThe cycle-consistent contrastive loss ensures that as features evolve, they continue to support accurate and reliable action prediction, rather than drifting too far away from the static understanding of the observations. This helps the model maintain consistent interpretations and results in action trajectories that are both smooth and appropriate for the task context.\n\n**Why Better than Static Feature:** \nA static feature cannot adjust to local context or new intentions, it is “locked in” after initial encoding of the raw observation. By contrast, our dynamic and action-driven features enable the model to fine-tune its internal state at every decision step, leading to more adaptive, robust, and smooth actions, as both our theoretical work and empirical results demonstrate.\n\n**Proposed Revisions for Clarity:**\n\n1. We will add **Intuition and Motivation** section in a plain-language in the Introduction, explaining that our method allows the model to think twice and adapt its understanding at every step, just like people do in real-world tasks.\n\n2. For each component of our DP-AG (variational inference, VJP update, contrastive loss), we will provide an **Intuitive Explanation** sidebar in the Methods section, explaining what it does and how it contributes to robust and adaptive behavior.\n\n3. We will include a simple scenario (e.g., driving or robotic grasping) where, even if the visual input does not change, the model’s interpretation is refined as it acts, mirroring how our DP-AG works in practice.\n\n4. We will clearly connect our theorem and proofs to this intuition, explaining how bounded and action-driven feature evolution guarantees both stability and improved performance in practice.\n\n---\n## **Q2: Role of Consistency Loss: Does It Enforce a Null Space in the Action Score Estimator?**\n\nThe cycle-consistent contrastive loss keeps noise predictions from static and evolved features closely aligned, regularizing updates so that latent features adapt to support better actions while staying near their original, semantically meaningful encoding. **Rather than enforcing a null-space constraint, the loss maintains bounded, meaningful drift, ensuring that adaptations never stray too far from the original representation**. This principle is formalized in Theorem 1. Standard contrastive learning promotes clustering by pulling similar samples together and pushing dissimilar ones apart, encouraging compact and well-separated groups in feature space but not forcing representations into a null space. Our cycle-consistent loss extends this effect: it explicitly maintains alignment between static and updated features, ensuring smooth adaptation within each cluster and stable, continuous action trajectories. Theoretical and experimental results show our DP-AG enables flexible and task-driven latent evolution while preserving the semantic structure developed by clustering.\n\n---\n## **Q3: Alternative Smoothness Techniques.**\n\nWe appreciate this insightful suggestion and agree that comparing against alternative smoothness regularization techniques is important for understanding the unique value of our cycle-consistent contrastive.\nIn **Section H.3** of the Appendix, we have already conducted an ablation study where we replace the cycle-consistent contrastive loss with a simple MSE loss, applied to encourage the noise prediction to be smooth between the original and evolved features. The results show that while MSE provides some regularization, it is less effective than our contrastive loss in promoting both smoothness and task performance. Specifically, the contrastive loss not only enforces local smoothness but also maintains the semantic alignment between trajectories.\n\n### **Additional Ablation:** \nAs suggested by the reviewer, we conducted an additional ablation study on the Push-T benchmark, implementing input perturbation-based smoothness, and will include these results in the revised appendix for the camera-ready. \nFor each training sample, we will generate a perturbed version of the input observation by adding small Gaussian noise. Both the original and perturbed observations will be processed through the encoder to obtain their respective features. We have added different smoothness objectives (MSE and cosine similarity) between the predicted action noise scores from the original and perturbed features, enforcing that the policy’s score predictions remain stable under small input changes. \n\n**Table 1: Comparing DP-AG and Input Perturbation Smoothness Baselines on Push-T Benchmark.**\n\n| Method                  | Success Rate (img) | Success Rate (kp) | Action Smoothness (img ↑) | Action Smoothness (kp ↑) |\n|-------------------------|:------------------:|:-----------------:|:-------------------------:|:------------------------:|\n| Perturbation MSE        |       0.85         |      0.92         |          0.83             |         0.87             |\n| Perturbation Cosine     |       0.88         |      0.95         |          0.86             |         0.92             |\n| DP-AG (Contrastive Loss)|      **0.93**      |     **0.99**      |         **0.91**          |        **0.95**          |\n\nThe **action smoothness** is quantified as the mean inverse jerk over each action sequence, normalized between 0 and 1 for interpretability. **Table 1** demonstrates that DP-AG with contrastive loss clearly outperforms both input perturbation baselines on the Push-T benchmark for both image and keypoint inputs. DP-AG achieves the highest success rates and action smoothness, indicating it not only solves tasks more reliably but also generates more continuous actions. \n\n---\n## **Q4: Clarification of “Act to See, See to Act” Terminology.**\n\nThank you for raising this point. This seems to be the source of a key misunderstanding. In our paper, **see** refers to **the evolving internal representation**, **not the raw observation**. While no new sensory data is acquired, the model continually refines its internal understanding through action feedback, updating how it **interprets what it has already seen**. This aligns with cognitive science perspectives that view perception as an active process rather than a one-time event.\n\nWe will revise the title and related phrasing in the camera-ready to clarify that **seeing** in our context refers to the model’s dynamic, evolving internal feature representation, not to repeated acquisition of external observations."}, {"review_id": "DBUVUnn7cR", "reviewer": "Reviewer_t9vC", "summary": "This paper proposes Action-Guided Diffusion Policy (DP-AG), a new imitation learning framework that models a dynamic interplay between perception and action. The encoded observations are evolved with an action-guided SDE, and then construct a cycle-consistent contrastive loss to create a perception-action loop.", "questions": "I am still confused by the motivation of DP-AG:\n- \"Existing imitation learning methods decouple perception and action\": common IL methods optimize the perception (representation) in an end-to-end manner, where the gradient of the perception comes from the downstream action prediction objective. What do you mean by *decoupled*?\n- Line 130: \"However, it still treats perception as static: once extracted, observation features remain fixed during the corresponding action generation and cannot adapt to refinements\": during the diffusion process $\\epsilon_\\theta\\left(\\hat{a}_t^k, z_t, k\\right)$, why would $z_t$ need to evolve w.r.t. $\\hat{a}_t^k$ as $z_t$ only needs to serve as a compact representation of $o_t$?\n- In Eq. (6), I notice that DP-AG implements the drift coefficient with VJP (or JVP, if more precisely). What is the advantage of such implementation? Also, I notice that DP-AG does not require using the reverse SDE of Eq. (6), then what is the purpose of proposing the forward SDE?", "limitations": "NA", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n- DP-AG creates a perception-action loop that allows the gradient of the actions to flow back to the encoded observations.\n- DP-AG outperforms baselines in simulated and real-world benchmarks.\n\nCons:\n- The VJP can be very expensive to compute, especially with a high-dimensional action space or encoded observation space.\n- Lack of imitation baselines: DP-AG should include classic Diffusion imitation learning baselines like Diffusion Policy, Diffuser, and Hierarchical Diffuser.\n\n\nVisuomotor Policy Learning via Action Diffusion\n\nPlanning with Diffusion for Flexible Behavior Synthesis\n\nSimple hierarchical planning with diffusion", "was_revised": true, "final_justification": "The authors provide a reasonable justification for the evolving feature design, and the experiments with suggested baselines also prove the effectiveness of the proposed method. After reading the rebuttal and summary from other reviewers, I have decided to raise my score to borderline accept.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback, and for recognizing the technical contributions and potential impact of our work. We appreciate your insightful questions, which have helped us clarify and better communicate the novelty and significance of our DP-AG. We hope our clarifications fully address your comments. If you have any further concerns or questions, we would be very happy to discuss them during the rebuttal. \n\n---\n## **W1: VJP Computation in High-Dimensional Spaces.**\n\nWe appreciate the reviewer’s concern about VJP computation cost in high-dimensional spaces. In both simulation and real-robot experiments, we found the overhead to be negligible thanks to efficient **autodiff** libraries in PyTorch. For instance, on Franka Kitchen (action dim 9, feature dim 256), VJP adds less than 10% to per-step runtime compared to baseline DP.\n\n**More Importantly, this small overhead does not affect real-time deployment or practical robot responsiveness.** In our real-world UR5 robot experiments (see Table 3 and video demonstration), DP-AG generates smoother and more responsive actions than the DP baseline, both qualitatively and quantitatively. Our DP-AG maintains real-time inference frequencies, and the robot motions generated by DP-AG are even **more fluid, with lower jerk and faster task completion**. For example, in the Painting and Candy Push tasks, the robot controlled by DP-AG exhibits both faster real-time response, demonstrating that the VJP computation in no way limits control performance.\n\nMoreover, DP-AG converges faster than the DP baseline, reducing total training time (see Figure 5 and Appendix Figures 12, 13), which more than compensates for the already minor VJP cost. VJP operations are also fully parallelizable for larger-scale problems. Across all experiments, DP-AG maintains real-time control and improves action smoothness and responsiveness, showing that VJP overhead is negligible.\n\n---\n## **W2: Lack of Imitation Baselines.**\n\nWe sincerely thank the reviewer for suggesting additional baselines to further strengthen the evaluation of our DP-AG method.\n\n### **Clarification on DP Baseline:** \n\nWe would like to take this chance to clarify that **DP-AG is built directly on Diffusion Policy (DP) [Chi et al., 2023]**, serving as both the architectural and experimental foundation of our work. Our main contribution is adding an action-guided feedback loop to DP, enabling dynamic feature updates during action prediction. DP is not just a baseline but the core of our method, and we will emphasize this direct lineage more clearly in the camera-ready version and table captions.\n\n### **Suggested Baseline Experiments:**\n\n**Evaluation Benchmarks:** We have added results for Diffuser [Janner et al., 2022] and Hierarchical Diffuser [Chen et al., 2024] as baselines. For comprehensive evaluation, we use the Maze2D and Multi2D benchmarks from D4RL, which evaluates long-horizon navigation in various maze layouts, and the FrankaKitchen benchmark, which evaluates multi-stage robotic manipulation using an offline RL protocol. Performance is measured by average success in reaching goals for navigation tasks, and by the number of completed sub-tasks per episode for FrankaKitchen, reflecting the agent’s ability to generalize and coordinate complex skills.\n\n**Implementation Details:** We follow the experimental protocol of [Chen et al., 2024] and use the D4RL suite. All settings, trajectory segmentation, planning horizons (e.g., K = 15 for Maze2D/Multi2D), match the baselines. Models are evaluated by average return or goal success rate over 100 seeds, with identical data splits and metrics as in the baselines for fair comparison.\n\n**Results:** As shown in Tables 1 and 2, DP-AG consistently outperforms Diffuser and Hierarchical Diffuser on both long-horizon planning and manipulation tasks. On Maze2D and Multi2D, DP-AG achieves an average score of 161.5, compared to 124.4 for Diffuser and 145.0 for Hierarchical Diffuser. On the challenging FrankaKitchen, DP-AG reaches 80.4, well above the baselines.\n\n**Table 1: Trajectory Planning Experiments on Maze2D and Multi2D (Long-horizon).**\n\n| Environment| Diffuser| Hierarchical Diffuser | **DP-AG (Ours)**|\n|----------------|:--------------:|:---------------:|:---------------:|\n| Maze2D U-Maze| 113.9 ± 3.1| 128.4 ± 3.6 | **142.7 ± 2.9**|\n| Maze2D Medium| 121.5 ± 2.7| 135.6 ± 3.0| **150.2 ± 2.6**|\n| Maze2D Large| 123.0 ± 6.4| 155.8 ± 2.5| **172.1 ± 2.2**|\n| Multi2D U-Maze| 128.9 ± 1.8| 144.1 ± 1.2 | **168.2 ± 1.2**|\n| Multi2D Medium| 127.2 ± 3.4| 140.2 ± 1.6| **153.7 ± 1.3**|\n| Multi2D Large| 132.1 ± 5.8| 165.5 ± 0.6| **182.2 ± 0.5**|\n| Average | 124.4 | 145.0| **161.5**|\n\n**Table 2: Multi-Stage Robotic Manipulation on FrankaKitchen (Long-Horizon Generalization).**\n\n| Task| Diffuser| Hierarchical Diffuser | **DP-AG (Ours)**|\n|-------------|:-------------:|:-----------------:|:---------------:|\n| Partial Kitchen| 56.2 ± 5.4| 73.3 ± 1.4| **82.2 ± 1.3**  | \n| Mixed Kitchen | 50.0 ± 8.8| 71.7 ± 2.7| **78.5 ± 2.1**  | \n| Average| 53.1| 72.5  | **80.4** | \n\n---\n## **Q1: Clarification on Decoupling.**\n\nThank you for asking about this key point. By **decoupled**, we mean that in standard IL, the perception module develops a fixed feature from each observation, which is then used by the policy to predict actions, without any further adaptation during rollout. There is no real-time feedback from actions back to the observation features; the features remain static throughout the diffusion. In contrast, DP-AG closes this loop: as actions are refined during diffusion and inference, the observation features are dynamically updated based on the evolving action predictions. This real-time feedback allows the perception to continuously adapt to the current action context, rather than relying on a static encoding. This closed-loop interplay between perception and action is the core innovation of our work.\n\n---\n## **Q2: Why Evolve Features with Respect to Actions During Diffusion?**\n\nWe greatly appreciate this insightful question, as it gets to the heart of the main novelty and practical significance of our DP-AG. \n\nIn standard policy learning, $z_t$ is a fixed, compact representation of the raw observation $o_t$, and it remains unchanged as long as $o_t$ does not change. While this may appear sufficient, it fundamentally ignores a critical aspect of intelligent decision-making: **the fact that the meaning and salience of what is observed can, and often should, change as actions unfold.**\n\nThe main contribution of our DP-AG is to make $z_t$ dynamic, explicitly allowing it to evolve in response to the ongoing sequence of action refinements $\\hat{a}_t^k$. By doing so, DP-AG ensures that the feature representation at each diffusion step not only encodes what is seen, but also incorporates feedback from the agent’s own evolving intentions and actions. This mechanism is what enables the perception-action loop in our DP-AG and is, in our view, the missing ingredient in the existing methods. \n\nConsider a driver at a busy intersection: the scene itself does not change, but their focus shifts depending on whether they’re turning, accelerating, or stopping. Each action changes what information matters most. Similarly, DP-AG continually updates its understanding of a fixed observation based on its evolving actions, allowing it to adapt in real time. This leads to smoother, more robust, and context-aware actions, as shown in both our synthetic and real-world experiments.\n\n---\n## **Q3: Why Use VJP/JVP for Drift? Why Propose the Forward SDE?**\n\nOur latent evolution SDE is not an isolated process with its own noise direction; instead, it is **tightly coupled to the phase of the DP’s SDE by design**. To be specific, the drift of our latent SDE is **determined directly by the VJP of the DP’s noise prediction**. This means the direction of evolution for the observation features is always **aligned with the direction of the DP’s own SDE**, automatically **forward** during training and **reverse** during inference:\n\n**During Training (Forward Noise-Adding Phase):** The DP’s SDE injects noise into actions, and the noise prediction used in the VJP has a positive sign, pushing the latent features to adapt forward in the same direction as the noisy action trajectory. \n\n**During Inferring/Generating Actions (Reverse Denoising Phase):** The DP’s SDE removes noise (the noise prediction’s sign flips), and so does the drift in our feature SDE, meaning the latent features now evolve backward, directly in sync with the reverse denoising process of actions. \n\nThus, we do not need to manually set forward or reverse modes for the latent SDE, **its direction automatically stays in sync with the action SDE because the drift is always computed using the VJP of the DP’s current noise prediction**. The VJP is essential here: it precisely and efficiently propagates gradients from action to feature space, capturing the correct direction and scale of updates at every step. This ensures that feature evolution is always aligned with the phase of action SDE, which would not be possible with simple gradients or heuristic mappings. In short, the VJP acts as a mathematical bridge, tightly coupling perception and action dynamics throughout both training and inference, and enabling DP-AG to adaptively refine features in ways that static or decoupled approaches cannot.\n\nWe recognize that the phase-synchronized evolution was not made explicit enough in our manuscript and thank you for highlighting this point. We will clarify this in the camera-ready.\n\n---\n### **Reference**:\n\n**[Chi et.al, 2023]** Chi et.al., Diffusion policy: Visuomotor policy learning via action diffusion, IJRR 2023.\\\n**[Janner et.al., 2022]** Janner et.al., Planning with diffusion for flexible behavior synthesis, ICML 2022.\\\n**[Chen et.al., 2024]** Chen et.al., Simple hierarchical planning with diffusion, ICLR 2024."}, {"review_id": "qawbLiv5KM", "reviewer": "Reviewer_HxxU", "summary": "This paper introduces Action-Guided Diffusion Policy (DP-AG), a novel imitation learning framework that addresses the limitation of static perception in existing methods. The core contribution is a dynamic perception-action loop where the policy's latent observation features evolve during the action generation process, guided by the Vector-Jacobian Product (VJP) of the diffusion model's noise predictions. A cycle-consistent contrastive loss reinforces this interplay, and the method demonstrates significant performance gains over state-of-the-art baselines on a variety of simulation and real-world robotic manipulation tasks", "questions": "1. Can you provide more semantic intuition for the VJP update term? What does this structured force represent in the context of perception and action, beyond its mathematical formulation?\n2. Regarding the Peg-in-Hole task, what was the primary failure mode for the baseline DP (e.g., incorrect depth estimation, inability for fine-grained correction)?\n3. Why is the relative-similarity-based contrastive loss superior to the absolute-target-based MSE likelihood objective for this task?", "limitations": "Yes, the authors have adequately addressed the limitations. The Broader Impacts section discusses the reliance on expert demonstrations, which could introduce biases, and correctly points to the need for validating demonstration data. This is a sufficient acknowledgment of the primary limitations of the imitation learning paradigm.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The core idea of creating a dynamic perception-action loop within a single action generation step is highly novel and significant. It directly addresses the key limitation of static perception in many IL policies.\n2. The empirical validation is extensive, showing strong results on standard benchmarks and introducing a new dynamic task (Dynamic Push-T) that effectively showcases the method's advantages. The model's success in the difficult Peg-in-Hole task (85% vs. 0% for the baseline) provides compelling evidence for the practical benefits of the proposed perception-action interplay.\n3. The method is supported by solid theoretical grounding, including a variational lower bound and a proof connecting the contrastive loss to trajectory continuity, which adds rigor to the paper.\n4. The paper is well-written and clearly presented. The figures effectively illustrate the core concepts.\n\nWeakness\n1. Several simulation benchmarks in Table 2 are near-saturated, with both DP-AG and baselines achieving perfect scores. While this demonstrates non-regression, these tasks don't fully differentiate SOTA methods. The paper's impact is more clearly shown in the more challenging dynamic and real-world settings.\n2. The paper lacks a deep semantic intuition for why the VJP is the ideal guiding force for latent updates. Explaining what this signal represents (e.g., a direction of uncertainty reduction) would improve clarity for readers not versed in stochastic adjoint methods.\n3. The work could be better contextualized against model-based RL methods that also use latent dynamics (e.g., Dreamer). A clearer distinction between DP-AG's intra-policy state refinement and a world model's future state prediction would sharpen the paper's claimed contribution.", "was_revised": true, "final_justification": "The authors' response has addressed my concerns. I will maintain my score, i.e., borderline accept for the paper.", "rebuttal": "We sincerely thank the reviewer for their thorough evaluation and recognition of our work’s key contributions, including the novelty of our dynamic perception-action loop, comprehensive empirical validation, strong theoretical foundation, and clear presentation. Your constructive feedback has been invaluable in helping us clarify the core mechanisms and further strengthen both the theoretical and experimental sections of our paper. \n\n---\n## **W1: Near-Saturated Simulation Benchmarks.**\n\nThank you for highlighting the issue of benchmark saturation. We fully agree that the existing imitation learning (IL) benchmarks are overly deterministic and lack diversity, making them easy to solve by memorization rather than true adaptability. This limitation motivated us to introduce the **Dynamic Push-T benchmark**, which goes beyond static setups by requiring real-time adaptation to unpredictable events. In Dynamic Push-T, a moving ball disrupts the robot’s task, forcing the agent to react and adjust its strategy on the fly. Because each episode is different, the task truly evaluates the policy’s adaptability, not just memorization. Inspired by your feedback, we also plan to develop even more challenging and diverse IL benchmarks in future work, aiming to better evaluate adaptability and bridge the gap between simulation and real-world deployment.\n\n---\n## **W2 and Q1: Semantic Intuition and Role of VJP.**\n\n### **What the VJP Signal Represents:** \n\nThe VJP in our DP-AG is much more than a technical gradient; it is a principled, dynamically generated signal that identifies the direction in feature space that will **most efficiently reduce uncertainty in the next action**. At each diffusion step, the VJP effectively answers the question: “If I could refine my understanding of the observation right now, what change would most help the model be more certain and robust about its next action?” \n\nSemantically, this means the VJP functions as a real-time attentional force, **dynamically guiding the latent features towards focusing on precisely those aspects of the observation that would most significantly improve immediate action selection**. For example, in Peg-In-Hole task where depth ambiguity limits success, the VJP will naturally point the latent representation to sharpen depth cues, and this directional force is instantly aligned with the needs of the current stage of action denoising. This may explain why the DP baseline, which lacks such synchronized perception refinement, fails to complete the peg-in-hole task in multiple trials, while our DP-AG succeeds.\n\n### **What Does VJP Represent in the Context of Perception and Action:**\n\nBeyond its mathematical role, the VJP in DP-AG serves as a real-time synchronization signal between perception and action. The latent SDE’s drift, including both direction (forward or reverse) and scale, is directly determined by the VJP of the action SDE (i.e., DP), so the exploration mode of the action SDE is automatically propagated to the latent SDE. This context-aware feedback loop ensures perception and action evolve together at every diffusion step, enabling the agent to adapt in real time. \n\nIn practice, this means perception dynamically adapts to each action refinement, allowing the agent to respond to changing needs and uncertainties in real time. This dynamic coupling leads to smoother and more adaptive behavior, overcoming the limitations of static or unsynchronized methods.\n\n### **How VJP Differs from Stochastic Adjoint and Other Gradient-Based Methods:**\n\nThe stochastic adjoint methods compute gradients in two stages: forward evolution and then backward updates, so latent features remain fixed during action inference, with no real-time adaptation. In contrast, DP-AG’s VJP enables immediate feedback-driven updates at every action diffusion step, capturing the instant sensitivity between actions and features. This real-time signal uniquely preserves the high-dimensional action-feature relationship, keeps feature updates phase-aligned with action diffusion, and lets perception adapt instantly to each action, capabilities not possible with standard adjoint methods.\n\n---\n## **W3: Contextualization Against Model-Based RL.**\n\nThank you for raising this important point about contextualizing DP-AG against model-based RL methods with latent dynamics, such as Dreamer [Hafner et al., 2025] and Unified World Models (UWM) [Zhu et.al., 2025]. While world models focus on learning a latent generative model that predicts future states in the latent space to support long-horizon planning and imagination, DP-AG takes a fundamentally different method focused on real-time intra-policy state refinement.\n\n### **Key Distinction:**\n\n**World Models:** These models learn to “imagine” possible futures by generating latent trajectories or predicting future observations, which are then used for planning or behavior cloning. The latent features are updated via the generative process, but once encoded, they are typically static within a policy rollout, there is no action-conditioned adaptation of the latent features at inference time.\n\n**DP-AG:** In contrast, our DP-AG does not attempt to reconstruct or predict the future environment. Instead, it enables real-time adaptation of the latent observation features within a single action diffusion step, guided by immediate action feedback (via VJP). \n\n### **Integration and Experimental Evidence:**\n\nAs suggested, we explored combining DP-AG’s action-guided latent updates with a generative world model (UWM). In this hybrid, UWM’s powerful generative latent predictions should be enhanced by DP-AG’s intra-step and action-driven refinement, producing more adaptive and context-aware behaviors. \n\n### **Experimental Results on LIBERO**\n\nFollowing the evaluation protocol of UWM, we fine-tune each model on the LIBERO benchmark tasks before evaluating their performance. The results are summarized below:\n\n**Table 1. Success rate for combining DP-AG with world model on LIBERO tasks.**\n\n| Method| Book-Caddy| Soup-Cheese| Bowl-Drawer| Moka-Moka| Mug-Mug| Average|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n| DP |0.78|0.88|0.77|0.65|0.53|0.71| \n| DP-AG | 0.86 | 0.92| 0.85| 0.72 | 0.60 | 0.79 | \n| UWM| 0.91 | 0.93 | 0.80 | 0.68 | 0.65 | 0.79 |\n| UWM&DP-AG| **0.94** | **0.95** | **0.87** | **0.75** | **0.70** | **0.84** |\n\nAs shown in Table 1, the hybrid **UWM&DP-AG** outperforms both UWM and DP-AG alone, especially on manipulation tasks that need real-time adaptation. This confirms that action-guided latent updates make world models more responsive.\n\n---\n## **Q2: Primary Failure Mode for DP in Peg-In-Hole.**\n\nIn the Peg-in-Hole experiment, both DP and DP-AG rely only on RGB images from a scene and wrist-mounted camera, **no depth sensors or ground-truth 3D info are used**.\n\n**Why the DP Baseline Fails:** Baseline DP encodes observation features once per frame and keeps them static throughout the action sequence. So, if the peg is misaligned due to changes in viewpoint or lighting, DP cannot recognize or correct the failure, its internal state cannot adapt to the failed actions, causing repeated ineffective attempts when blocked.\n\n**How DP-AG Succeeds:** Our DP-AG, however, uses action feedback to guide real-time updates of its latent features. When the peg bumps the rim, the VJP-driven update acts like a dynamic attentional shift, making the model focus on the contact area and refine its perception based on action outcomes. Over multiple steps, DP-AG can implicitly infer depth or alignment cues and adapt its actions accordingly.\n\nIn essence, DP-AG adapts perception on the fly, much like a person adjusting their focus when a key does not fit, using feedback to “discover” the right correction through continuous perception-action interplay.\n\n---\n## **Q3: Relative-Similarity-Based Contrastive vs. Absolute-Based MSE.**\n\nThe key advantage of our relative-similarity-based contrastive loss over an MSE objective is how it encourages robust feature representations, as demonstrated in contrastive learning for representation learning.\n\nThe contrastive loss pulls together positive pairs (static and VJP-evolved features of the same observation) while pushing apart negatives (different samples), preserving relative relationships and supporting more context-aware adaptation. This clustering effect keeps the latent evolution aligned with the original context but not rigidly fixed, allowing for smooth transitions and greater robustness to noise or variations across diffusion steps.\n\nIn contrast, an MSE loss enforces strict matching to a fixed target, which can suppress adaptive adjustments and may lead to instability or over-correction, especially in noisy or multi-modal settings. Our ablation study (Appendix H.3) shows that the contrastive loss achieves higher success rates and produces smoother, more natural actions than MSE-based regularization.\n\n### **Additional Ablation:** \n\nAdditional ablations (including input perturbation with both MSE and cosine consistency), as suggested by **Reviewer fkRG Q3**, further confirm that contrastive loss better supports stable and cycle-consistent adaptation in practice. These results are summarized in Table 1.\n\n**Table 1: Comparing DP-AG and Input Perturbation Smoothness Baselines on Push-T Benchmark.**\n\n| Method| Success Rate (img) | Success Rate (kp) | Action Smoothness (img ↑) | Action Smoothness (kp ↑) |\n|--|:--:|:--:|:--:|:--:|\n| Perturbation MSE|0.85|0.92|0.83|0.87 |\n| Perturbation Cosine|0.88 |0.95|0.86|0.92|\n| DP-AG|**0.93**|**0.99**|**0.91**|**0.95**|\n\n---\n## **Reference**\n\n**[Jia et.al., 2024]** Jia et.al., Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations, ICLR 2024.\\\n**[Hafner et.al., 2025]** Hafner et.al., Mastering Diverse Domains through World Models, Nature 2025.\\\n**[Zhu et.al., 2025]** Zhu et.al., Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets, Arxiv 2025."}, {"review_id": "lCGokUaEGb", "reviewer": "Reviewer_uXWV", "summary": "This paper proposes perception-action interplay in diffusion policy learning. Instead of conditioning on a static encoding of the observation, the framework will refine the observation in the latent space by using an action-guided SDE. A contrastive learning method is applied to the action denoising module to make sure the latent observation is consistently changing with actions.\n\nEvaluations in both the simulator and the real world show the effectiveness of this method in improving performance and smoothness in manipulation.", "questions": "1. Could you please explain the motivation for refining the representation of real observations with guidance from predicted actions? \n2. It's unclear to me what drift of observation latent is expected to improve the action prediction. Could the authors consider replacing the variational posterior with a variational autoencoder so we can decode the evolved latent to see whether it has any interpretable semantic meaning?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The writing is clear and easy to follow.\n2. The theoretical proof and evaluation are thorough.\n\n# Weakness\n1. Some designs of the DP-AG do not seem to align with the motivation. Humans could leverage the observation-action-obsevation loop for adaptive behavior because new information might be discovered through interaction and changing viewing angle. However, this pipeline does not get any additional information in the interplay, instead, it changes the representation of observation through an SDE guided by action denoising. The reason why the representation from ground truth observation should be refined by a prediction needs further justification. Besides, the drift of the latent observation is constrained by contrastive learning. While this prevents the model from collapsing to a trivial solution (e.g., purely imagining the observations), it is unclear to me how to determine the balance of evolution versus keeping similar to the static observation latent (i.e., what is the expected amount and direction of drift that is beneficial for action prediction).\n2. The authors could consider comparing with UWM [1] fine-tuned on the selected tasks, which has a similar motivation of coupling perception and action.\n\n[1] Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets", "was_revised": true, "final_justification": "During rebuttal. The authors explain the intuition of this method, which mostly solves my concern about motivation. The authors also mentioned they will add new visualizations and change the writing in revision. Therefore, I raise my rating to 4.", "rebuttal": "We sincerely thank the reviewer for their recognition of our work’s strengths, including the clarity of writing and the thoroughness of theoretical proofs and evaluations. Your thoughtful feedback and constructive questions have been invaluable in helping us enhance our motivation and method choices. In response to your suggestions, we will incorporate all requested changes into the camera-ready. We welcome any further suggestions or questions during the rebuttal and would be happy to provide more details as needed.\n\n---\n## **Why Refine Representation Without New Sensory Input (W1 and Q1)**\n\nWhile humans often adapt by seeking new views, they also frequently adjust their interpretation of the same sensory input based on feedback from their actions, even without receiving any new information.\nOur DP-AG is designed to capture this adaptive process. Unlike standard policies that keep their features fixed until a new image arrives, DP-AG continually updates its latent representation at every action diffusion step, using real-time feedback from its own predictions. This allows the model to **adapt immediately and respond to small errors or unexpected actions, rather than waiting for new observations**. This is important for tasks where external views do not change, but adaptation is still needed.\n\n**Concrete Example (Peg-in-Hole):** \nIn our real-world Peg-In-Hole task, both DP-AG and the baseline DP use only RGB images, **no depth or extra sensors**. As the peg approaches the hole, the visual input stays almost unchanged. The baseline DP extracts features once and reuses them throughout the diffusion; if the peg is blocked, it simply repeats the same action without adapting, much like a person ignoring feedback. In contrast, DP-AG uses each action noise prediction to update its internal focus via VJP, paying more attention to cues near the problem area. Even without new input, DP-AG adapts to the details that matter, recovering and succeeding where the static DP fails (85% vs. 0% success). It is like parking with a rearview camera: the view does not change, but bumps prompt the driver to refocus and adapt.\n\n---\n## **How The Drift Improve Action (Q2)**\n\nIn DP-AG, the drift of the observation latent is not arbitrary, it is directly calculated using the Vector-Jacobian Product (VJP), a standard tool from automatic differentiation. The VJP determines exactly how the latent features should be adjusted to most effectively reduce the uncertainty in the next action prediction.\n\n**What Drift does The VJP Produce:**  Whenever the model experiences high uncertainty or an unsuccessful action, the VJP points to the direction in feature space where even a small adjustment would lead to a more confident and accurate next action. **This drift always aims to reduce the prediction error or uncertainty, rather than simply moving features at random.**\n\n**How does This Help:** For example, in the Peg-In-Hole task, if the peg is blocked or misaligned, the model’s uncertainty rises. The VJP then guides the latent feature update to emphasize visual cues (like edges or shadows near the hole) that are most relevant for correcting this misalignment. **The refined feature is immediately used to predict the next action**, helping the robot to better adjust its movement in real time. As the task progresses, feature updates keep shifting focus toward observation details that are actually helping the policy reduce error and uncertainty, reinforcing cues that support successful actions.\n\n---\n## **How To Determines Drift's Amount and Direction (W1)**\n\nThe core of DP-AG is its **tight synchrony** between the action SDE (DP) and the latent SDE for feature evolution. The drift of the latent feature is set in real time by the current state of the action SDE:\n\n**Direction:** At each diffusion step, the VJP of the DP’s noise prediction determines the direction of feature evolution, always pointing toward reducing the next action’s uncertainty. This ensures the latent SDE stays phase-aligned with the action SDE, forward during training, reverse during inference, with no manual mode switching needed.\n\n**Magnitude:** The scale of the drift is adaptive: features update more when uncertainty is high and less when confident. The cycle-consistent contrastive loss anchors these updates, pulling features back toward the original observation and preventing excessive drift.\n\nIn short, the VJP prescribes exactly how much and in which direction to update features based on the DP's current confidence, while contrastive loss keeps the adaptations grounded and meaningful.\n\n---\n## **How to Determine the Balance (W1)**\n\nThank you for highlighting this important question about how the contrastive loss controls the balance between latent feature evolution and staying close to the static observation encoding. \n\nThe key lies in the **clustering effect** observed in contrastive representation learning. In self-supervised learning, contrastive loss encourages features from similar data (positives) to form tight and semantically meaningful clusters in the latent space, while pushing apart features from different data (negatives). We apply this principle in our DP-AG to maintain a task-driven clustering in the latent space:\n\n**Pull effect:** For each diffusion noise prediction, the contrastive loss pulls the action-evolved latent feature toward its static feature, ensuring the representation remains anchored to the true observation and avoiding excessive or trivial drift.\n\n**Push effect:** At the same time, the loss pushes the evolved latent features away from those of different observations. This keeps the adaptation for each trajectory distinct, preserving task-relevant differences and preventing collapse into a single and average solution.\n\nThis clustering mechanism allows the model to adapt the latent features in a task-driven way using action feedback, while the contrastive loss keeps these adaptations within the semantic vicinity of the observation. The balance between evolution and stability is therefore achieved by enabling flexible and action-informed updates that remain constrained to the meaningful structure of the data, just as contrastive learning achieves robust and organized representation spaces in self-supervised learning.\n\n---\n## **Comparing with UWM (W2)**\n\nThank you for this valuable suggestion. We agree that comparing to **Unified World Models (UWM) [1]** is especially meaningful given its recent advances in joint perception and action learning. \n\nBeyond direct comparison, we also explored combining our DP-AG’s action-guided latent adaptation with the UWM’s world model. This hybrid framework leverages the strengths of both methods, using UWM’s latent generative modeling for future state prediction and our DP-AG’s latent representation refinement for more context-aware and adaptive control. By integrating these two methods, we can take advantage of the world model’s generalization while enhancing adaptability through our action-guided latent updates.\n\n**How to Combine DP-AG with UWM:** UWM predicts future video frames and action sequences in latent space for planning and behavior cloning, but its latent encoding lacks real-time action feedback during rollout. To address this, we insert DP-AG’s VJP-based latent update into UWM’s latent space ($[o, t_a, t_{o’}]$ per [1]). At each diffusion step, DP-AG’s action-guided update refines the latent before it enters UWM’s diffusion transformer. We also apply cycle-consistent contrastive loss to ensure smooth and consistent adaptation. This hybrid UWM&DP-AG model combines UWM’s long-horizon planning with DP-AG’s real-time perception-action interplay for more robust and context-aware behavior.\n\n**Experimental Results on LIBERO:** Following the evaluation protocol of [1], we fine-tune each model on the LIBERO benchmark tasks before evaluating their performance. The results are summarized below:\n\n**Table 1. Success rate for combining DP-AG with world model on LIBERO tasks.**\n\n| Method| Book-Caddy| Soup-Cheese| Bowl-Drawer| Moka-Moka| Mug-Mug| Average|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n| DP |0.78|0.88|0.77|0.65|0.53|0.71| \n| DP-AG | 0.86 | 0.92| 0.85| 0.72 | 0.60 | 0.79 | \n| UWM| 0.91 | 0.93 | 0.80 | 0.68 | 0.65 | 0.79 |\n| UWM&DP-AG| **0.94** | **0.95** | **0.87** | **0.75** | **0.70** | **0.84** |\n\n**Table 1** shows that the hybrid UWM&DP-AG consistently outperforms both standalone UWM and DP-AG, especially on manipulation tasks requiring context-sensitive adaptation. This demonstrates that action-guided latent refinement gives the hybrid a clear advantage where either approach alone may struggle. The combined model retains UWM’s long-horizon planning and adds DP-AG’s real-time adaptability, highlighting DP-AG’s broad impact and strong synergy with world models.\n\n---\n## **Variational Autoencoder to Visualize Semantic Meaning (Q2)**\n\nWe greatly appreciate the valuable suggestion. To address this, we have integrated a VAE decoder into both DP-AG and the DP baseline for further analysis. For a given action sequence:\n\n**Reconstruct observation sequences:** Visualize the original camera image, the reconstruction from the static latent, and the reconstruction from the evolved latent at each diffusion step, showing how the model’s internal representation shifts as action diffusion unfolds.\n\n**Decode end-effector positions:** Map the latent at each step to end-effector positions, illustrating how latent evolution relates to the robot’s actual behavior.\n\n**Plot latent trajectories:** Use t-SNE to visualize the continuity and clustering of the evolved latent features for both DP-AG and DP, highlighting the effect of cycle-consistency contrastive loss.\n\nPreliminary results show that DP-AG generates smoother and more continuous latent trajectories aligned with successful actions, while the DP baseline shows abrupt and less interpretable shifts (as in Fig. 3). We will include these visualizations in the camera-ready."}]

Author rebuttal:

Meta-review: The paper tackles the limitation that imitation learning policies generate actions from static observation embeddings within a short rollout, missing perception–action reciprocity. It proposes DP-AG, which evolves observation features during diffusion-based action generation via an action-guided latent SDE whose drift is the VJP of the policy’s noise predictor, aligning latent updates with action denoising; a cycle-consistent InfoNCE aligns noise predictions from static vs. evolved latents to close the loop. The authors derive an ELBO for this latent process and, under Lipschitz assumptions, prove that the contrastive objective induces continuity in both latent and action trajectories. Empirically, DP-AG outperforms recent baselines on Robomimic, Franka Kitchen, Push-T, and Dynamic Push-T, and achieves large gains in real UR5 tasks (Painting, Candy Push, Peg-in-hole) with smoother and more successful control. Rebuttal results indicate feasibility for online training and multimodal strategies, and extensions to flow matching and world models are discussed.

Strengths
- Clear mechanism to “close the loop” within each diffusion rollout by using VJP as a structured, action-conditioned force on latent perception. The cycle-consistent contrastive objective is well-motivated and connects to provable continuity.
- Provides an ELBO and continuity results; method is simple to graft onto DP and, per appendix, extendable to flow matching. Empirical coverage is quite broad (simulation + real robot) with consistent improvements and faster convergence; inference latency unchanged.
- Strong real-world results, especially Peg-in-hole with only RGB, where DP-AG shows large gains over DP. Dynamic Push-T illustrates benefits under disturbances. Rebuttal adds baselines (Diffuser/Hierarchical Diffuser), online learning evidence, and multimodality analyses.
- Writing generally clear; figures help; rebuttal further improved intuition and positioning.

Weaknesses 
- Motivation/intuition gap (fkRG, uXWV, HxxU): Why evolving features without new sensory input helps; why VJP is the right signal. The rebuttal provides semantic intuition (VJP as action-uncertainty-reduction/attentional force; phase-synchronized with action SDE), added visualizations (planned VAE decodes, t-SNE), and connects theory to practice. Reviewers indicated concerns were addressed and raised/maintained positive ratings.
- “Decoupling” claim (t9vC): Clarified that decoupling refers to static within-rollout features in standard DP/IL; DP-AG closes the loop at diffusion-step granularity.
- Objective alignment: Paper derives an ELBO but trains primarily with DP loss + contrastive + KL; rebuttal justifies omitting likelihood regression term to avoid redundancy/instability and backs it with ablations showing contrastive > MSE/perturbation consistency.
- Computational overhead & scalability (t9vC): Authors report modest training overhead (~4–10%), negligible inference cost, faster convergence; real-time control preserved; VJP efficiently supported by autodiff.
- Baselines and breadth (t9vC): Rebuttal adds Diffuser/Hierarchical Diffuser results and clarifies DP lineage; also presents UWM comparisons and hybridization benefits.
- Terminology “Act to See” (fkRG, uXWV): Authors will clarify “see” as internal representational shifts rather than acquiring new frames to avoid misleading interpretations.
- Benchmark saturation (HxxU): Acknowledged; dynamic tasks and real-world results better differentiate methods; authors propose expanding challenging benchmarks.

Overall, this is a technically solid, well-motivated, and practically relevant contribution that advances diffusion-based policies by establishing an effective perception–action interplay. The paper offers a clear and novel mechanism for within-rollout perception–action coupling with broad empirical validation and compelling real-robot results. Reviewers are overall positive (1 accept, 4 weak accepts), with initial concerns largely addressed in rebuttal. The method introduces a new, practical, and general augmentation into imitation learning and diffusion-policy research. Nevertheless, please incorporate all suggestions by reviewers into the camera-ready version, including clarify intuition and positioning, objective discussion, baselines and scope, practical guidance, real-robot safety and failure analysis.
Decision: Accept (poster)