Responsive Noise-Relaying Diffusion Policy: Responsive and Efficient Visuomotor Control
Authors: Zhuoqun Chen
Xiu Yuan
Tongzhou Mu
Hao Su
Venue: Accepted by TMLR ()

Abstract: Imitation learning is an efficient method for teaching robots a variety of tasks. Diffusion Policy, which uses a conditional denoising diffusion process to generate actions, has demonstrated superior performance, particularly in learning from multi-modal demonstrates. However, it relies on executing multiple actions predicted from the same inference step to retain performance and prevent mode bouncing, which limits its responsiveness, as actions are not conditioned on the most recent observations. To address this, we introduce Responsive Noise-Relaying Diffusion Policy (RNR-DP), which maintains a noise-relaying buffer with progressively increasing noise levels and employs a sequential denoising mechanism that generates immediate, noise-free actions at the head of the sequence, while appending noisy actions at the tail. This ensures that actions are responsive and conditioned on the latest observations, while maintaining motion consistency through the noise-relaying buffer. This design enables the handling of tasks requiring responsive control, and accelerates action generation by reusing denoising steps. Experiments on response-sensitive tasks demonstrate that, compared to Diffusion Policy, ours achieves 18% improvement in success rate. Further evaluation on regular tasks demonstrates that RNR-DP also exceeds the best acceleration method (DDIM) by 6.9% in success rate, highlighting its computational efficiency advantage in scenarios where responsiveness is less critical.

Published in Transactions on Machine Learning Research (05/2025)
Responsive Noise-Relaying Diffusion Policy: Responsive and Efficient Visuomotor Control
Zhuoqun Chen UC San Diego
shc0570ucsd edu
Xiu Yuan" UC San Diego
a yluyannnhccdd.com edu
Tongzhou Mu
t3mu@uced.ccm edu
UC San Diego
Hao Su UC San Diego
haosu@ucsd edu
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id- LLWJRR6gaI
Abstract
Imitation learning an efficient mothod for teaching robots a variety of tasks. Diffusion Policy, which uses conditional denoising diffusion process generate actions, has demon- strated superior performance, particularly in learning from multi-modal demonstrates. How- ever, it relies on executing multiple actions predicted from the same inference step to retain performance and prevent mode bouncing, which limits its responsiveness, as actions are not conditioned on the most recent observations. To address this, we introduce Responsive Woise-Relaying Diffusion Policy (RNR-DP), which maintains a noise-rclaying buffer with progressively increasing noise levels and employs a sequential denoising mechanism that generates immediate, noise-free free actions at the head of the sequence, while appending noisy actions at the tail. This ensures that actions are responsive and conditioned on the latest observations, while maintaining motion consistency through the noise-relaying buffer. This design enables the handling of lasks requiring responsive control, and accelerates action generation by reusing denoising steps. Experiments on response-sensitive tasks demonstrate that, compared to Diffusion Policy, ours achieves 18% improvement in success rate. Further evaluation on regular tasks demonstrates that RNR-DP also exceeds the best acceleration method (DDIM) by 6.9% in success rate, highlighting its computational efficiency advan- tage in scenarios where responsiveness is less critical. Our project page is available at https://mmr--d..itthbb
1 Introduction
Imitation learning is powerful approach for training robots to perform complex tasks, including grasping (Johns, 2021; Xie et al., 2020; Stepputtis et al., 2020), legged locomotion (Ratliff et al., 2007; Al-Hafez et al., 2023; Yang et al., 2023b), dexterous manipulation (Qin et al., 2022; Radosavovic et al., 2021), and mobile manipulation (Wong et al., 2022; Du et. al., 2022). Advances in computer vision and natural language processing have led to incrcasingly sophisticated imitation lcarning frameworks, achieving impressive succoss across diverse tasks (Chen et al., 2021; Abramson, 1970; Florence al., 2022; Shafiullah el. al., 2022). A notable breakthrough, Diffusion Policy (Chi et al., 2023), models robot action sequences through condi- tional denoising diffusion process, setting new benchmarks over traditional imitation learning techniques. *These authors contributed equally to this work.

---

Published in Transactions on Machine Learning Research (05/2025)
Consequently, Diffusion Policy has rapidly gained traction and is now widely adopted foundational
framework in both research and real-world applications.
However, Diffusion Policy faces significant limitations. As shown in Chi et al. (2023), its performance heavily depends on having a relatively large action horizon, Ta, which corresponds to executing multiple actions in the environment. The policy widely achieves optimal performance at Ta 8 but suffers substantial degrada- tion when To 1, where only single action is executed per in- ference. We attribute this to the nature of modeling multi-modal data: each inference independently samples actions aligned with a specific mode. Consequently, a larger action horizon is essential to ensure sequence of actions adheres to the same mode, maintaining consistency. Conversely, using Te leads to severe mode bounc- ing, causing significant, performance drops. However, employing a large action horizon (e.g., Ta 8) also introduces drawbacks, most actions are not conditioned on the latest observations, thereby reducing responsiveness and adaptability to environmental changes. Empirically, we observe that Diffusion Policy struggles particularly with tasks requiring responsive control, such as handling dynamic objects.
Response-sensitive Tasks
60%
Success Rate
DP
40%
RNR-DP
20%
State
Visual
Overall
Regular Tasks
60%
DDIM
Success Rate
CP
40%
RNR-DP
20%
State
Visual
Overall
Figure 1: Our RNR-DP consistently delivers responsive and efficient. con-
trol.
To address these issues, we propose Responsive Noise-Relaying Diffusion Policy (RNR-DP). We define ne- sponsiveness as the degree to which the current action leverages the most recent observation. Our approach fundamentally relies on noise-relaying buffer that contains increasing noise levels and implements a se- quential denoising mechanism. After each denoising step, conditioned on the most recent observations, the model exccutes an immodiate noisc-frec action at the buffer's head and appends fully noisy actions at the buller's tail. The noise-relaying bulfer, combined with the sequential denoising mechanism, not only reuses denoising steps from previous outputs aalpoiinllooigggg allowing one denoising step to generate one action buto also ensures active control based the most recent observations and prevents frequent mode bouncing, maintaining action consistency throughout the entire process. We name it Responsive Noise-Relaying Diffusion Policy because it employs a noise-relaying buffer at its core and has the ability to respond quickly and actively to the environment.
We evaluate our approach across a range of benchmarks, focusing on 9 tasks from three well-cstablished datasets: ManiSkil12 (Gu et al., 2023), ManiSkill3 (Tao al., 2024), and Adroit (Rajeswaran al., 2017). Our primary evaluation targets tasks involving dynamic object manipulation that demand responsive control. Empirical results show that RNR-DP significantly outperforms Diffusion Policy, delivering much more responsive control. Additionally, control. The results indicate that, even on these simpler tasks, RNR-DP functions as superior acceleration method compared to popular alternatives such as DDIM (Song et al., 2020) and Consistency Policy (Song et al., 2023; Prasad et al., 2024). Overall, our evaluations systematically demonstrate that RNR-DP provides both highly responsive and efficient control.
extend our evaluation to tasks that do not require responsive
To summarize, our contributions are as follows:
We identify a key limitation of Diffusion Policy: its reliance on a relatively large action horizon Ta. which compromises its responsiveness and adaptability to environmental changes.
We propose Responsive Noise Noise-Relayinn Relaying Diffusion Policy (RNR-DP) which maintains a noise-relaying buffer with progressively increasing noise levels and employs a sequential denoising mechanism.
We conduct extensive experiments on 5 tasks involving dynamic object manipulation and 4 simpler tasks that do not demand responsive control. The results consistently demonstrate that RNR-DP delivers both highly responsive and efficient control.
2

---

Published in Transactions on Machine Learning Research (05/2025)
2 Related Work
Diffusion Model Acceleration Techniques Diffusion models (Ho et al., 2020) have garnered significant attention for their capacity to model complex distributions. However, their iterative sampling processes be computationally expensive due to the large number of diffusion steps required. One approach to address the low inference speed of diffusion models is by reducing the number of denoising steps needed, as seen in works like (Song ct al., 2020) and (Karras et al., 2022). Another linc of rescarch employs distillation-based techniques, which begin with a pretrained teacher model and train a new st student model to take larger steps over the ODE trajectories that the teacher has already learned to map, as demonstrated in (Song et al., 2023), (Liu et al., 2023), and (Salimans & Ho, 2022). Among the most commonly used acceleration techniques in the robotics commmuntty are DDIM (Song et al., 2020) and Consistency Models (Song et al., 2023), which are particularly effective for speeding up the diffusion policy process.
Diffusion Model for Motion Synthesis Diffusion Model, renowned for its strong representational capa- bilitics, has been widely applied to motion synthesis tasks (Shafir et al., 2023). Building on this foundation, Zhang et al. (2024) proposed an innovative framework that incorporates temporally varying denoising and maintains motion buffer comprising progressively noised poses, enabling long-term motion synthesis. In- spired by TEDi, develop noise-relaying buffer with incrementally increasing noise levels and implement sequential denoising mechanism to enhance Diffusion Policy, ensuring more efficient and responsive control.
Diffusion Model as Policy With the success of diffusion models in image synthesis and video generation (Ho et al., 2020), they have become a popular choice as policy backbones in the robotics community. These models are utilized in two main ways: 1) As policies reinforcement learning (RL) methods, including al., 2023; Mao al., 2024), offlnne-toonllnc RL (Ding & Jin, 2023), and online RL (Yang et al., 2023a); 2) As policies in imitation learning (Chi et al., 2023; Reuss et al., 2023). Diffusion Policy belongs to the second category and has demonstrated state-of-the-an performance compared to other imitation learning methods (Shafiullah et al., 2022; Florence et al., 2022; Abramson, 1970). Furthermore, it exhibits significant potential for future rescarch and practical applications (Yuan et al., 2024).
offline RL (Wang et al., 2022; Hanson-Estrute
3 Background
Diffusion Policy (Chi et al., 2023) models control policies using Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al., 2020), which have shown strong performance in generative modeling. In control, Diffusion Policy predicts the future action sequence A, using a noise prediction network &o (A(( O, k). where A, A, + e is a perturbed version of the clean action sequence A, with added Gaussian noise at noise level k. The model learns to estimate and remove noise by minimizing the mean squared error (MSE) loss:
C = C=| O,,k) 111
During inference, given an observation O,, the trained network iteratively refines the action sequence over denoising steps following:
A (1=1) a (A!!) -A
(k), Ac, (
where the initial action sequence A, is sampled from NN 1), and e~ 00 represents Gaussian noise. The predefined noise schedule functions a, 7, and a are part of the DDPM scheduler (Ho et al., 2020). Once denoised to (()) the agent executes the first Tg future steps after time t.
3

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: We compare the performance of Diffusion Policy under different action horizon Ta with multi- data and RL data.
Traj Num Ta=1
Ta=2
Ta=4
Ta=8
Demo Type
Multi-Modal
1000
0.78
0.93
0.95
0.96
RL (Single-Modal)
500
0.62
0.59
0.58
0.55
4 Limitations of Diffusion Policy
This section explores the key limitations of Diffusion Policy in detail. summary, Diffusion Policy relies on a relatively large action horizon, Te. to ensure sequence of actions adheres to the same mode and avoid frequent mode bouncing. However, using large action horizon results in most actions being unconditioned the latest observations, thereby limiting the policy's responsiveness to environmental changes. Section 4.1 delves into why Diffusion Policy requires a large action horizon, while Section 4.2 examines how this impacts responsiveness.
4.1 Why Diffusion Policy Needs Large Action Horizon?
As shown in Chi et al. (2023), Diffusion Policy performs best with relatively large action horizon Ta (e.g., To 8), while T, = 1 siguificantly degrades performance. This is because each action sequence is independently denoised from noise, and in multi-modal settings, Ta = allows actions to switch modes, causing inconsistencies. Executing multiple actions within the same mode is crucial for stable performance. To validate this, we train Diffusion Policy on 500 single-modal demonstrations from an RL agent in the ManiSkil12 StackCube task. Results in Table 1 show performance drop with T₃ 1, even slightly improving over Ta = 8, confirming that mode bouncing in multi-modal settings is the main reason for requiring large action horizon.
4.2 How A Large Action Horizon Limits Responsiveness?
As discussed above, large action horizon Ta is crucial for Diffusion Policy, particularly when modeling multi- modal data. However, an excessively large Ta can hinder responsiveness to rapid environmental changes. Consider dexterous robotic hand tasked with picking up a ball from a surface and transporting it to a goal position. This task demands continuous fine-grained control and rapid adaptation to unforeseen disturbances, such as the ball slipping or shifting unpredictably within the fingers. the policy commits executing T, future actions, may struggle react promptly to subtle variations in grip force contact dynamics. For example, if the ball begins slip, long action sequence could delay corrective actions, making recovery difficult. In contrast, short action horizon (T. 1) allows the policy to continuously refine its grip based on real-time feedback, ensuring stable and controlled relocation. This example highlights that in contact-rich object manipulation tasks, a large action horizon forces the policy to predict complex interactions prematurely, making precise control more challenging. Empirically, we observe that Diffusion Policy struggles with such dynamic, contact-rich manipulation tasks, where real-time adaptability essential, further underscoring the dilemma between morioorizon consistency and responsiveness.
5 Responsive Noise-Relaying Diffusion Policy
To achieve responsive and efficient control, our proposed approach relies on a noise-relaying buffer and the implemented soquential denoising machanism its core (Section 5.1) with additional key design choices (Section 5.2). More details of the implementation including policy architecture and hyperparameters are summarized in Appendix B.
4

---

Published in Transactions on Machine Learning Research (05/2025)
Action Denoising: denoise and execute action
clean action
fully noisy action
Shifting
yprnnal
proceed to next control step
Ok-1
O2
Visual
Visual
Enqueue
conditioning
Denoiser
E
V
obs feature
k
noise levels
Dequeue
past steps in episode
V (0)
future steps in episode
Figure 2: Inference Overview of Responsive Noise-Relaying Diffusion Policy (RNR-DP). The core of RNR-DP is the noise-relaying buffer (Section 5.1) and it has 3 stages during the entire control-loop, as exemplified by the transition between time step f. and time step t+1, (1) The buffer contains noisy actions with increasing noise levels (2) After denoising once, each action in the buffer denoised for one step, clean action at the buffer's head removed and executed (dequeue) (3) The remaining noisy actions are left shifted for one slot and a fully noisy action is appended the buffer's tail (enqueue). The conditioning data is discussed with more details in Section 5.2 and Appendix B.3.
5.1 Noise- Relaying Buffer
(f-1) 2 a contains a sequence of noisy actions with linearly increasing noise levels from S, where is the buffer capacity as well as the total number of noise levels. As shown in Figure after each denoising step, the trained network transforms Qe into Q = {a, (0) A++1.... f a(+5-1), producing a noise-free action a; at the head. This clean action is immediately executed, and a fully noisy action is appended to the buffer's tail. For the next step, the buffer reuses denoising steps from previous outputs, ensuring consistency and avoiding full denoising from scratch. This sequential denoising mechanism conditions clean actions on the latest observations, enabling responsive and long-tery mng-term active control. Pseudocode is provided in Algorithm 1.
The noisc-rclaying buffer Q: = {a, (1)
(f)
(f-2)
(f-1)
5.2 Key Design Choices
Mixture Noise Scheduling We train the denoiser network following the DDPM (Ho et al., 2020) frame- work, and allow each action in A, perturbed by independent noise levels. Given a fixed variance schedule 3 (31 < By), any a, A, can be perturbed by one of the I levels. During training, we use a mixed per-action noise injoction scheme (mirture schedule): with probability Plinear, actions are perturbed by linearly increasing variances (lineur schedule); with probability 1- Plinear, actions perturbed by ran- dom variances from 81 to By (random schedule). The random schedule trains the model to denoise actions independently (*-1) (A) S k < f, while the linear schedule ensures smooth transitions across consecutive actions during inference. Unlike Diffusion Policy, which applies a single variance level to all actions in A, per iteration, our mixture schedule enables diverse and robust training. An illustrative visu- alization is included in Appendix
Laddering Initialization During inference, we use noiso-relaying buffer with noisy action frames, following Algorithm 1. Initially, the buffer contains f fully noisy actions sampled from AO I), i.e., Q
5

---

Published in Transactions on Machine Learning Research (05/2025)
Initialization
noise levels
denoise sufficient times
Figure 3: A visulization of the initialization process. Noise-rclaying buffer contains only fully noisy actions sampled from standard multivariate Gaussian distribution of dimension Ta al each action index before the initialization; the buffer is denoised for f times until the buffer head contains minimal noise level and can be executed with one more model forward call.
{zz IO, 1 To align with training and avoid performance drops, we initialize the buffer by iteratively denoising it ʃ times using the random schedule conditioned on the initial observation Og- This transforms the buffer to follow the linear schedule, ensuring smooth and responsive control. Since this process transitions the buffer from uniform noise to monotonically increasing variances like ladder, we term it laddering initialization, as illustrated in Figure 3. A more detailed illustrative visualization is included in Appendix J.
Noise- Aware Conditioning Unlike Diffusson Policy (Chi et al., 2023), which encodes single diffusion step to one time embedding, our approach uses mixed scheduling and noise-relaying buffer to handle multiple noise levels, requiring awareness of multiple diffusion steps. For each action's 's noise level ky (1 S ky s f), we use an MLP to encode a time embedding, yielding f embeddings. These are appended to the observation features from the encoder Eots- This allows the bulfer, during both training and inference, to decode actions at each noise level using up-to-date observations, enabling more dynamic and consistent behaviors. See Appendix B.3 for more details.
6 Experiments
The goal of our experimental evaluation to study the following questions:
1. Can Responsive Yoise-Relaying Diffusion Policy outperform Diffusion Policy by delivering more responsive control? (Section 6.3)?
Can Responsive Noise-Relaying Diffusson Policy function as a superior acceleration method compared to commonly used alternatives simpler tasks that do not require responsive control (Section 6.4)?
What are the effects of the components introduced by Responsive Noisc-Relaying Diffussoo Policy (Section 6.5)?
6.1 Experimental Setup
To validate the responsiveness and elficiency of Responsive Noise-Relaying Diffusion Policy, our experimental setup incorporates variations in the following dimensions:
Task Types: Stationary robot arm manipulation, mobile manipulation, dual-arm coordination, dexterous hand manipulation, articulated object manipulation, and high-precision tasks.
Demo Sources: Teleoperation, Task and Motion Planning, RL, and Model Predictivo Control.
Observation Modalities: State observation (low-dim) and visual observation (high-dim).

---

Published in Transactions on Machine Learning Research (05/2025)
Response-sensee
Regular
PushT
RollBall
PushChair
Relocate
StackCube
TurnFaucet
Pen
Hammer
Figure 4: Task Visualization in Simulation. ManiSkill tasks including PushT, RollBall, PushChair StackCube, Furneancet; Adroit tasks including Relocate, Door, Pen, Hammer. See Appendix A.1 for more properties about each task.
6.1.1 Task Descriptions
Our experiments are conducted on tasks from 3 benchmarks: ManiSkil12 (robotic manipulation; 3 tasks), ManiSkill8 (robotic manipulation; 2 tasks) and Adroit (dexterous manipulation; 4 tasks). These tasks are separated into groups to validate the responsiveness and efficiency of Responsive Noise-Relayine Diffusion Policy. A detailed discussion of task groupings and classification criteria be found in Appendix D.
Response-sensitive Group: Tasks Involving Dynamic Object Manipulation Requiring Respon- sive Control We consider 5 challenging tasks involving contact-rich, dynamic object manipulation to assess the responsiveness of RNR-DP. PushT requires using a stick to push a T-shaped block to a target location and orientation. RollBall involves pushing and rolling a ball to a randomized goal region. PushChair tests bimanual manipulation of articulated objects with variations. Adroit Relocate involves picking up a ball and moving it goal position, while Adroit Door requires unlocking and opening door. See Figure for task visualizations, and more details are included in Appendix A.1.
Regular Group: Simpler Tasks that Do Not Require Responsive Control We consider the rest 4 simpler tasks that do not require responsive control to validate the efficiency of RNR-DP. StackCube requires picking up cube and stack it onto another cube. TurnFaucet uses a stationary arm to turn on faucets of various geometries and topology. Adroit Pen repositions the blue pen to match the orientation of the green target. Adroit Hammer picks up hammer and drives a nail into board. More details are included in Appendix A.1.
6.2
Baselines
We compare our Responsive Noise-Relaying Diffusion Policy against a set of strong baselines related Diffusion Policy.
Diffusion Policy (DDPM) is the original setting from Chi et al. (2023), which utilizes conditional Denoising Diffusion Probalistic Model with discrete time scheduling to generate actions. We use 100 DDPM denoising steps in our experiments.
Diffusion Policy (EDM) is introduced in Prasad et al. (2024) to train a teacher model and employs the EDM (Karras et al., 2022) framework with continuous time scheduling. This model takes the current position Lt, time i, and conditioning o as inputs along a PFODE, and is used estimate the derivative of the PFODE's trajectory. We utillie the EDM model with Heun's 's second-order solver taking 80 denoising steps, which requires neural network evaluations per discretized step in the ODE, resulting in total of 159 neural function evaluations (NFEs).
Diffusion Policy (DDIM) is used in Chi et al. (2023) to accelerate Diffusson Policy with Diffusion Denoising Implicit Model framework (Song et al., 2020). We test. 1, 2, 4, and 8 DDIM denoising steps.
Consistency Policy (CP) introduced Prasad al. (2024), which employs the Consistency Trajectory Model (Kim et al., 2023) distill the knowlodge from Teacher Model (EDM). We evaluate both the 1-step Consistency Policy and the 8-step chaining Consistency Policy.

---

Published in Transactions on Machine Learning Research (05/2025)
Evaluation on tasks requiring responsive control (Response-sensitive Group) from ManiSkill and Adroit benchmarks (State Observations). We report average success rate (1) of the best checkpoint for 1000 episodes across 10 random seeds. Results that are stalistically belter highlighted in bold. Our results are highlighted in light-blue cells.
Table 2:
Relocate
Door
PushChair RollBall PushT
Method
w/ g w/o g
DP
0.422
0.558
0.495
0.635
0.083
0.470
RNR-DP
0.585
0.629
0.547
0.694
0.121
0.491
Table 3: Evaluation on tasks (Ressovse-sensitiive Group) from ManiSkill and Adroit benchmarks (Visual Observations). We report values under the same settings as in Table 2. Tasks in which none of the methods achieve reasonable success rate under visual observations are omitted.
Door (Adroit) RollBall (MS3) PushT (MS3)
Method
DP
0.079
0.080
0.349
RNR-DP
0.122
0.131
0.379
Strcaming Diffusion Policy (SDP) is recent advancement over Diffusion Policy that stays close to our approach. See Appendix C.2 for detailed discussion.
6.3 Results & Analysis on Responsive Control
First, we provide comprehensive evaluation and validate the responsiveness of RNR-DP on 5 tasks (Response- sensitive Group) involving contact-rich dynamic object manipulation, a primary focus of our policy. Figure I shows an overview of performance improvement. As shown in Table 2 and Table 3, RNR-DP consistently outperforms Diffusion Policy across all tested scenarios in both state and visual experiments. Specifically, in state experiments, RNR-DP achieves 15.1% improvement over Diffusion Policy, while in visual experiments, it demonstrates 24.9% improvement. Overall, RNR-DP surpasses Diffusion Policy by 18.0%. Notably, in the Relocate task, RNR-DP achieves significant performance boost of 38.6% over Diffusion Policy.
We attribute these significant performance gains to the design of our noise-relaying buffer and sequential denoising mechanism, which offer two key advantages. The noise-relaying buffer ensures that the denoising of each action remains consistent, allowing actions to follow the same mode. This effectively climinates frequent mode bouncing, enabling RNR-DP to support single-action rollouts (Ta = 1). Furthermore, the dingle-action rollout in RNR-DP ensures that all actions are conditioned on the latest observations, resulting in significantly more responsive control to environmental changes compared to Diffusion Policy.
6.4 Results & Analysis on Efficient Control
In this section, conduct extended experiments on 4 simpler tasks (Regular Group) that do not require responsive control to evaluate the efficiency of RNR-DP. An overview of the performance improvements is provided in Figure 1. Section 6.4.1 introduces a metric designed for fair efficiency comparisons, while Section 6.4.2 provides detailed analysis of the empirical results compared to commonly used acceleration methods.
6.4.1 Neural Function Evaluations per Action (NFEs/a)
Adopting the settings from Chi et al. (2023), Diffusion Policy and related methods utilize a relatively large action horizon (Ta) achieve better performance, whereas RNR-DP employs a single-action rollout (T, = 1) each inference. To facilitate a fair comparison of efficiency, we introduce a new metric, Neural Function Evaluations per Action (NFEs/a), defined in Equation (1).
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Evaluation on simpler tasks (Regular Group) not requiring responsive control from ManiSkill and Adroit benchmarks (State Obsorvations). We report average success rate (1) and overall averago success rale of all tasks (t). Particularly, DDMMs with NFEs/a = 1, CPs with NFEs/a are highlighted in light-green, light-red cells respectively. Our results highlighted in light-blue cells.
StackCube
Faunet
Pen
Hammer
Avg. SR of tasks
Method Steps (S) NFEs/a
w/ g w/o g
DDPM
100
12.5
0.960
0.495
0.595
0.508
0.120
0.536
0,125
0.000
0.000
0.000
0.110
0.000
0.000
0.25
0.214
0.433
0.594
0.190
0.000
0.286
DDIM
4
0.5
0.959
0,482
0.612
0.476
0.000
0.506
0.964
0.477
0.614
0.483
0.000
0.508
EDM
80
0.955
0.449
0.575
0.517
0.120
0.523
0.125
0.214
0.051
0.028
0.326
0.000
0.124
CP
0.680
0.112
0.299
0.483
0.041
0.323
RNR-DP
0.935
0.531
0.594
0.487
0.139
0.537
NFEs To
NFEs/a-
(1)
6.4.2 Empirical Comparison with Commonly Used Acceleration Methods
To evaluate the efficiency of Responsive Noise-Relaying Diffusion Policy (RNR-DP), we compare it against common acceleration methods, including DDIM (Chi al., 2023) and Consistency Policy (Prasad et al., 2024), on simpler tasks that do not require responsive control (Regular Group). As shown in Table 4 and Appendix C.1, RNR-DP consistently achieves the highest overall performance among all DDIM and Con- sistency Policy variants. For fair comparison, we specifically evaluate RNR-DP against 8-step DDIM (NFEs/a = 1) and 8-step-chaining Consistency Policy (NFEs/a = 1). In state-based experiments, RNR-DP outperforms 8-step DDIM by 5.7% and 8-step-chaining Consistency Policy by 66.2%. In vision-based experi- ments, it surpasses S-step DDIM by 8.5% and 8-step-chaining Consistency Policy by 3.4%, yielding an overall improvement of 6.9% over 8-step DDIM and 32.0% over 8-step-chaining Consistency Policy. Additionally, both DDIM and Consistency Policy struggle in certain tasks. For instance, 8-step DDIM achieves 0% success rate on Adroit Hammer, while Consistency Policy performs poorly StackCube and TurnFaucet. In contrast, RNR-DP demonstrates consistent and robust performance across all tasks. Notably, RNR-DP achieves an average success rate comparablc to Diffusion Policy across all state and visual experiments while being 12.5 times faster. These results confirm that even on simpler tasks, RNR-DP serves as a superior acceleration method, enabling efficient control.
We attribute this robust performance to the design of the noise-relaying buffer and soquential denoising mechanism. By reusing denoising steps from previous outputs, RNR-DP requires only one denoising step to generate single action, while ensuring all actions undergo sufficient denoising to maintain high action quality. In contrast, DDIM and Consistency Policy significantly reduce the number of denoising steps per action, leading to pronounced performance drops.
6.5 Ablation Study
We conduct various ablations to provide further insights on the effects of components of RNR-DP.
Noise Scheduling Scheme We conduct comprchensive study to evaluate how different noise scheduling schemes impact the performance of the Responsive Noise-Relaying Diffusion Policy. As shown in Table 5, the results demonstrate that relying solely on either a linear schedule or a random schedule significantly roduces task success rates. This highlights the importance of integrating mizture scheduling to fully exploit the potential of the noise-relaying buffer design. By combining the two schedules, our model effectively utilizes different noise levels, enhancing the robust and adaptability of action generation.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Table Ablation study on noise scheduling scheme during training (Section 5.2). Numbers represent aver- age success rates (t). Numbers in parenthesis indicate the performance drop after removing key component of our Responsive Noise-Relaying Diffusion Policy.
Relocate (Adroit)
Door (Adroit)
Ablation
Linear
0.323 (-26.2%)
0.404 (-22.5%)
Random
0.389 (-19.6%)
0.360 (-26.9%)
Mixture (Ours)
0.585
0.629
Table 6:
Ablation study on noise-relaying buffer initialization (Section 5.2).
Relocate (Adroit) Door (Adroit)
Ablation
Pure noise
0.522 (-6.3%)
0.516 (-11.3%)
Laddering (Ours)
0.585
0.629
Table 7:
Ablation study on model prediction type (Section 5.2).
Relocate (Adroit) Door (Adroit)
Ablation
Action
0.154 (-43.1%)
0.374 (-25.5%)
Noise (Ours)
0.585
0.629
Yoise-Relaying Buffer Initialization We conduct ablation study to evaluate the effectiveness of our initialization scheme. Pure noïse directly uses a fully noisy buffer as the initialization for subscquent operations. As shown in Table 6, laddering inilialization achieves significant performance gains compared to pure noise. This result highlights the critical role our initialization scheme in enabling strong and robust performance.
Noise-Relaying Buffer Capacity We also evaluate the impact of the noise-relaying buffer capacity to the performance, the only tuned hyperparameter in our approach. As shown in Figure 5, the Adroit Relocate task, RNR-DP achieves peak performance of 58.5% at buffer capacity of 84 and maintains strong performance within the range of 56 to 92, demonstrating a wide tolerance. Performance de- clines when the buffer capacity is too small (c.g., 48) or too large (c.g., 100). Notably, Diffusion Policy achieves only 42.2%, while RNR-DP with buffer capacities between 48 and 100 consistently outperforms it.
65%
40 50
60
80
90
100
110
Figure Noise-relaying buffer ca-
pacity v.s. average success rate.
This highlights that for single task, the noise-relaying buffer offers robust performance over broad range and is not difficult to tune. Empirically, starting with a value between 56 and 84 and making adjustments suffices most cases.
Model Prediction Type In addition, we investigate the impact of the model's prediction type. As shown in Table 7, models predicting the added noise component outperform those directly predicting the action sequence. This finding aligns with the common practice of using noise prediction in diffusion models within the vision domain (Ho al., 2020).
7 Conclusion and Future Work
In this paper, we identify the key limitation of Diffusion Policy: its reliance on relatively large action horizon Ta compromises its responsiveness and adaptability to environment. changes. To address these
10

---

Published in Transactions on Machine Learning Research (05/2025)
issues, propose Responsive Noise-Relaying Diffusion Policy which maintains noise-relaying buffer with progressively increasing noise levels and employs a sequential denoising mechanism. Our method provides more responsive control than Diffusion Policy on 5 tasks involving dynamic object manipulation, and delivers more efficient control than commonly used acceleration methods on 4 simpler tasks.
Limitations. limitation of this work is the lack of real-robot evalutions. We discuss our method's potential advantages in real-robot deployment in Appendix E. We will conduct complex real world deployment for future rescarch.
11

---

Published in Transactions on Machine Learning Research (05/2025)
References
Norman Abramson. The aloha system: Another alternative for computer communications, In Proceedings of the November 17-19, 1970, fall joint computer conference, pp. 281-285, 1970.
Firas Al-Hafcz, Guoping Zhao, Jan Peters, and Davide Tateo. Locomujoco: comprehensive imitation learning benchmark for locomotion. arXiv preprinl arXiu:3311.02496 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084 15097, 2021.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, pp. 027886449441233668, 2023.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint Xiv:2990. 16984, 2023.
Yuqing Du, Daniel Ho, Alex Alemi, Eric Jang, and Mohi Khansari. Bayesian imitation learning for end- to-end mobile manipulation. In International Conference on Machine Learning, pp. 5531 5546. PMLR, 2022.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pp. 158-168. PMLR, 2022.
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
Philippe Hansen-Estrudh Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Sigmund H. Heeg, Yilun Du, and Olav Egeland. Streaming diffusion policy: Fast policy synthesis with variable noise diffusion models, 2024. URL https://arxxw.org/aaat wrgg 04806.
Edward Johns. Coarse-to-fin imitation learning: Robot manipulation from a single demonstration. In 2021
IEEE international conference on robotics and automation (ICRA), pp. 4613-4619. IEEE, 2021.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. ar preprint orxv::2310 02279, 2023.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality buffe text-to-image generation. In The Twelfth International Conference on Learning Represen- lations, 2023.
Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning. Xi preprint 00X2:22407.20109, 2024.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Aduances in Newral Information Processing Systems, 36:62244 62269, 2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent. Dumoulin, and Aaron Courville. Film: Visual rea- soning with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Aaditya Prasad, Kevin Lin, Jimmy Wu, Linqi Zhou, and Jeannette Bohg. Consistency policy: Accelerated visuomotor policies via consistency distillation. arXiv preprint wxXi:2445.075003 2024.
Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pp. 570-587. Springer, 2022.
Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. State-only imitation learning for dexterous manipulation. In 2021 (REE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7865 7871. IEEE, 2021.
Aravind Rajoswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXio preprint or orxiv:1700 net:17099000077 2017.
Nathan Ratliff, J Andrew Bagnell, and Siddhartha S Srinivasa. Imitation learning for locomotion and manipulation. In 2007 7th SGEE-RAS international conference on humanoid robots, pp. 192-397. IEEE, 2007.
Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. arxic preprint urXiv:2304.0255 2023.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusson models. arXiv preprint wwXiv:2202.00122 2022.
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusson as generative prior. arxvv preprint orXiv:2308.01418 2023.
Nur Muhammaa Shafiullah, Zichen Cui, Ariuntuya Arty Allanzaya, and Lerrel Pinto. Behavior transformers: Cloning modes with one stone. Advances in neural information processing systems, 35:22955- 22222992222988,, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. arXiv:2010.c 02502, 2020.
Denoising diffusion implicit models. arXio preprinl
Yang Song, Prafulla Dhariwal, Mark Chen, and Nyy Sutskever. Consistency models. ar preprinl arXiv:2303.01469, 2023.
Simon Stepputtis, Joseph Campbell, Mariano Phielipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Landuagge-onnditoone imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139-13150, 2020.
Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskill3; Gpu parallelized robotics simulation and rendering for generalizable embodied ai, 2024. URL https: org/ tabss/411... 04425.0042....
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint orXtv:2208.0614 06193, 2022.
Josiah Wong, Albert Tung, Andrey Kurenkov, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, and Roberto Martin-Martin Error-aware imitation learning from telcoperation data for mobile manipulation. In Conference Robot Learning, pp. 1367-1378. PMLRR 2022.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Fan Xie, Alexander Chowdhury, M De Paolis Kaluza, Linfeng Zhao, Lawson Wong, and Rose Yu. Deep imitation learning for bimanual robotic manipulation. Advances in neural information processing systems. 33:2327 2337, 2020.
Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprinl ar Xiv:2300. 13122, 2023a.
Ruihan Yang, Zhuoqun Chen, Jianhan Ma, Chongyi Zheng, Yiyu Chen, Quan Nguyen, and Xiaolong Wang. Generalized animal imitator: Agile locomotion with versatile motion prior. aiv preprinl arXiv:3310001408, 2023b.
Xiu Yuan, Tongzhou Mu, Stone Tao, Yunhao Fang, Mengke Zhang, and Hao Su. Policy decorator: Model- agnostic online refinement for large policy model. arXiv preprint arXiv:24 13630, 2024.
Zihan Zhang, Richard Liu, Kfrr Aberman, and Rana Hanocka. Tedi: Temporally--ntanglle diffusion for long-term motion synthesis. In SIGGRAPH, Technical Papers, 2024. doi: 10.1145 141144//6611119.6555555
14

---

Published in Transactions on Machine Learning Research (05/2025)
A Further Details on the Experimental Setup
A.1 Task Descriptions
We consider total of 9 continuous control tasks from 3 benchmarks: ManiSkill2 (Gu el al., 2023), ManiSkill3 (Tao et al., 2024), and Adroit (Rajeswaran et al., 2017). This section provides detailed task descriptions overall information, task difficulty, object sets, state space, and action space. Some task details are listed in Table 8.
Table 8: We consider contimuous tasks from 3 benchmarks. We list important task details below.
Task
State Obs Dim Cstate
Act Dim C.
Max Episode Step
ManiSkill3: PushT
31
100
ManiSkill3: RollBall
44
4
80
ManiSkill2: StackCube
55
4
200
ManiSkil12: TurnPaucet
43
7
200
ManiSkil12: PushChair
131
20
200
Adroit: Door
39
28
300
Adroit: Pen
46
24
200
Adroit: Hammer
46
26
400
Adroit: Relocate
39
30
400
A.1.1
ManiSkill2 Tasks
StackCube
Overall Description: Pick up red cube and place it onto green one. See Figure 6 for episode visualization.
Task Difficulty: This task requires precise control. The gripper needs to firmly grasp the red eube and accurately place it onto the green one.
Object Variations: No object variations
Action Space: Delta position of the end-effector and joint positions of the gripper.
State Observation Space: Proprioceptive robot state information, such as joint angles and velocities of the robot arm, and task-specific goal information
Visual Observation Space: one 64x64 RGBD image from a base camera and one 64x64 RGBD image from hand camera.
Figure 6: StackCube Episode Visualization.
TurnFaucet
Overall Description: Turn faucet by rolating its handle.
Task Difficulty:
This task needs to handle object variations. See Figure 7 for episode visualization.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Object Variations: We have source environment containing 10 faucets, and the dataset is collected in the source environment. w/o means the agent directly interacts with the source environment online; w/ g means the agent interacts with the target environment online, which contains 4 novel faucets.
Action Space: Delta pose of the end-effector and joint positions of the gripper.
State Observation Space: Proprioceptive robot state information, such as joint angles and velocities of the robot arm, the mobile base, and task-speciiic goal information.
Figure 7: TurnFaucet Episode Visualization.
PushChair
Overall Description: A dual-arm mobile robot needs push swivel chair to target location the ground (indicated by red hemisphere) and prevent it from falling over. The friction and damping parameters for the chair joints are randomized. See Figure 8 for episode visualization.
Task Difficulty: This task needs to handle object variations.
Object Variations: We have source environment containing 5 chairs, and the dataset is collected in the source environment. w/o mcans the agent directly interacts with the source environment online; w/ g means the agent interacts with the target environment online, which contains 3 novel faucets.
State Observation Space: Proprioceptive robot state information, such joint angles and velocities of the robot arm, task-specific goal information.
Visual Observation Space: three 50x125 RGBD images from three cameras 120 apart from each other mounted on the robot.
Figure 8: PushChair Episode Visualization.
A.1.2 ManiSkill3 Tasks
PushT
Overall Description: It is a simulated version of the real-world push-T task from Diffusion Policy: https:// wwtptssion-ooiccccc.oooo
this task, the robot needs to precisely push the T- shaped block into the target region, and move the end-effector to the end-zone which terminates the episodes. The success condition that the T block covers 90% of the 2D goal T's area. See Figure for episode visualization.
The task involves manipulating a dynamic T-shaped object, which introduces non- linear dynamics, friction, and contact forces.
Task Difficulty:
Object Variations: No object variations.
16

---

The T-Block is pushed from sampled initial configuration to the Published in Transactions on Machine Learning Research (05/2025)
Action Space: Delta pose of the end-effector and joint positions of the gripper.
State Observation Space: Proprioceptive robot state information, such as joint angles and velocities of the robot arm, and task-specific goal information.
Visual Observation Space: one 64x64 RGBD image from a base camera.
Figure 9: PushT Episode Visualization.
goal area.
RollBall
Overall Description: A task where the objective to push and roll ball to goal region at the other end of the table. The success condition is that The ball's xy position is within goal radius (default 0.1) of the target's xy position by euclidean distance. See Figure 10 for episode visualization.
Task Difficulty: The task involves manipulating dynamic ball, which introducees non-linest dynam- ics, friction, and contact forces.
Object Variations: No object variations.
Action Space: Delta position of the end-effector and joint positions of the gripper.
State Observation Space: Proprioceptive robot state information, such as joint angles and velocities of the robot arm, and lask-specific goal information.
Visual Observation Space: one 64x64 RGBD image from base camera.
Figure 10: Rollball Episode Visualization. The blue ball is pushed and rolled from sampled initial configu- ration to the target red circle.
A.1.3 Adroit Tasks
Adroit Door
The environment is based on the Adroit manipulation platform, 28 degree of freedom system which consists of 24 degrees freedom Shadow Hand and a 4 degree freedom arm. The task be completed consists on undoing the latch and swing the door open. See Figure for episode visualization.
Overall Description:
Task Difficulty: The latch has significant dry friction and biass torque that forces the door to stay closed. No information about the latoch is explicitly provided. The position of the door is randomized.
Object Variations: No object variations.
Action Space: Absolute angular positions of the Adoit hand joints.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Figure 11: Door Episode Visualization.
State Observation Space: The angular position of the finger joints, the pose of the palm of the hand, as well as state of the latch and door.
Visual Observation Space: one 128x128 RGB image from a third-person view camera.
Adroit Pen
Overall Description:
The environment is based on the Adroit manipulation platform, 28 degree of freedom system which consists of 24 degrees of freedom Shadow Hand and a 4 degree of freedom arm. The task to be completed consists on repositioning the blue pen to match the orientation of the green larget. See Figure 12 for episode visualization.
Task Difficulty:
The target also randomized to cover configurations.
Object Variations: No object variations.
Action Space: Absolute angular positions of the Adroit hand joints.
State Observation Space: The angular position of the finger joints, the pose of the palm of the hand, as well as the pose of the real pen and target goal.
Visual Observation Space: one 128x128 RGB image from a third-person view camera.
Figure 12: Pen Episode Visualization.
Adroit Hammer
Overall Description: The environment is based on the Adroit manipulation platform, 28 degree of freedom ShadowHand and a 4 degree of freedom arm. The task to be completed consists on picking up a hammer with and drive a nail into a board. See Figure 13 for episode visualization.
Task Difficulty:
The nail position is randomized and has dry friction capable of absorbing up to 15N
force.
Object Variations:
No object variation.
Action Space: Absolute angular positions of the Adroit hand joints.
State Observation Space: The angular position of the finger joints, the pose of the palm of the hand, the pose of the hammer and nail, and external forces on the nail.
Visual Observation Space: one 128x128 RGB image from a third-person view camera.
Adroit Relocate
18

---

Published in Transactions on Machine Learning Research (05/2025)
Figure 13: Hammer Episode Visualization.
Overall Description: The environment is based on the Adroit manipulation platform, a 30 degree of freedom system which consists of 24 degrees freedom ShadowHand and a 6 degree freedom arm. The task be completed consists on moving the blue ball the green target. See Figure 14 for episode visualization.
Task Difficulty:
The positions of the ball and target randomized the entire workspace.
Object Variations: No object variations.
Action Space: Absolute angular positions of the Adroit hand joints.
State Observation Space: The angular position of the finger joints, the pose of the palm of the hand, as well as kinematic information about the ball and target.
Visual Observation Space: one 128x128 RGB image from a third-person view camera.
Figure 14: Relocate Episode Visualization.
A.2 Demonstrations
This subsection provides the details of demonstrations used in our experiments. See Table 9. ManiSkil2 and ManiSkill3 demonstrations are provided Gu al. (2023) and Tao et al. (2024), and Adroit demonstrations are provided in Rajeswaran et al. (2017).
Table 9: Demonstration sources, numbers and generation methods.
Task
Traj Num for Training
Generation Method
ManiSkil13:
PushT
1000
Reinforcement Learning
ManiSkill3:: RollBall
1000
Reinforcement Learning
ManiSkill2: StackCube
1000
Task & Motion Planning
ManiSkill2: TurnFaucet
1000
Model Predictive Control
ManiSkil12: PushChair
1000
Reinforcement Learning
Adroit: Door
25
Human Demonstration
Adroit: Pen
25
Human Demonstration
Adroit: Hammer
25
Human Demonstration
Adroit: Relocate
25
Human Demonstration
B Implementation Details
B.1 Noise-Relayine Diffusion Policy Inference
We summarize the inference pseudo-code of our RNR-DP in Algorithm
19

---

Published in Transactions on Machine Learning Research (05/2025)
Algorithm Noise-relaying Dilfusion Policy Inference
1: Require: denoising model, EØ: obsorvation, O₃; noise-relaying buffer, Q; buffer capacity f:
while task exccution do
3: Q f Eo (Q: O,, (........ .f)
:
b Eo is trained using f noise levels
4:
f Qc-pop(O)
D a clean action (fully denoised) D z is random noisy action sampled from (0, I) executes a (0) and envrionment updates observation update timestep for the next iteration
5:
Q: f Qt.push(2)
O, env.step(a," (
7:
+++1
B.2 Noise-Reiie Deise-Relaying Diffusion Policy Training
We summarize the training pseudo-code of RNR-DP in Algorithm 2.
Algorithm 2 Responsive Noise- Relaying Diffusion Policy Training
1: Require: demonstration dataset, D ((O,, A denoising model, Eg; number diffusion steps, f
2: repeat
3: Sample (O,A) ~ DD
Sample p Unif(0, 1); Sample noise e~ (~(00,)) and reshape to RC.x
if Plinear
6:
k (h = 1, ,,, =
lincar schedule
7: else
8:
k = (ki Unif(11,... .}),,.. kk, (uif(((,...;))
random schedule
for all a, E A indexed by frame index do
10:
å / āi + VI-ai,e
perturbe cach ay independently
11:
= (Āg, 1
12: Take gradient descent step to update 0 on
13:
Volle (A;
noise-aware conditioning
14:
until converged
B.3 Policy Architecture
We build our RNR-DP on top of the UNet-based architecture Diffusion Policy (Chi al., 2023). The model includes downsampling modules and upsampling modules with each module containing residual blocks. The residual block consists of 1D temporal convolutions (Convld), group normalizations (GN), and Mish activation layers. The encoded noise-aware conditioning data (Section 5.2) is fused into each residual block through the FiLM transformation (Perez et al., 2018). The raw conditioning data is of shape R CCmntt for stale policies and of shape x/ CoeaaaaaCC for visual policies. See Figure 15 for the visualization of a visual policy. We follow the UNet denoiser design for the observation that transformer- basod policies are more sensitive to hyperparameters and often require more tuning (Chi al., 2023). The choice of policy architecture is orthogonal to our method and we believe our design would also improve this policy class.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Embedding
Noise-aware Conditioning
UNet
Conv1d
kj PosEnc
Down
GN
Mish
Down
kj
MLP
MLP
FiLM
Convid
Up
GN
kg
Up
Mish
Figure 15:
The detailed policy architecture for our RNR-DP. We only extract visual features for visual
policies.
B.4
Important Hyperparameters
B.4.1 Key Hyperparameters of RNR-DP
We summarize the key hyperparameters of RNR-DP in Table 10. The observation horizon To and noise- relaying buffer capacity f for each task is listed in Table 11. The number of trainable parameters for each task is listed in Table 12.
Table 10: We list the key hyperparameters of RNR-DP used in our experiments.
Hyperparameter
Value
RNR-DP Noise Scheduling Scheme
Mixture Sampling Prinear)) Plinear 0.4 Noise
RNR-DP Model Prediction Type
Diffusion Step Embedding Dimension
64
UNet Downsampling Dimensions
[64, 128, 256]
Optimizer
Adam
Weight Decay
1e-6
Learning Rate
le-4
Learning Rate Scheduler
Cosine
EMA Model Update
0.9999
Online Evaluation Episodes
1000
B.4.2
Key Hyperparameters of Diffusion Policy
We summarize the key hyperparameters of Diffusion Policy in Table 13. The observation horizon Tap action executation horizon T. and action prediction horizon Tp for each task are listed in Table 14.
B.5 Training Details
We train our models and basclines with cluster assigned GPUs (NVIDIA 2080Ti & A10). We use AdamW optimizer with an initial learning rate of le-4, applying 500 warmup steps followed by cosine decay. We use batch size 1024 for state policies and 256 for visual policies for both ManiSkill and Adroit benchmarks. We evaluate DP, CP and RNR-DP model checkpoints using EMA weights every 10K training iterations for ManiSkill tasks and every 5K for Adroit tasks. DDIMs are evaluated using the best checkpoints of DDPMs in offline manner. CPs trained using the best checkpoints of EDMss.
21

---

Published in Transactions on Machine Learning Research (05/2025)
Table 11: The observation horizon and noise-relaying buffer capacity of our RNR-DP for each task.
Task
Obs To
Capacity ʃ
ManiSkill3:
PushT (Visual)
48
ManiSkill3: RollBall (Visual)
64
ManiSkil12: StackCube (Visual)
2
84
Adroit: Pen (Visual)
2
4
Adroit: Hammer (Visual)
64
Adroit: Door (Visual)
56
ManiSkill3: PushT (State)
32
ManiSkill3: RollBall (State)
4
ManiSkill2: StackCube (State)
2
84
ManiSkil12: TurnFancet w/g (State)
2
64
ManiSkil12: TurnFaucet w/o (State)
72
ManiSkill2: PushChair w/g (State)
56
ManiSkill2: PushChair w/o g (State)
48
Adroit: Door (State)
74
Adroit: Pen (State)
4
Adroit: Hammer (State)
32
Adroit: Relocate (State)
2
84
Table 12: The number of our RNR-DP trainable parameters for each task. Noise-relaying buller size doesn't 't affect the number of trainable parameters for each task.
Task
Trainable Params
ManiSkil13: PushTT (Visual)
11.42M
ManiSkill3: RollBall (Visual)
11.59M
ManiSkill2: StackCube (Visual)
10.85M
Adroit: Pen (Visual)
14.67M
Adroit: Hammer (Visual)
14.72M
Adroit: Door (Visual)
14.41M
ManiSkill3: PushT (State)
4.53M
ManiSkill3: RollBall (State)
4.73M
ManiSkill2: StackCube (State)
4.91M
ManiSkil12: TurnFaucet (State)
4.71M
Adroit: Door (State)
4.66M
Adroit: Pen (State)
4.75M
Adroit: Hammer (State)
4.77M
Adroit: Relocate (State)
4.66M
C Additional Results
C.1 Empirical Comparison with Acceleration Methods on Visual Observations
We summarize the results of vision-based experiments in Table 15. As shown in Table 15, our RNR-DP ourperforms all DDIM variations and CP variations and particularly has an overall improvement over 8-step DDIM by 6.9%, over 8-step-chaining CP by 3.4%.
22

---

Published in Transactions on Machine Learning Research (05/2025)
Table 13: We list the key hyperparameters of Diffusion Policy baseline used in our experiments.
Hyperparameter
Value
Diffusion Step Embedding Dimension
64
UNet Downsampling Dimensions
664, 128, 256]
Optimizer
AdamW
Weight Decay
1e-6
Learning Rate
lc-4
Learning Rate Scheduler
Cosine
FMA Model Update
0.9999
Online Evaluation Episodes
1000
Table 14: We list the observation horizon, action executation horizon and action prediction horizon of Diffusion Policy baseline for each task.
Task
Obs To
Act Exec To
Act Pred Tр
ManiSkill3; PushT (Visual)
2
16
ManiSkill3; RollBall (Visual)
2
4
16
ManiSkill2: StackCube (Visual)
2
8
16
Adroit: Pen (Visual)
2
8
16
Adroit: Hammer (Visual)
2
8
16
Adroit: Door (Visual)
16
ManiSkill3: PushT (State)
16
ManiSkill3: RollBall (State)
2
4
16
ManiSkill2: StackCube (State)
2
8
16
ManiSkil12: TurnFaucet (State)
2
8
16
Adroit: Door (State)
2
8
16
Adroit: Pen (State)
2
16
Adroit: Hammer (State)
Adroit: Relocate (State)
Table 15: Evaluation on simpler tasks (Regular Group) not requiring responsive control from ManiSkill and Adroit benchmarks (Visual Observations). We follow our evaluation metric and report values under the same settings as Table Tasks in which none the mothods achiove a reasonable success rate under visual observations are omitted.
StackCube
Pen Hammer
Avg. SR of tasks
Method
Steps (S)
NFEs/a
DDPM
100
12.5
0.958
0.133
0.123
0.404
0.125
0.000
0.000
0.000
0.000
0.25
0.946
0.042
0.000
0.329
DDIM
0.5
0.947
0.125
0.000
0.357
8
0.946
0.139
0.009
0.365
EDM
80
20
0.930
0.156
0.067
0.384
0.125
0.615
0.127
0.088
0.277
CP
0.910
0.161
0.077
0.383
RNR-DP
0.924
0.154
0.110
0.396
23

---

Published in Transactions on Machine Learning Research (05/2025)
C.2 Comparsion with Streaming Diffusion Policy (SDP)
Streaming Diffusion Policy (SDP) (Hoeg et al., 2024) is recent advancement over Diffusion Policy that stays close to our approach. In this section, we compare our method with SDP in terms of motivation (Appendix C.2.1), methodology (Appendix C.2.2), and empirical results (Appendix C.2.3).
C.2.1 Motivation Comparison
SDP accelerates Diffusion Policy inference by reducing the number of denoising steps required to generate an action sequence. While improving diffusion inference speed is a relevant research topic, its impact in robotics is less compelling, as DDIM and Consistency Policy already provide reasonable speedups with strong performance. In contrast, our method addresses a fundamental limitation of Diffusion Policy-it its lack of responsivenes-whiiee significantly hinders performance rapidly changing environments (e.g., contact-rich dynamic object manipulation). This challenge is far more critical to advancing robotic control. Although our approach also serves effective acceleration method, view this secondary benefit compared to its primary advantage of enabling more responsive control.
C.2.2 Method Comparison
Rollout Method SDP also employs an action buffer structure but partitions the prediction horizon T, into multiple action chunks, ensuring that (1) each chunk maintains the same noise level and (2) noise levels increase across chunks. This chunk-wise design focuses solely reducing denoising steps and accelerating inference. However, it does not address responsiveness and thus retains the limitations of Diffusion Policy. In contrast, our sequential denoising scheme conditions all actions on the latest observations, enabling responsive control while leveraging the noise-relaying buffer to maintain efficiency.
Policy Architecture SDP fuses all time embeddings along the temporal dimension into single embedding, In contrast., our architecture retains multiple time embeddings, ensuring that noisy actions within the noise- relaying buffer can perceive time step changes based on the latest observation features. This design preserves temporal dynamics, allowing each action to adapt to varying time steps, thereby improving responsiveness and consistency in action generation.
C.2.3 Empirical Results Comparison
To comprehensively compare Diffusion Policy, Streaming Diffusion Policy, and our method, we conduct experiments on both response-sensitive tasks and regular tasks to evaluate their responsiveness and effi- ciency, as shown in Table 16 and Table 17. The empirical results indicate that on response-sensitive tasks, Strcaming Diffusion Policy performs similarly to Diffusion Policy, whereas our method achioves significantly more responsive control than both. On regular lasks, both SDP and our method successfully preserve the performance of DP, but our method achieves 6,25 times faster inference.
Table 16: tasks.
We compare Diffusion Policy, Streaming Dilfusion Policy with our method response-sensitive
DP SDP
RNR-DP
Task
Relocate (Adroit)
0.422
0.436
0.585
PushChair w/ (ManiSkil12)
0.495
0.500
0.547
PushChair w/o g (ManiSkill2)
0.635
0.633
0.694
Task Grouping Discussion
In this section, we provide a detailed discussion of the criteria used for grouping tasks into response-sensitive tasks and regular tasks. Appendix D.1 outlines the general rules for task grouping, while Appendix D.2 explains the specific reasoning behind the classification of each task.
24

---

Published in Transactions on Machine Learning Research (05/2025)
Table 17:
We compare Streaming Diffusion Policy with Diffusion Policy and our method on regular tasks.
StackCube TurnFaucet (w/ g) TurnFaucet (w/o g) Hammer
Method
Steps (S)
NFEs/a
DP
100
12.5
0.960
0.495
0.595
0.120
SDP
50
6.25
0.961
0.480
0.615
0.136
RNR-DP
1
1
0.935
0.531
0.594
0.139
D.1
General Rules for Task Grouping
For response-sensitive tasks, inaccurate actions based on outdated observations can easily lead to states not covered by the demonstration dataset. For example, the robot pushes swivel chair too forcefully, it may fall and become unrecoverable. In contrast, regular tasks are more tolerant of inaccuracies from outdated observations. For instance, in the stack cube task, even the gripper doesn't precisely stop above the cube, this state still falls within the distribution.
D.2 Detailed Task Separation Criteria for Each Task
Relocate (Adroit) (Response-Sensitivv) This task requires controlling a high-dimensionall dexterous hand to pick up a ball from a surface and transport it to a goal position. Due to the potential for the ball to slip on the surface or shift unpredictably within the fingers, responsive real-time feedback is essential for successful exccution.
Door (Adroit) (Response-Sensitive) This task requires controlling high-dimensional dexterous hand to apply the appropriate force to turn the handle. Insufficient force fails to open the door, while excessive force causes the hand to slip off. Since each phase relies on the precise execution of the previous one, real-time control is iasential for success.
PushChair (ManiSkill2) (Response-Sensitive) This task requires controlling a mobile bimanual system to push swivel chair goal position. Effective force control is crucial crucial-pushing too hard can cause the chair to topple over, making recovery impossible. Precisely stopping the chair the goal position demands highly adaptable control to make fine adjustments. Without real-time feedback, the chair tends to bounce around the goal position instead of coming to exact stop.
RollBall (ManiSkill3) (Response-Sensittve This task requires precisely rolling a ball to reach the goal position. Successful exccution depends on applying the appropriate force and accurately controlling the rolling trajectory. Even a slight control error can result in missing the target, making highly adaptable and responsive control essential.
PushT (ManiSkilla) (Response-Sensitive) This task involves pushing T-shaped block to goal re- gion. Successful completion requires the block to cover 90% of the goal area, necessitating real-time fine adjustments to its position. Without adaptable control, precisely aligning the T-shaped block to the goal position becomes challenging and prone to failure.
StackCube (ManiSkill2) (Regular) This task requires controlling a gripper to pick up a cube and stack onto another cube. It tolerant to minor inaccuracies in control, as the gripper does not need stop precisely above the cube; slight deviations still fall within the expected distribution.
TurnFaucet (ManiSkil12) (Regular) This task requires controlling gripper to turn faucet. Since it involves simple manipulation without complex contact-rich operations, and the faucet remains fixed in place, it is tolerant to inaccuracies in control and does not require real-time adjustments.
Pen (Adroit) (Regular) This task involves controlling a high-dimensional dexterous hand to rotate a pen and align it with a goal pose. Successful completion the datasct requires an average of only 30 steps, with minimum of 13 steps, indicating very short-horizon control requirement. Additionally, numerous studies (Nakamoto et al., 2023; Florence et al., 2022) have identified it one of the easiest tasks in Adroit.
25

---

Published in Transactions on Machine Learning Research (05/2025)
Hammer (Adroit) (Regular) This task involves controlling high-dimensional dexterous hand to strike nail. The hammering motion does not require fine-grained, real-time adjustments, as the primary objective is to deliver sufficient force to the nail. Minor deviations in trajectory or impact position do not significantly impact the task's success.
E Complex Real-world Scenarios Discussion
RNR-DP's responsiveness is crucial in complex real-world scenarios, where dynamic environments, distur- bances, and multi-object interactions create significant variability. The ability to quickly adapt to changing conditions is key to achieving robust performance. This responsiveness could enable RNR-DP to tackle real-world challenges like object variations, complex interactions, human disturbances, and environmental uncertainties, broadening its applicability to a wider range of lasks.
We believe RNR-DP offers significant advantages for real-robot deployment. First, it provides responsive control to handle rapid environmental changes, which are crucial in both simulation and real-world scenarios. Second, in real-robot settings, the control policy must meet specific control rate, unlike in simulation where the environment can wait for policy inference. While DP requires acceleration via DDIM for real-robot deployment, as noted the DP paper, this does not preserve performance well in our experiments. RNR- DP, on the other hand, better preserves the DP performance while achieving the necessary control rate for real-robot applications.
F Multi-Modality Property of RNR-DP
To demonstrate the preservation of multi-modality in practice, we visualize action distributions from state in the ManiSkil12 StackCube task, using our RNR-DP. We sample 1000 actions from our policy then applies PCA dimensionality reduction for visualization. We use histograms to visualize the discrete relative density of these action samples and use kernel density estimation (KDE) to visualize the estimated probability density function. The results are shown in Figure 16.
KDE Density
Relative Density
0.20
Action Relative Density
0.15
0.10
0.00
-0.02
-0.01
0.00
Action (1st principal component)
Figure 16: Conditional action distribution visualization from our experiments. Our method demonstrates clear bimodal distribution, showing that RNR-DP preserves multi-modality property.
We can see that the dineneion-reduueed action distribution exhibits a clear bimodal distribution, confirming that multi-modality is preserved in method.
26

---

Published in Transactions on Machine Learning Research (05/2025)
G Noise-Relaying Buffer Capacity Discussion
We find that the optimal buffer capacity closely related the task horizon, or the number of steps required to complete the task in the dataset. The rationale is that if task is solved in 30 steps, an 84-step noise- relaying buffer would be inappropriate. Notably, Adroit Pen has the shortest task horizon, steps with minimum of 13 steps, significantly lower than other tasks, and thus requires a much smaller buffer capacity. In practice, we set the buffer capacity to a value lower than the minimum task horizon in the dataset and make adjustments around this value. For every task, including Adroit Pen, the buffer capacity has a wide range of workable values.
average of 30
H Noise Scheduling Discussion
In this section, we aim to discuss two noise scheduling method from Streaming Diffusion Policy (Höeg et al., 2024), 67% Independent, 33% Linear and Chunk-Wise Noise Scheduling.
H.1 67% Independent 33% Linear Noise Scheduling
The key difference between Mixture Noise Scheduling and the '67% Independent, 33% Lincar' schedule is the proportion of linear noise and random (or independent) noise. In our setup, we use 60% random noise (or independent. noise from the SDP paper) and 40% linear noise from TEDi (Zhang et al., 2024), which has proven effective and robust across all tasks. We also tested the '67% independent (random), 33% linear" noise schedule from SDP (Hoeg et al., 2024) for the Adroit Relocate task, and the results show performs well, the two noise schedules (67-33 60-40) are quite similar. Refer to Table 18 for detailed results.
H.2 Chunk-Wise Noise Scheduling
To leverage current observations, RNR-DP uses a single-action rollout during each inference (T, = 1). Chunk-wise noise scheduling is inappropriate for our setting (Table 18), it is designed for action sequence rollouts (c.g., Ta 8 in DP and SDP). Our scheduling is essentially special case of chunk-wise noise perturbation, where the chunk size is set to 1.
Table 18:
Ablation study on noise scheduling scheme during training. Numbers represent average success
rates (t).
Relocate (Adroit)
Ablation
Chunk-Wiss
0.001
100% Linear
0.323
100% Random (Independent)
0.389
67% Independent (Random) 33% Linear
0.558
Mixture (60% Random 40% Linear) (Ours)
0.585
Mixture Scheduling Visualization
In this section, provide detailed discussion of the mechanism and motivation behind Mizture Noise Scheduling. As illustrated in Figure 17, the random schedule is tcaching the model to denoise actions independently, where each action is assigned random noise level. The linear schedule, on the other hand, maintains an increasing noise level across actions, closely aligning with our inference process through the noiso-rclaying buffer. Our mixture schedule not only trains the model denoise actions independently, as in the DP setting, but also ensures smooth transitions between consecutive actions. This better aligns with the noise-relaying buffer structure, resulting in more diverse and robust trainings.
27

---

Tasks and Published in Transactions on Machine Learning Research (05/2025)
Training
Inference
random schedule [ (3. B1, (a, (
ability to denoise each action independently
noibe lovel
linear schedule (6s, 82, 8a. 84, (a)
enforce inference structure (buffer)
mm
mixed schedule
Placoat Piacor
inherit advantages from both schedules
Bernoullif-)
Figure 17: Detailed illustration of the process of Mixture Scheduling.
J Laddering Initialization Visualization
In this section, we provide clear and concise discussion of Laddering Initialization. As illustrated in Figure 18, to transition from random noise to increased noise level suitable for inference through the noise-relaying buffer, we perform several denoising steps. This process results in buffer with laddered noise, ensuring more structured and effective initialization.
Gaussian
before denoising
after denoising
4144
3.3
2:2
14
dequeue & enqueue
Laddered ?
denoising step
denoising step 2
denoising step 3
denoising step 4
Figure 18: Detailed illustration of the process of Laddering Initialization. Everything happens before policy interacts with the environment.
K Qualitative Examples of How DP Struggles with How RNR-DP Resolves Them
Comparison videos of response-sensitive tasks between DP and RNR-DP can be found on this page. The videos clearly demonstrate that while DP struggles with responsive control, RNR-DP effectively completes the tasks with greater precision and adaptability.
L Discussion on Robosuite Tasks
We initially decided not to experiment with Robosuite for two reasons: (1) nearly all tasks can be solved by DP with close to 100% success, as demonstrated in the DP paper, and (2) the tasks primarily involve alacce,, which does not require highly responsive control.
M Response Sensitivity Analysis with RL Policies
this section, we assess response-sensitivity through RL training and further validate our task classification. We design two experimental settings. In Action Horizon sctting, RL agent directly outputs the next T, actions (as in Diffusion Policy); Action Repeat setting, RL agent outputs one action which is then executed in the environment repeatedly for Ta steps (a strong challenge to responsiveness). We experiment with
28

---

Published in Transactions on Machine Learning Research (05/2025)
Adroit Relocate (response-sensttivee, Adroit Pen (regular), and ManiSkill2 StackCube (regular) using SAC algorithm.
Adroit: Relocate
SAD (Action Hortzon 4)
8
(Action Repeat =
Success Rate %
HW Mh
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps
1e7
Figure 19: RT. training results on Adroit Relocate (classified as response-sensitive task). Both the Action Horizon and Action Repeat experiments fail to achieve any success on Adroit Relocate, indicating that the task has very low tolerance to outdated actions and strongly depends on the most recent observations. This supports our classification of Adroit Relocate as response-sensilive task.
ManiSkill2: StackCube
100
SAC
SAC (Action Horizon =
SAC (Adtion Repaat
Success Rata %
60
40
20
Environment Steps
167
Figure 20: RI. training results ManiSkil12 StackCube (classified as regular task). Action Horizon achieve decent success rate, while Action Repeat runs cannot, indicating that ths task has some tolerance to outdated actions. This supports our classification of ManiSkill2 StackCube as regular task.
Adroit: Pen
100
SAC
SAC (Adtion Horizon
80
SAC (Aation Pepest 8)
Success Rate %
40
0
0.0
0.5
1.0
1.5
2.0
2.5
Environment Steps
166
Figure 21: RL training results on Adroit Pen (classified as regular task). Both Action Horizon and Action
Repeat runs have reasonable success rate, indicating that the task has very high tolerance to outdated actions. This supports our classification of Adroit Pen as a regular task.
29

---

Published in Transactions on Machine Learning Research (05/2025)
In the Action Horizon setting, Ta is unsolvable due to the large action space, which hinders RL explo- ration. Behavior Cloning handles this better since it's supervised. Hence, we focus Ta = 4, where Adroit Pen and ManiSkil22 StackCube achieve reasonable success, while Adroit Relocate struggles and achieves nearly-zero success rate (See Figures 19 to 21). In the Action Repeat sctting, repeating the same action for Ta sleps poses a high responsiveness challenge. Even so, Adroit Pen still achieves some success, showing strong tolerance to outdated actions (See Figure 21). In summary, these experiments confirm that Adroit Pen and ManiSkill2 StackCube tolerate outdated actions better than Adroit Relocate, supporting our task classification.
30


Reviews:
[{"review_id": "Co7yzJIyzZ", "reviewer": "Reviewer_BUSv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n1. The paper clearly defines the limitations of Diffusion Policy, establishing a strong motivation for the proposed RNR-DP approach. The analysis of how large action horizon requirements constrain responsiveness is well-structured and logically presented.\n\n2. The method effectively decouples the denoising process along two dimensions: the action horizon (a₀–a_T) and the progressive denoising of individual actions (a_k). By reusing previously denoised actions, the approach achieves a 6.9% computational acceleration.\n\n3. The final denoising step for each action incorporates current observation conditioning, enabling RNR-DP to achieve high responsiveness.\n\n4. The experimental design encompasses diverse robotic tasks and observation modalities. Comparisons with established baselines (Diffusion Policy variants, Consistency Policy, etc.) demonstrate rigorous evaluation, while ablation studies systematically quantify component contributions.\n\n## Weakness\n1. The discussion of RNR-DP's applicability could be enhanced by extending its theoretical analysis to complex real-world scenarios involving multi-object interactions, dynamic environmental changes, and unmodeled disturbances.\n\n2. The proposed Mixture Noise Scheduling mechanism appears to partition actions at p_linear=0.4 (Table 10), suggesting approximately 60% of actions receive random perturbations (p>0.4) while 40% undergo linear variance perturbations. This resemblance to the \"67% Independent 33% Linear\" configuration in SDP (Table 2) shows different performance characteristics compared to SDP's optimal \"80% CW 20% Constant\" setting. Could the authors clarify the difference between \"Mixture Noise Scheduling\" and \"67% Independent 33% Linear\" setting?\n\n3. While Figure 5 demonstrates the performance sensitivity to noise-relaying buffer capacity, with authors indicating optimal ranges between 56-92 units, Appendix Table 11 reveals substantial cross-task variability in empirical observations (e.g., Adroit: Pen requiring 4 units vs. other tasks needing 84). Could the authors clarify the rationale for task-specific buffer capacity variations and the underlying factors contributing to its substantial impact?\n\n4. As shown in Table 5, the authors ablate both linear and random perturbations. However, since SDP is closely related to this work, it would be helpful for the authors to ablate the chunk-wise noise perturbation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies mode bouncing as a critical limitation in Diffusion Policy that compromises responsiveness and environmental adaptability. The authors propose RNR-DP (Noise-Relaying Diffusion Policy), which maintains temporal consistency through a noise-relaying buffer with progressively intensified noise levels while accelerating inference via sequential denoising conditioned on real-time observations. Extensive experimental validation confirms the method's simultaneous achievement of highly responsive control and operational efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) To effectively showcase fast, responsive closed-loop control, it's essential to conduct real-robot experiments that include human disturbances and retry behaviors.\n2) While Section C.2 provides some comparison to streaming diffusion policies, expanded comparisons across diverse tasks and simulations would strengthen the analysis.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gKGWsIz6Rw", "reviewer": "Reviewer_41NZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. I find the paper well-written and easy to follow. The authors made it very clear in the introduction what the scope and key contributions of this paper are.\n2. The motivation for proposing this method is made very clear in Section 4.1 and results in Table 1, which show that a longer action horizon in DP enables better performance in multi-modal tasks, and hence building a more responsive variant of DP that maintains the “action chunking” aspect is a meaningful thing to do.\n3. I find results on the response-sensitive group of tasks (Tables 2 and 3) statistically significant and proof that this method does help improve policy reactiveness.\n\n**Weaknesses:**\n1. This paper aims to remove a limitation in Diffusion Policy, that is of reliance on a long action horizon for multi-modal action prediction, which reduces the reactiveness of the policy. However, I do not see any qualitative or quantitative experiments showing that the change proposed in this paper did not reduce the multi-modality in the action prediction. Hence, this is to ask, is RNR-DP strictly better than DP in terms of showing both multi-modal behavior and responsiveness, or, does it trade off multi-modality for responsiveness?\n2. Since this paper ties closely with Diffusion Policy, additional experiments on the same set of simulation environments as done in the DP paper (from Robosuite) would greatly help a reader put RNR-DP in the same context.\n3. There is a lack of qualitative examples that could show that RNR-DP helps the model perform better at the response-sensitive group of tasks specifically due to higher reactivity.\n4. There is no discussion around how this method could be beneficial in real-robot deployment, if at all. There are also no real-robot experiments in this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel denoising diffusion process for robotic action prediction. They improve over the existing state-of-the-art, Diffusion Policy (DP), which predicts actions as “chunks” and rolls out parts of these chunks in an open-loop fashion. Such open-loop rollout makes DP less reactive to dynamic changes in the environment. This paper presents a solution that can help alleviate that issue by using a noisy action buffer where there is always a full-denoised action available at the head of the buffer to immediately utilize.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Qualitative or quantitative experiments showing that the change proposed in this paper did not reduce the multi-modality in the action prediction. (critical)\n2. Additional experiments on the same set of simulation environments as done in the DP paper. (strengthen)\n3. Qualitative examples that could show that RNR-DP helps the model perform better at the response-sensitive group of tasks specifically due to higher reactivity. (critical)\n4. Discussion around how this method could be beneficial in real-robot deployment. (critical)", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A5LE5wQUVu", "reviewer": "Reviewer_rwF3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The diffusion policy responsiveness problem is well motivated and is of interest of robot learning practitioners---resolving the responsiveness problem of diffusion policies will enable wider range of robotic applications, especially manipulation in dynamic scenes.\n- The noise-relaying buffer, if I understand correctly, provides a mechanism to ensure future actions come from the same mode since the backward process uses the partially denoised action from previous step.\n- The ablation experiments seem to suggest that the noise-relaying buffer does improve the performance of the policy in the considered environments.\n\n**Major Comments**\n\n***Writing***\n- I believe the writing can be further improved\n\t- Page 1, abstract, \"... executing multiple actions to retain performance..., which limits its responsiveness, ...\": It is unclear that multiple actions here are coming from the same inference step.\n\t- Page 1, abstract, \"Further evaluation on regular tasks demonstrates that RNR-DP also exceeds the best acceleration method...\": Here it is unclear what RNR-DP is exceeding.\n\t- Figure 1: What is the y-axis?\n\t- Page 3, background: $\\beta$ is not shown in this paper and it seems to be $\\beta = \\gamma$ here?\n\t- More in the points below.\n\n***Conceptual***\n- There seems to be two types of acceleration: (1) on inference time, and (2) on action horizon. Some of the existing methods aims to speed (1) up but may not necessarily improve (2), and vice versa. Can this paper outline these for clarity?\n\t- Furthermore, even if the improvement is on (2), one of the current limitations of diffusion policies appear to be large sampling time, resulting in poor policy responsiveness---does the paper improve this? If so, I believe this will greatly improve the quality of the paper. Otherwise, I would love the paper to clarify this.\n- Regarding section 5.1: While I believe that noise-relaying buffer can help alleviate mode bouncing, I believe it depends on whether the model is \"autoregressive\", in the sense that, e.g., conditioning on $a_{<t}$ for $a_t$, like diffusion policy did. Can the paper clarify if this is necessary?\n- Regarding section 5.2: I have some trouble understanding the key design choices\n\t- Mixture Noise Scheduling: My understanding is that linear schedule is to essentially trying to enforce structure similar to inference (which is to alleviate mode bouncing), while random schedule is more similar to standard diffusion model training (which is to help noise-relaying buffer initialization).\n\t- Ladder Initialization: The description here is not really clear to me. Does that mean, for every $a_f$ the model runs the backward step $f$ times with random variance each step, rather than following the linear schedule? How does this align to Figure 3 where it says, \"the buffer is denoised for $f$ times until the buffer head contains minimal noise level.\" How exactly does the buffer head contain minimal noise level and how is that determined?\n\n***Experiments***\n- Are all of these experiments executed with only a single seed each? If so, doesn't few percentage improvement in the success rate very low, and potentially statistically insignificant?\n- Regarding task descriptions,I have trouble understanding why the task groups are separated this way---how are pick-and-place and adroit pen not response sensitive tasks? Are they not dynamic? Perhaps for tasks such as pushing and pick-and-place, the environments consider the force applied onto the object and enforce the force to be below some threshold---here I suspect it's more clear that the model needs to be response sensitive. Alternatively, one may include a task where the agent needs to place item onto a moving platform.\n\n**Minor Comments**\n- Section 4.2, on the claim that diffusion policy struggles with dynamic, contact-rich manipulation tasks: While I agree this is likely the case, is there anywhere on the manuscript that supports this? If so, could the paper refer to it?\n- Based on the amount of visual-observation experiments done for this paper, I don't completely see that this work applying only to visuomotor control (which is a good thing in my opinion), but perhaps I am missing something here?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper outlines the two problems when using diffusion policies for robotic control: (1) Limited policy responsiveness due to large action horizon, and (2) with small action horizon, diffusion policy exhibits mode bouncing where consecutive actions, computed at different model inference steps, may come from different modalities. This paper proposes a method, RNR-DP, that aims to address both problems by incorporating a noise-relaying buffer.\nThis noise-relaying buffer incorporates partially denoised actions for future timesteps, so that the policy can perform less denoising steps.\nFurthermore, the partially denoised actions include information about the current action, thereby enabling the policy to ensure future actions come from the same mode.\nFinally, this paper provides an empirical example to demonstrate the mode-bouncing phenomenon, and further conduct experiments to show the improved responsiveness and improved computation efficiency when compared to existing methods such as DDPM, DDIM, EDM, etc.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Changes for Securing Recommendation**\n- The term \"responsiveness\" can be broken down into whether the current action applied is leveraging the current observation, or the inference time. The paper should clearly define what responsiveness means here, otherwise it might be misleading to future readers.\n- The writing on section 5.2 in particular appears to be critical in this paper, but I have trouble understanding ladder initialization in particular. If the paper can clarify how we started with standard Gaussian to \"laddered noise\" with random schedule that should address it.\n- Experiments appear to have limited evaluations---for an empirical paper that only considers simulated environment, I believe the analyses need to be more rigorous.\n- There should be an explanation on why specific task is in particular task group---based on the task description in the appendix I am not convinced that some tasks in the \"non-sensitive group\", e.g., pick-and-place is not any more sensitive, are less sensitive to ones in the \"sensitive group\". Some other environment recommendations such as dynamic scene manipulation will be more convincing.", "broader_impact_concerns": "As far as I know there isn't any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers recommend acceptance, acknowledging that the paper addresses an important problem in diffusion-based imitation learning policies. Initial concerns about task classification, hyperparameter sensitivity, missing comparisons, and real-world evaluation were adequately addressed in the author response. The authors provided additional visualizations, statistical analysis, and RL experiments to validate their approach.

While the paper lacks real-world robot experiments, the reviewers agreed this wasn't essential for acceptance given the comprehensive simulation results. The noise-relaying buffer mechanism is well-motivated and empirically validated across diverse robotic tasks. The authors have also demonstrated that their approach preserves the multi-modality capabilities of diffusion policies while improving responsiveness.
Decision: Accept (poster)