Cold Diffusion on the Replay Buffer: Learning to Plan from Known Good States
Authors: Zidan Wang
Takeru Oba
Takuma Yoneda
Rui Shen
Matthew Walter
Bradly C. Stadie
Venue: CoRL 2023 Poster (2023)
Keywords: Imitation Learning
Reinforcement Learning
Diffusion
Cold Diffusion
Planning
Safety

Abstract: Learning from demonstrations (LfD) has successfully trained robots to exhibit remarkable generalization capabilities. However, many powerful imitation techniques do not prioritize the feasibility of the robot behaviors they generate. In this work, we explore the feasibility of plans produced by LfD. As in prior work, we employ a temporal diffusion model with fixed start and goal states to facilitate imitation through in-painting. Unlike previous studies, we apply cold diffusion to ensure the optimization process is directed through the agent's replay buffer of previously visited states. This routing approach increases the likelihood that the final trajectories will predominantly occupy the feasible region of the robot's state space. We test this method in simulated robotic environments with obstacles and observe a significant improvement in the agent's ability to avoid these obstacles during planning.

Cold Diffusion on the Replay Buffer: Learning to Plan from Known Good States
Zidan Wang!
Takeru Oba
Takuma Yoneda
Rui Shen' Matthew R. Walter
Bradly Stadie
Northwesterrn University
Toyota Technological Institute
Toyota Technological Institute Chicago *Yale University
zidanvang20200u northwestern edu, sd215028toyota-tt jc.jp, takuma@ttic. edu rui www.shaeelllle.omee edu, mwalter@ttic. edu, stadiebradl
Abstract: Learning from demonstrations (LiD) has successfully trained robots exhibit remarkable generalization capabilities. However, many powerful imitation techniques do not prioritize the feasibility of the robot behaviors they generate. In this work, we explore the feasibility of plans produced by LfD. As in prior work, we employ a temporal diffusion model with fixed start and goal states to facilitate imitation through in paintinnnn. Unlike previous studies, we apply cold diffusion to ensure the optimization process is directed through the agent's 's replay buffer of previously visited states. This routing approach increases the likelihood that the final trajectories will predominantly occupy the feasible region of the robot's state space. We test this method in simulated robotic environments with obstacles and observe significant improvement in the agent's 's ability to avoid these obstacles during planning.
Keywords: Imitation Learning, Diffusion, Planning and Safety
1 Introduction
Consider robot in a large, enclosed garden with rock at its center. Humans initialize the robot's behaviors by collecting demonstration data of various tasks around the garden, such pulling weeds, watering plants, and pruning hedges. During these demonstrations, humans naturally avoid the large rock, preferring to walk around it. At deployment, the robot successfully accomplishes the demonstrated tasks. However, with slight change in its start location, the robot, attempting to generalize to this new situation, devises a plan that unfortunately sends it barreling straight into the rock-a classic case of faulty generalization. This scenario prompts the question: how should an agent determine which paths are suitable for generalization and which are not?
Recent techniques in LID demonstrate significant potential by utilizing diffusion models to facili- tate planning. In Diffuser [1], a robot's start and goal positions are pinned, and a temporal diffusion model fills the rest of the trajectory, process often referred to as in-painting. One major ad- vantage of in-aainting with diffusion is that enables planning over an entire trajectory. Instead of auto-regressively considering one planned state at time, in-painting allows agents to generate plans with temporal coherence across all planned states.
However, a notable limitation of Diffuser is that the algorithm's 's prior is d-dimensional Gaussian space. In the context of imitation, this implies the algorithm begins with the assumption that all intermediate states between the start and goal can occur anywhere within a d-dimensional ball. This assumption becomes problematic if the robot's actual state space contains holes or infeasible regions (e.g., due to collisions). Although the diffusion process should, in theory, converge feasible trajectory that navigates around these areas according to the data distribution, we often find this not to be the case in practice. Small variations in the robot's 's start or goal positions frequently
7th Conference on Robot Learning (CoRL 2023), Atlanta, USA.

---

Gaussian Diffusion
Cold Diffusion
Figure 1: Cold Diffusion on the Replay Buffer: In contrast to (top) Gaussian diffusion, (bottom) we define a degradation operator that iteratively maps the points of trajectory to randomly sampled points on the agent's replay buffer. We then train restoration operator R to reverse this noising process. At test time, the agent randomly samples points from the replay buffer and reverses the diffusion process obtain a trajectory that completes the desired task.
result trajectories that attempt to traverse these infeasible regions, much like a garden robot might inadvertently crash into rock.
In this paper, explore the use of an agent's past experiences to guide the optimization of a dif- fusion model. Our proposed algorithm, Cold Diffusion on the Replay Buffer (CDRB), starts with replay buffer filled with past trajectories. We iteratively apply a noising process that maps states along the trajectory to nearby points in the replay buffer. As this process repeats, the original trajec- tory degrades, transforming into set of randomly distributed nodes across the replay buffer. Using cold diffusion, we train restoration operator to reverse this noising process, mapping these noisy nodes back onto the original trajectory. During sampling. achieve new goals by pinning set of start and end positions and applying the restoration operator to generate feasible path through the replay buffer.
In summary, our work makes significant contributions in the following ways:
We introduce novel cold diffusion model, CDRB, tailored for roboties environments. CDRB addresses challenges of diffusion-basse planning methods that often generate infeasible plans.
We demonstrate the effectiveness of our method in various planning tasks using simulated robotic environments with and without significant obstacles that may obstruct the robot's path.
Related Work
Learning from Demonstrations (Lff) is cornerstone of robotics [2, 3]. The density match- ing approach that we explore in this paper was pioneered by maximum entropy formulations of apprenticeship learning [4, 5, 6, 7]. GAIL [8] achieves density matching using a GAN-like dis- criminator [9] as a reward function. This formulation was subsequently extended to accommodate general f-divergence metrics [10, 11]. The constraint of this f-divergence optimization process has been further investigated [12, 13]. Meanwhile, others have demonstrated the utility of hindsight labeling and Q-leaming within the realm of imitation learning [14, 15]. Menda et al. [16] examined the concept of safe imitation learning by utilizing ensembles to correct learned policies mid-rollowt

---

as in DAgger [17]. Yin et al. [18] and Chen et al. [19] provide further insights into safe imitation learning, especially in the context of self-driving.
When it comes learning action distribution, especially in continuous action space, the distributions have been often modeled unimodal Gaussian. Since multiple "correct" actions may be possible in certain states, trying to model it with unimodal distribution leads to failure. For example, one can imagine single state that shows up multiple times in the demonstration data is labeled with two very different actions. Trying to fit a Gaussian distribution will end up in placing its mean between those actions. To overcome this limitation, there have been several attempts to model multi-modal action distributions, with implicit functions [20], conditional VAE [21] or by discretizing action space[22]. On the other hand, diffusion models can naturally learn and generate a complex multi- modal distribution since it is designed to iteratively add or remove Gaussian noises. Given the success in computer vision community, people have recently started to explore their applications to control domains.
Diffusion Models in Robotics have enjoyed recent success, underscoring their potential and adaptability in addressing complex robotic tasks [23, 24, 25, 26, 27, 28, 29, 30, 31]. Diffuser [1] learns a diffusion process to generate entire trajectories conditioned on start and end state infor- mation. Diffuser primarily focuses on integrating Gaussian diffusion into the pejiectory-llaniiie framework in robotic settings, while recognize the importance of designing trajectory diffusion model that better aligns with the imitation learning context.
Pearce et al. [29] and Wang et al. [30] also apply diffusion models to control tasks, but they do so as state-conditioned action diffusion models, instead of trajectory diffusion models. In contrast to their approach, which looks at the problem from a policy optimization standpoint, we approach it through the lens of trajectory planning. Their framework takes in states and generates an action. Specifically, Pearce et al. [29] operates in analogy to other behavioral cloning algorithms, and Wang et al. [30] resembles the actor-critic algorithm, incorporating Q-value function guidance during training as the critic. In contrast, our approach can be viewed in analogy to multi-goal imitation learning, where the model learns to generate entire trajectories given specific start and goal states based on expert demonstration. As a result, both Pearce et al. [29] and Wang et al. [30] sample actions with real-time feedback, step by step, requiring continuous interactions with the environment, while our methodology has the capability to operate and generate complete plans without any real-time interactions during the sampling process.
Evaluation results show that Janner et al. [1]. Pearce et al. [29]. and Wang et al. [30] outperform state-of-the-art models across multiple decision-making tasks. Since they represent cutting-edge diffusion model applications in robotic environments, we adopt these methods as baselines in Sec- tion 5. providing rigorous benchmark for our method.
Concurrent to our work, Xiao et al. [32] propose SafeDiffuser to ensure that diffusion probabilistic models satisfy specifications by using class of control barrier functions. While the foundational motivation of SafcDiffuserr resonates with ours, it relies heavily on predetermined safety parameters, such as specific subspaces that identify obstacles or traps. Yet, acquiring detailed and precise data of this nature is not always practical. Unlike SafeDiffuser, our method operates without predetermined safety specifies-we we simply need expert trajectories. By design, these trajectories would not include any infeasible states, allowing our model to learn from actual experiences.
Safe Planning and Exploration is a perennial problem in the RL and control literature. There is a long history of research in the robotics community on sample-based motion planning that seeks to identify feasible trajectories in the presence of obstacles [33, 34, 35, 36, 37, 38, 39]. Work on the Bug Trap domain, where the agent has to escape a narrow corridor, was particularly inspiring for our current work [40]. Policy learning and exploration under constraints also trace their mod- ern roots back to constrained MDPs [41]. Hans et al. [42] later crystallized the notion of learning that obeys feasible state constraints, though their approach requires human labels. Classic trajectory optimization literature at times maintains fcasibility by recasting the problem as sequential convex
3

---

optimization with collision checking [43]. Modern work has sought to bridge modern RL and safe planning. often by optimizing stability [44, 45), reachability [46], or reversibility [47] to encourage safe behaviors. In the prior works that have delved into the challenge of generating collision-fre trajectories [23, 24), their experiments heavily relied on predefined cost functions (i.e., energy func- tions). On a final note, a large number of works try infer safety directly from human preferences [48, 49, 50, 51]. See Pecka and Svoboda [52] and Garefa and Fernández [53] for comprehensive literature reviews of safety in RL.
3 Background
Learning from Demonstrations (LfD) is general problem setting where a robotic system learns to copy demonstrated behavior. Suppose we start with some demonstration trajec- tory- T= (su, ao 81, sr, ar]. In the action-density matching variant of LfD, we assume there exists an underlying conditional action density pr (a sT). A policy π is then trained to recover this distribution. In other words, we train т to recover r's conditional state-action distribution. Meanwhile, in the state-density matching variant of LID, we train such that (p (s Sa, ST) Il p+(s| 8;, sT)) is minimized. In this variant, we only care about state-density matching, and action choice is optimized only to facilitate this matching.
The recently proposed Diffuser algorithm is an interesting case, because it considers the problem of density matching over an entire trajectory's 's state-action pairs. Given the initial (so, ag) and final ar) state-action pairs, Diffuser attempts to recover all intermediate state-action pairs simulta- neously. Let 7 (so, 4o 21.., 21, ,T; @T] be trajectory with the desired start and end points, but the middle contains randomly sampled Gaussian noise. The Diffuser objective is to learn mapping R: i + T, minimizing the loss function DRL, (p(((((((F)) Il p(T)).
Given its name, it should come as no surprise that Diffuser accomplishes this density matching by leveraging diffusion models. Recently gaining popularity, diffusion models are generative models that learn to iteratively map some noisy input onto a target distribution. In the context of LfD, the model takes some noisy intermediate states 21,s, 21,a--..; 21,1,, 27- and iteratively denoises them to recover 81, a1,..... 8T -1- Let us assume that we start with a noisy trajectory r(t), where t indexes the diffusion process timestep. not the temporal stage of the trajectory. The trajec- tory can be expressed as
(() = a()=(0) - -,,,,,,--000
(1)
where (t) scales the noise magnitude at timestep f, and ax()) scales based on the data magnitude. The LfD problem then to gradually remove the noise from rt)) and recover (0). This can be done using a reverse stochastic differential equation that flows backwards from t step making use of noise-perturbed score function (T). For example, if we assume Gaussian noise is added to the trajectory at each timestep, then the sampling procedure can be defined as
t 0, at each
dt = d(oo(())
d[o²(t) dt
V. (x))
dt +
dw
(2)
dt
with dw being a Wiener process. Our goal is to learn the noise-perturbed score function (TPr(T). This can be done with demoising score matching. In particular, let ()())t) be the parameterized score model. The goal is then learn the noise scale time
L (0)0 [Ilo,so(7(1)))) -
(3)
where p(7) Paaa is the demonstration density. During training, we consider weighted sum over Lt, trying to learn the appropriate noise level across various points in time. We refer the reader to Ho et al. [54] and Song et al. [55] for more discussion on possible weighting strategies. At test time, new trajectories can be generated by fixing (so, 8r), sampling T()) ~ (00,0²) and iteratively applying the
method with the learned (t() to recover r(0)
r(t) = 7(t + At) + (o²(t) o²(t (t))ss(T(()) + Vo((t) ot + At)z.
(4)

---

Note that and ST are held fixed during this process. This is referred to as pinning.
In contrast to diffusion, the recently proposed cold diffusion method [56] does not require a random degradation process. Instead, it learns score function with respect to any degradation process D(r I) that transforms the data from (() ((T())). We write the cold diffusion procedure iteratively
T(t 1) - - D(C0,, 1)
(5)
4 Methodology
We start by considering the Gaussian diffusion process in a simple 2D maze environment pictured in Figure 1. In this environment, the agent starts at some random position so in the environment and must learn to navigate to one of the four larget goal positions ST while avoiding the walls.
The diffusion process samples noisy intermediate states from Gaussian r(t) 21, 22. 2T
followed by reversing the noising process to obtain a planned trajectory (0)) 81, 52, 83 8T -1 connecting 8o and 8T.
If the entire region connecting so and ST is feasible, then the above diffusion process presents no issues. However, what happens when there an infeasible region between the two points, for example a wall? Since the Gaussian score function at time is completely divorced from the realities of the environment, it is likely sample intermediate points 2; from the infeasible region, e.g., it will sample a point inside the wall. We hope that reversing the diffusion process will fix this issue and generate a feasible trajectory. However, we find in practice that the agent often tries to generalize incorrectly and generates a path that cuts through an infeasible region, especially when 8o and 8T were not present in the training data.
Perhaps the simplest fix this infeasibility issue by projecting the fully denoised trajectory 7(0) back onto the agent's feasible region. This can be accomplished with minimal overhead by keeping a replay buffer B = (s, a) that contains all state-action pairs present in the expert demonstrations т. We then project each trajectory point s; s;()), (0) E 70) onto the closest replay buffer point
arg min|| (0)] + l|a" a0()))
66)
8" EB
Such a feasibility projection intuitively satisfying. However, if we think about it, we will soon realize that such projection will not work in our simple maze environment. The issue is that the entire planned trajectory attempts to go through the wall. Projecting this trajectory back onto the replay buffer will simply cluster all the infeasible points on either side of the wall. If the agent tries to follow this plan, it will naively go towards the wall and get stuck on one side, rather than correctly planning to avoid the wall entirely.
Nevertheless, we like the idea of this feasibility projection. Thus, we seek to incorporate it into the diffusion process itself, ensuring the entire optimization procedure is routed through the play buffer. Specifically, we define degradation process that consistently maps trajectory points (s;, a;) to random points chosen from the replay buffer based on some distance schedule, de- noted as (st, a;) ~ B. As f increases, the maximum allowable distance between successive points (s,,,, a,(()) and (t+(t)) (t,(t=++++)) (t+ 1)) also increases. Cold diffusion can then be used to reverse this degradation process by starting from a random sample of points on the replay buffer r(1) B, and learning to iteratively map them back onto a trajectory 700) that starts so and reaches sT. This procedure is described in detail below.
Degradation We want to define a degradation process ((T(0), t) = 7(t) that takes points in a trajectory and iteratively spreads them out across the replay buffer. We want this degradation to vary continuously in t, with larger values of leading to sampled points that are more spread out across the buffer. We can consider this process in a pointwise fashion for each s т. Intuitively, at time k, we want to sample some point from the replay buffer that is within some small epsilon ball about
Forward Process:
Pk Bec(s) 7
5

---

We then replace point from the trajectory 7 with the point from the replay buffer, D(so, k) + Pk- As k increases, f will also increase, which will lead to more degradation as the sampled points stray farther from the original source s. We consider several schedules for increasing size eg our experiments. For example, linear distance schedule, some max ball size fixed based on the radius of the environment, dmax. Then, Ej varies linearly from dmax
E/ dmax
Backward Process: Restoration The reverse process involves the restoration operator R. that (approximately) inverts the degradation operator D. This operator has the property that
R. (r()),))) r0)
In practice, implement this operator neural network parameterized by 0. We train the restora- tion network using the following objective
min BToe(r) (DR,(D(, -
(10)
where 7 denotes a random trajectory sampled from distribution and || denotes norm, which we take to be lz in our experiments. This restoration process aims to recover the original trajectory from the degraded one. particular, DTT, 1) degrades the trajectory to r(t), and then (T t) attempts reverse the degradation and recover 70) т. At test time, trajectories are generated by iteratively applying the restoration operator and then adding noise back to the states, with the level of added noise decreasing over time. Throughout this process, the start state so and goal state sT remain pinned. Appendix A provides the full pseudocode of the training and sampling processes.
Experimental Results
We conducted experiments on a range of environments consisting of various challenging obstacles that prevent the agent from reaching target objects or goal locations in trivial manner (i.e., straight line). We incorporated obstacles into existing environments to evaluate the agent's ability navigate around obstacles and achieve its goals.
Figure 2: Environments considered in this paper. Left: Fetch robot must pick and place a box while avoiding wall. Center: Robot starts between two walls and must reach point outside the walls. Right: Robot must push a block to a desired point while avoiding an obstacle.
1) Maze and Maze Tight: These environments simulate multi-goal maze with obstacles placed to divide the area at the center from the outside. There are four possible goals lo- cated at the corners of the environment, and one of them is randomly selected at each run. The Maze Tight variant is an adaptation of the original Maze, featuring narrower passages between the obstacles to test the algorithm's 's precise planning skills.
2) Reach and Reach Obstacle: In these environments, the primary objective to navigate the robotic arm (gripper)
specified target location. The Reach Obstacle variant introduces two tall walls each side of the starting gripper position test the robot's ability reach the goal outside of the walls.

---

3) Pick and Place and Pick and Place Obstacle: These environments challenge the robot to grasp a block using the gripper and relocate to specified goal location. 'he Pick and Place Obstacle version of this environment adds further complexity by including a tall wall that the robot arm must go up or around to reach the block or goal location.
4) Push and Push Obstacle: In these environments, the robot's objective is to push block to a target location without activating the gripper fingers. The Push Obstacle environment includes a short wall placed in the center.
For each environment, we first generated expert trajectories through the training of proficient agents via goal-conditioned policies. For the maze environments, multi-goal Soft Actor Critic (SAC) [57] models were used. For Reach, Pick and Place, Push, and their obstacle variants, we utilized Hind- sight Replay Buffer (HER) [58] to train the agent. We compare against three recently proposed methods that leverage Gaussian diffusion for imitation learning: Diffusion-QL. [30], Diffuser [1], and Diffusion BC [29]. We also included baseline comparison where multi-layer perceptron (MLP) is trained to concurrently predict entire trajectories, given specified start and goal states.
Success Rate (%)
MLP
Diffuser
Diffusion BC
Diffusion-QU
CDRB (ours)
Maze
74.22 #3.81
100.0 10.00
90.13 11.21
100.0 :0.00
95.33 #2.05
Maze Tight
38.29 1.80
97.00
+0.82
89.21 11.20
84.02 #3.86
98.00 +0.00
Reach Obstacle
80.74 +5.20
97.56
+0.74
89.63 +0.41
74.03 +8.12
97.92 +0.18
Pick & Place Obstacle
0.00 +0.00
1.22 46.68
58.80 2..34
24.19 +-8.11
70.88 +1.01
Push Obstacle
2.43 11.10
11.45 58.64
34.81 17.79
29.41 115.24
71.04 11.58
Reach
90.34 +2.18
100.0
+0.00
94.02 1.03
58.60 +14.69
100.0 +0.00
Push
2.45 11100
70.56
10.68
68.23 18.13
65.60 +12663
89.92
10.69
Pick & Place
0.00 10.00
92.36 1121
61.00 13.56
23.60 19.63
95.76 10.83
Table 1: Benchmark results of test-time success rates goal reaching. Results over seeds.
Table shows the quantitative results over various planning methods across eight different domains. The mean and standard deviations are computed from trials over different seeds. This shows that our approach consistently outperforms the baseline approaches in success rates except for Maze domain.
Qualitatively, we observe that Diffuser often generates an infeasible trajectory (e.g., some states are in the wall), while CDRB mostly generates path in the feasible region. In addition, we also find that the plans generated by CDRB are often a lot smoother than those by Diffuser, as discussed in Appendix
Figure 3 shows an example of trajectories generated by Diffuser that directly cut through the wall and result in failures at execution time. In contrast, CDRB -oRB-ggneeaated trajectories would not cause the gripper to collide with the wall during execution.
6 Further Discussion
k-Means Clustering Keeping a full replay buffer can be costly, since diffusion across a larger buffer could have higher propensity to make small adjustments at each reverse diffusion step. We thus experimented with applying k-Means clustering to reduce the buffer size. After clustering the buffer, clip the resultant cluster centroids to the closest buffer points to ensure the feasibility of these states. The cluster centroids then become new buffer. In general, we found this improved computational efficiency without degrading performance (see Figure 4(a)).
Temporal Transformers Recent work has proposed the of temporal transformers for fiffusion-based imitation [29]. In contrast, the original Diffuser method uses temporal U-Net [1].
7

---

Expert
Diffuser
CDRB
Push Obstacle
Pick & Place Obstacle
Figure 3: Trajectories on Pick & Place Obstacle and Push Obstacle. Diffuser- generated trajectories frequently struggle with crashing into the wall. In contrast, CDRB successfully learns navigate around environmental wall constraints.
1.0
1.0
0.8
0.8
Success Rate
Suceess Rate
0.6
0.4
0.4
CDRB k-50
CDRB UNet
CDRB 1-100
0.2-
CDRB Transfommrr
CDRB 5-500
Diffuser UNet
Diffuser
0.0
Diffuser Transfommr
2000
5000
10000
2000
4000
6000
10000
Iterution
Iteration
(a) k-Means Size
(b) Transformer vs U-Net
Comparative analysis of replay buffer efficiency via k-means clustering and performance dynamics of temporal architectures in diffusion processes.
Figure 4:
Interested in the comparison, explored both architectures for learning Rg. Looking Fig- 4(b), we see that transformers generally converge much faster, but ultimately achieve worse final results. This is consistent with results in Table 1, where Pearce et al. [29] often perform slightly worse than Diffuser.
7 Limitations and Discussion
While we were generally impressed with the capabilities of CDBR, the algorithm has several both- ersome limitations at this time. Among them, the algorithm seems sensitive to the number of reverse diffusion steps at test time. In fact, sensitivity in this regard is worse than most extant baselines. Because of our dependence on the replay buffer, the algorithm tends to struggle areas of low visitation density, such as the Bug Trap environment that inspired much of our analysis [40].
Much like Diffuser [1]. we find that directly using inffusiion-gneertee actions often works poorly, and had to resort to inverse dynamics control. Yet, recent works have seen success with directly using diffusion for control [30, 31], which suggests should be possible. In contrast to prior work [30, 31], we found our trained policies struggle to learn from noisy data. When initializing from suman-provided demonstrations, rather than RL-bbased demonstrations, we found performance of all diffusion-based imitation learners suffered. This is well-known problem in LID [59]. We remain optimistic that future efforts in diffusion-based imitation will bear fruit in this area.

---

Acknowledgments
We would like to thank David Yunis for providing crucial advice making the implementation of our forward diffusion process tremendously faster through vectorization. We also express gratitude to our colleagues and reviewers for their invaluable feedback and insights that significantly improved this paper. This work was supported in part by the National Science Foundation under HDR TRIPODS (No. 2216899).
References
[1] M. Janner,, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
[2] Schaal. imitation learning the route humanoid robots? Trends in Cognitive Sciences, (6):233-245) 242,, 1999.
[3] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: survey learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[4] A. Ng and S. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 05 2000.
[5] Abbeel and Ng. Apprenticeship learning via inverse reinforcement learning. In Proceed- ings of the International Conference Machine Learning (ICML), 2004.
[6] B. D. Ziebart, A. L. Maas, A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforce- ment learning. Proceedings of the National Conference on Artificial Intelligence (AAAI), pages 1433-1438, 2008.
[7] J. Ho, J. Gupta, and S. Ermon. Model-free imitation learning with policy optimization. In Proceedings of the International Conference on Machine Learning (ICML), pages 2760-2769, 2016.
[8] J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2016.
[9] Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances Neural Information Processing Systems (NeurIPS), 2014.
[10] K. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imitation learning methods. In Proceedings of the Conference on Robot Learning (CoRL), pages 1259-1277, 2019.
[11] X. Zhang, Li, 2. Zhang, and 7.-1.. Zhang. f-GAIL: Learning f-divergence for generative ad- versarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 12805-12215, 2020.
[12] I. Durugkar, M. Tec, S. Niekum, and P. Stone. Adversarial intrinsic motivation for reinforce- ment learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 8622- 8636, 2021.
[13] Y. Ma. J. Yan. D. Jayaraman, and O. Bastani. How far I'll go: Offline goal-conditioned rein- forcement learning via f-advantage regression. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[14] S. Reddy, A. Dragan, and S. Levine. SQIL: Imitation learning via reinforcement learning with sparse rewards. In Proceedings of the International Conference on Learning Representa- tions (ICLR), 2020.

---

[15] Y. Ding, C. Florensa, P. Abbeel, and M. Phielipp. Goal-conditioned imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[16] K. Menda, K. proach safe imitation learning- In Proceedings of the IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 5041-5048, 2019.
and M. Kochenderfer. EnsembleDAgger Bayesian ap-
[17] Ross, G. Gordon, and A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627-635, Ft Lauderdale, FL, 2011.
[18] H. Yin, P. Seiler, M. Jin, and M. Arcak. Imitation learning with stability and safety guarantees. IEEE Control Systems Letters, 6:409-414, 2022.
[19] J. Chen, B. Yuan, and M. Tomizuka. Deep imitation learning for autonomous driving in generic urban scenarios with enhanced safety. In Proceedings of the IEEE/RSI International Confer- ence on Intelligent Robots and Systems (IROS), pages 2884 2890, 2019.
[20] Florence, C. Lynch, A. Zeng. O. Ramirez, Wahid, Downs, A. Wong, J. Lee, I. Mor- datch, and Tompson. Implicit behavioral cloning. CoRR, abs/2109.00137, 2021. URL https:/ /ttp///aaxvvvvvve wwwooo//gaa.. attp///...//....111
[21] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. CoRR, abs//8122/ 122.0900,, 2018. URL http::/aaxxvv.orree /112. 111./0012..0....
[22] Y. Tang and S. Agrawal. Discretizing continuous action space for on-policy optimization. CoRR, has/1901.10500 2019, URL htttt:://arriv..rr/bbb////0.0..
[23] Huang, Z. Wang, P. Li, B. Jia, Liu, Y. Zhu, W. Liang, and S.-C. Zhu. Diffusion-based gen- eration, optimization, and planning in 3D scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[24] J. Carvalho, M. Baierl, J. Urain, and J. Peters. Conditioned score-based models for learning collision free trajectory generation. In Proceedings of the 2022 NeurIPS Workshop on Score- Based Methods, 2022.
[25] W. Liu, Y. Du, T. Hermans, S. Chernova, and C. Paxton. StructDiffusion: Language-guided creation of physically-valid structures using unseen objects. In Proceedings of Robotics: Sci- ence and Systems (RSS), 2023.
[26] Urain, N. Funk, J. Peters, and G. Chalvatzaki. SEff)--iiftssionFiilld: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2023.
[27] C. Chi, Feng, Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[28] I. Kapelyukh, V. Vosylius, and E. Johns. DALL-E-Bot: Introducing web-scale diffusion mod- els to robotics. In Proceedings of the 2022 CoRL Workshop on Pre-training Robot Learning, 2022.
[29] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, and S. Devlin. Imitating human behaviour with diffusion models. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[30] Z. Wang. J. J. Hunt, and M. Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. In Proceedings of the International Conference Learning Repre- sentations (ICLR), 2023.
10

---

[31] T. Yoneda, L. Sun, G. Yang, B. C. Stadie, and M. R. Walter. To the noise and back: Diffusion for shared autonomy. In Proceedings of Robotics: Science and Systems, 2023.
[32] W. Xiao, T.-H. Wang, C. Gan, and D. Rus. SafeDiffuser: Safe planning with diffusion proba- bilistic models. arXiv preprint arXiv:2366.00144 2023.
[33] S. M. Lavalle. Planning Algorithms. Cambridge University Press, 2006. ISBN 0521862051.
[34] J. Kuffner and S. LaValee. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065), volume 2, pages 995-1001
vol.2, 2000. doi: do::10.1109990 00.1100000BBT.20000000 844730.
[35] S. LaValle and J. Kuffner. Randomized kinodynamic planning. In Proceedings 1999 IEEE In- ternational Conference on Robotics and Automation (Cat. No.99CH36288C)) volume 1, pages 473-479 479 vol.1, 1, 1999. doi: 190.1100//OBB0.1999990000000
[36] S. Karaman and F. Frazzoli. Incremental sampling-based algorithms for optimal motion plan- ning. CoRR, abs/1005.0416 2010. URL http::/arxiv..ortt http:///aaaii..ooggb////00000 abs/1005. 0416.
[37] S. Karaman and E. Frazzoli. Optimal kinodynamic motion planning using incremental sampling-based methods. In 49th IEEE Conference on Decision and Control (CDC), pages 7681-7687, 2010. 10.1 100///C..201...57775333
[38] L. Kavraki, P. Svestka, J.-C. Latombe, and M. Overmars. Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Transactions on Robotics and Au- tomation, 12(4):566-580, 1996. doi: 109/700.04489
[39] S. Karaman, M. R. Walter, A. Perez, F. Frazzoli, and S. Teller. Anytime motion planning using the r In 2011 IEEE International Conference on Robotics and Automation, pages doi: 10111199///.111111
1478 1483, 2011. 1.
[40] A. Yershova, L. Jaillet, T. Siméon, and M. Vallle Dynamic-domain RRTs: Efficient exploration by controlling the sampling domain. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 3856-3861, 2005.
[41] E. Altman. Constrained Markov decision processes, volume 7. CRC press, 1999.
[42] A. Hans, D. SchneegaB, A. M. Schäfer, and S. Udluft. Safe exploration for reinforcement learning. In ESANN, pages 143-148. Citeseer, 2008
[43] J. Schulman, Y. Duan, J. Ho. A. Lee, I. Awwal, H. Bradlow, J. Pan, S. Patil, K. Goldberg, and P. Abbeel. Motion planning with sequential convex optimization and convex collision checking. International Journal of Robotics Research, 33(9):1251-12700 2014.
[44] J. F. Fisac, N. F. Lugovoy, V. Rubies-Royo, S. Ghosh, and C. J. Tomlin. Bridging Hamilton- Jacobi safety analysis and reinforcement learning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 8550-8556, 2019.
[45] C. Gehring and D. Precup. Smart exploration in reinforcement learning using absolute tempo- ral difference enrors. In Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pages 1037-1044, 2013.
[46] T. W. Bickmore, H. Trinh, S. Olafsson, T. K. O'Leary, R. Asadi, N. M. Rickles, and R. Cruz. Patient and consumer safety risks when using conversational assistants for medical informa- tion: an observational study of Siri, Alexa, and Google Assistant. Journal of Medical Internet Research, 20(9), 2018.
[47] M. Moldovan and P. AbbeeL. Safe exploration in markov decision processes. In Proceedings of the International Conference Machine Learning (ICML), 2012.
11

---

[48] P. F. Christiano, J. Leike. T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[49] B. C. Stadie, P. Abbeel, and I. Sutskever. Third person imitation learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
[50] D. Hadfield-Menell. S. J. Russell, P. Abbeel, and A. Dragan. Cooperative inverse reinforcement learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS), 2016.
[51] Leike, D. Krueger, T. Everitt, Martic, V. Maini, and Legg. Scalable agent alignment via reward modeling: A research direction. arXiv preprint arXiv:181..0787,.22 2018.
[52] M. Pecka and T. Svoboda. Safe exploration techniques for reinforcement learning- an overview. In Proceedings of the International Workshop on Modelling and Simulation for Autonomous Systems (MESAS), pages 357-375, 2014.
[53] J. García and F. Fernández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015. 2015.
[54] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Advances Neural Information Processing Systems (NeurIPS), pages 6840-6851, 2020.
[55] Song, C. Meng, and Ermon. Denoising diffusion implicit models. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[56] A. Bansal, E. Borgnia, H.-M. Chu, J. S. Li, H. Kazemi, F. Huang, M. Goldblum, J. Geiping, and T. Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv: 09392, 2022.
[57] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[58) M. Andrychowicz, F. Wolski, A. Ray, Schneider, R. Fong, P. Welinder, B. McGrew, J. To- bin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[59] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Proceedings of the Conference Robot Learning (CoRL), pages 1678-1690) 2021.
12

---

A
Algorithm Pseudocode
Algorithm Training the Noise Model
Require: Expert trajectories T, distance schedule e, number of diffusion steps t, replay buffer B
Output: Trained restoration operator Ra
1: repeat
2: T T~7 T (Sample a trajectory from expert trajectories)
3: k~ (f.....,t 13) (Sample a diffusion step uniformly)
4: for = 0, T
5: sk) Be (st) B (Sample a RB point within ball around s; with radius e,)
6: end for
7: Construct noisy trajectory rkk) [k)) 81 (k)) (.................... 8T) [k()]
8: r(0) (T(T(k),
9: Perform gradient descent min E (ofrpp(r]/T((1 -|
10: until converged
Algorithm Sampling Process: Generating new Trajectories
Input: Replay buffer B, start state so, goal state ST
Required: Pretrained restoration operator Rg, position controller fo, number of diffusion timesteps
Output: State-action sequence r0)) = [so; .o....; ST; aT]
1: Sample T' + state-action pairs r(t) uniformly from B
Pinning start and goal states: 8g + T [l) - -(()
3: for k t, ........... do
+(0) R2( (T()
5: T(A = D(f(0);
Pinning start and goal states: so + T(k - 1)[0), 87 - + - 1)[7]
7: end for
for = 0,
9: Compute the action fo(si,
10: end for
B State Space Details
In our two maze environments, the state spaces are 4-dimensional, including x and coordinates of the agent's position and its linear velocities in those directions.
The state space for Reach is 10-dimensional and consists of the gripper's kinematic information. It includes end effector positions and velocities in the y, and z directions for the gripper, joint displacements, and velocities of the left and right gripper fingers.
The state spaces for Push and Pick and Place are 25-dimensional and consist kinematic informa- tion for both the block object and gripper. Gripper dimensions include end effector positions and velocities in the x, y, and z directions for the gripper, joint displacements, and velocities of the left and right gripper fingers. For the block, it involves positions. relative positions to the gripper, global rotation, relative linear velocity, and block angular velocity along the x, y, and axes.
13

---

C Experimental Details
In both the Maze and Maze Tight environments, we initially trained expert policies using the Soft Actor Critic (SAC) method for one million training timesteps, utilizing sparse reward [57]. Then, we collect the 8k demonstrations as the training dataset by using SAC, and train the diffusion models with this data. Since the lengths of the demonstration trajectories are different, we randomly crop the demonstration trajectories so that the trajectory length becomes equal for training. The cropped length is 168 in this experiment. In the inference stage, randomly sample starts and goals while ensuring that the goal can be reached within 168 steps. The goals are sampled from 4 points in the center of the colored circle near the corner in Fig. 1. The starts are sampled from the inner region surrounded by walls located around the center. Then, the diffusion model plans the trajectory by inpainting manner in the same way as diffusion planning [1]. The action is obtained by a posi- tion controller, which calculates the torque so that the agent follows the trajectory predicted by the success.
diffusion model. If the agent touches the goal circle, the planning regarded
In the Fetch environments, we obtained an expert policy with HER, trained for two million train- ing steps. These environments were all multi-goal with sparse reward binary feedback. Goals are sampled uniformly from the environment, and involve reaching a target location or moving a block to the target location. We collected 5,000 demonstrations for each environment, with varying tra- jectory lengths. Specifically, in the Reach and Reach Obstacle environments, the trajectories were of length 30, while in the Pick And Place and Push environments, they were of length 40. For the Pick And Place Obstacle and Push Obstacle environments, the trajectories had length of 60. It should be noted that on Pick And Place Obstacle and Push Obstacle, the best HER [58] controller we could train received only a 90% success rate, suggesting this environment is difficult even for RL agents. Like the maze environments, validation sets were used to measure the performance of diffusion models. Actions are obtained with a position controller, which allows the agent to follow the paths generated by the diffusion model. If the agent manages get within e of the target goal at the last timestep, the trajectory is counted success.
D Additional Ablation Analysis
1.0
1.0
0.8
0.8
Success Rate
Success Rate
0.6
0.6
0.4
CDRB (state-cction)
CDRB Schednle:linear
0.2-
CDRB (statc-only)
CDRB Scheduledog
Difflaee (state scate-action)
Diffuser
0.9-
Diffiaser (state-only)
2000
4000
6000
8000
10000
6000
10000
Iteration
Iteration
(a) Distance Schedule
(b) Action Inclusion
Figure 5: Comparative Analysis of Distance Schedules and Action Inclusion in Diffusion Processes.
Distance Schedule: We wondered if changing the rate of degradation in our forward diffusion process would impact learning (Equation 7). Using both log and a linear distance schedule, we see the log schedule learns slightly faster, but is generally less performant at convergence. See Figure (a). We suspect slower derogation schedule causes our algorithm to spend too much time optimizing smaller regions of the state space.
Given the use of inverse dynamics model to generate actions, the use of actions in the diffusion target was completely optional. We conducted ablation experiments comparing state-
Action Inclusion:
14

---

action to state-only sequences. Our findings indicate that CDRB's performance remains consistent, irrespective action inclusion. contrast, Diffuser's performance appears to be not only unstable when relying on states alone but also exhibits slower learning curve across varying iterations. See Figure (b).
How smooth are the generated trajectories? During evaluation, noticed that trajectories gen- erated by CDRB appeared smoother than those generated by Diffuser. See the example trajectories in Figure 6 (b) and (c). To test this, we divide the path length of the trajectories generated by Dif- fuser and CDRB by the path length of a trajectory generated by SAC. We see that under this metric, CDRB generates more efficient paths to the goal. particular, seems much better at locking onto smooth region the start of its trajectory, opposed Diffuser, which struggles at initial timesteps. This result is shown in Figure 6(a).
CDR3 50
CDR3 -5000
Path Lengih
D3
2000
4000
6000
3000
10/00
Itenation
(a) Normalized Path Length
(b) CDRB
(c)
Diffuser
Figure 6: Comparison of normalized path length and examples of the predicted trajectory by CDRB and Diffuser Maze Tight environment.
15


Reviews:
[{"review_id": "Y03DudAv16A", "reviewer": "Reviewer_Qj78", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a method to sample new trajectories using cold diffusion, given a set of expert demonstrations. They tackle the problem of wrong generalization apparent in diffusion models that rely on Gaussian noise for degradation by forcing the degradation process to go through the expert dataset. On a simple set of experiments, the authors show that the new degradation process results in better performance. ", "summary_of_recommendation": "I ask the authors to clarify the above points and rewrite the paper to address the lack of clarity and technical correctness. Additional results would also be desired to clarify whether the mentioned generalization issue is evident in the low-data regime. If my points are addressed or some misunderstanding from my side is clarified, I am willing to raise my score, but in its current state, I vote for reject.\n ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Irrelevant to robotics", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "### Main Points\n- the restoration operator is not well described. In the introduction, the authors talked about a planner. In the methodology, they did not mention this at all. \n- \"In this work, we explore the problem of ensuring the feasibility of plans\ngenerated by LfD\" & \"Because this entire diffusion process happens on the replay buffer, the resulting trajectory is guaranteed to be feasible.\" --> How can a trajectory generated with a trained inverse dynamics model be guaranteed to be feasible? Even with a perfect inverse dynamics model, how is the feasibility guaranteed when s_0 is not in the expert dataset?\n-  this approach seems to have bad generalization capabilities as the target restoration operator's domain only includes states in the dataset. How well does it generalize in practice, especially when using only a few samples?\n- this approach does not seem to scale well to complex higher dimensions as it needs to keep track of datasets. The proposed clustering solution is insufficient as it relies on a metric, which is difficult to get in all domains.\n- why is the target restoration operator in the equation under line 179 only approximately equal to $\\tau(0)$?\n- why is it called cold diffusion on the \"replay buffer\" if, in the end, just a set of expert demonstrations are taken?\n\n### Minor issues\n- formulas don't have numbers\n- line 116: correct sentence \n- line 122: should not be $\\tau(0)$\n- line 132: Figure 1 is meant \n- line 144: \"ineffability issue\"?\n- supplementary should be in a separate file \n- Algorithm 2 line 7: should be $s_{i+1}$ for the inverse dynamics model \n- line 239: change -> changing ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In its current stage, I think that the paper is not written detailedly enough. Many points need to be clarified and some technical points need to be corrected. While the idea might be interesting, more work must be done to put this paper in a publishable state. \n\n### Strengths\n- interesting idea\n- the paper is easy to read \n\n### Weaknesses\n- the paper is not well motivated\n- while the paper is easy to read, it lacks detailedness\n- some statements seem to be incorrect\n- the generalization capabilities of the approach seem to be very limited\n- many points need more clarification (see below)\n- no real-world robotics experiments\n\n\n\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "MaNR0ZQXO4", "reviewer": "Reviewer_JCXJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper deals with the problem of generating feasible trajectories for collision-free robot navigation. The paper proposes a novel algorithm, called CDRB that learns a diffusion model that maps points along the dataset, in contrast with classical diffusion-based methods.\n\nIn CDRB, rather than adding white noise in the degradation phase, as it is common in Denoising score matching, they map the points to nearby points that exist in the dataset. Then, the network is trained to map the points back from the noisy point to the original trajectory.\n\nIn the experimental section, the authors compare their method against other baselines in collision-free trajectory generation.The authors show the benefit of their proposed method in contrast with baselines.", "summary_of_recommendation": "The paper presents an interesting idea to solve a robotics relevant problem. \n\nNevertheless, in the current format, I suggest a rejection. I suggest the authors to consider the weaknesses I highlighted to improve the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "The questions are directly connected with the highlighted weaknesses.\n\nThe most intriguing one is the one of the collision-free guarantees (Weaknesses 1). It is important the authors to clarify if they are able to guarantee collision-free trajectories and in case they are not, to explain how they could improve the method to obtain them.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written and it is easy to follow. The motivation is well presented and the contribution is clear. It might be interesting to highlight in bold the contribution of the work or maybe adding bullet points for even easier understanding of the work’s contribution.\n\nThe proposed approach is original, yet it is not evident that it will always work. On the one side, the degradation phase requires searching for the sample that is closest to an applied perturbation. This might be limiting when the dataset is too big. On the contrary, with a small dataset, there might not be sufficient datapoints in the vicinity to move. \n\nAlso, given the noise is added per state and not in the trajectory, there are no guarantees that the generated trajectory will be collision free. Particularly, close to the corners of the collision bodies or when the collision body is small.\n\nThe authors on the other hand are missing a proper citation. There exists some works that have explored the problem of generating collision-free trajectories [1,2], against which an experimental comparison would be important. It would be also nice to cite the works in the field of diffusion models for Robotics [3,4,5,6,7,8].\n\n**Strenghts**\n\n1. The problem of generating collision-free trajectories is important for Robotics. The authors proposes a novel and original approach to find trajectories that are collision-free.\n\n\n**Weaknesses**\n\n1. It is not fully evident the paper will guarantee collision-free trajectories. Given the constraint is set in the chosen states, it is possible to generate state transitions that goes through a collision. Even if two states are collision-free, the transitioning between two collision-free states might go through a collision. This situation is evident when the collision body is small or in the corners of the collision bodies.\n\n2. The training algorithm seems to suffer with big datasets or too small datasets. Choosing a proper dataset might be hard in practise.\n\n3. The authors are not presenting real robot experiments.\n\n4. There exists previous works in the topic of generating collision-free trajectories with diffusion models [1,2]. It is essential to properly place the contribution of the work in contrast with previous works.\n\n\n\n[1] Huang, Siyuan, et al. \"Diffusion-based generation, optimization, and planning in 3d scenes.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2] Carvalho, Joao, et al. \"Conditioned Score-Based Models for Learning Collision-Free Trajectory Generation.\" NeurIPS 2022 Workshop on Score-Based Methods. 2022.\n\n[3] Liu, Weiyu, et al. \"Structdiffusion: Object-centric diffusion for semantic rearrangement of novel objects.\" arXiv preprint arXiv:2211.04604 (2022).\n\n[4] Urain, Julen, et al. \"Se (3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.\n\n[5] Chi, Cheng, et al. \"Diffusion policy: Visuomotor policy learning via action diffusion.\" arXiv preprint arXiv:2303.04137 (2023).\n\n[6] Kapelyukh, Ivan, Vitalis Vosylius, and Edward Johns. \"Dall-e-bot: Introducing web-scale diffusion models to robotics.\" IEEE Robotics and Automation Letters (2023).", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "EN8eQHtOkU7", "reviewer": "Reviewer_aADh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The focus of this paper is on addressing the issue of plan feasibility by utilizing a temporal diffusion model. This model incorporates cold diffusion and draws on the agent's previous experiences to guide the optimization of a diffusion planner. A noising process is used to map states to nearby points in the replay buffer, and then cold diffusion is applied to reverse the noising process, ensuring that the resulting trajectory is feasible. Additionally, cold diffusion is used to address the sub-optimality of employing Gaussian diffusion when obstacles are present in the environment. The authors conducted experiments in various simulated environments where obstacles blocked the start and goal locations.\n", "summary_of_recommendation": "Based on the points raised and careful consideration, I suggest a weak reject. The main concerns are the missing clarifications on how this work differs from [1], [2], and [3] as well as the lack of a comparison with the method proposed in [1] when it is built on top of this work. Moreover, the list of issues in the questions for rebuttal should also be addressed.\n\n[1] Janner et al, Planning with Diffusion for Flexible Behavior Synthesis, ICML 2022.\n\n[2] Pearce et al, Imitating Human Behaviour with Diffusion Models, ICLR 2023.\n\n[3] Wang et al, Diffusion Policies as An Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023.\n\n\n**Update:**\nBased on the clarifications presented in the rebuttal, I believe they've addressed most of my concerns about this paper, and I am changing my recommendation to a 'weak accept.'", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In this paper, my main concerns revolve around the identified weaknesses. Additionally, there are some important pieces of information that require clarification. \n* 5.1 K-Means Clustering: how much computational efficiency is improved?\n* Figure 4: reorder subplots to match the ordering of each topic in 5.1 Further Discussion.\n* Figure 5: reorder subplots to match the ordering of each subplot mentioned in the text. \n* Please formally introduce DDPM and Diffusion-QL.\n* Please ensure all the references are up-to-date (e.g., [1] has been accepted to ICML 2022).\n* What are the observation and action spaces in each environment?\n* Why do the trained policies struggle in multi-modal environments even though diffusion models are known to excel at covering multi-modal distributions [2]?\n* In the statement \"Because this entire diffusion process happens on the replay buffer, the resulting trajectory is guaranteed to be feasible,\" how can we ensure that the resulting trajectory is feasible unless every resulting state comes from the replay buffer? If this is the case, it could be clarified in the paper.\n\nTypos:\n* Line 103 - “map” to “mapping”\n* Line 239 - “change” to “changing”", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* This paper presents a planning framework that uses cold diffusion to enable optimizations through the agent’s replay buffer of prior states.   \n* Figures are clear and helpful in understanding the method and results.\n\nWeaknesses:\n* The contributions of the paper are not clearly stated in the introduction.\n* It needs to be clarified how this work is different from previous studies referenced in [1], [2], and [3].\n* The paper’s proposed method seems to be based on the work of [1]. Specifically, it introduces the idea of using the agent’s past experiences to help guide a diffusion planner’s optimization. However, no direct comparison is made against the method proposed in [1] in the experiments section, making it difficult to assess the benefits of utilizing the agent’s prior experiences in the diffusion planner’s optimizations. \n* No hardware experiments.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "6hLqWqvD6U", "reviewer": "Reviewer_MbuE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper implements “cold diffusion” for planning problems in robotics. The cold diffusion, unlike the standard diffusion process, is a Gaussian noise-free method that follows a degradation of the given data and learns a restoration model to reconstruct the original distribution. The authors employ a distance schedule to degrade a state in the trajectory by sampling a state in the dataset within a ball of scheduled radius around it. The planning formulation introduced by diffusion planning-Diffuser s followed to structure the trajectory as a sequence of (state, action) with given start and goal states. The authors show that the proposed method is better at handling infeasible regions or out-of-distribution areas and improves over prior diffusion model-based formulations. Further, implementation details have been discussed, and the choice of K-means clustering, UNet, and distance schedule has been justified. The results are presented in 4 simulated robotics environments.", "summary_of_recommendation": "The proposed method uses a recently introduced approach of cold diffusion to degrade the trajectory samples by using the data in the replay buffer and learns to restore the true trajectory from randomly sampled data from the replay buffer. Because everything happens inside the replay buffer, there is no possibility of generating datapoints in infeasible regions. This is a new direction in planning for robotics, and authors have shown success in several simulated robotics environments. However, there is also a problem that many papers have used Diffuser as a baseline but did not report this limitation. The motivation is somewhat questionable unless justified properly.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. What is exactly the K-Means step? Does k=100 mean that you only have 100 states after clustering and clipping? If yes, then the degradation is done only using those selected 100 states? Because this can eventually mean overfitting to the low number of data.\n\n2. Alg 2 in Appendix A in not complete, first it does not uses the sampling method mention in the paper and second, there is no inpainting of start and goal states. Appendix C is confusing without any proofs.\n\n3. See Review Above\n\n4. How will the proposed method behave if the start and goal states are out of distribution of the replay buffer? Like in Fig 5 (b), if replay buffer contains data from blue, green and yellow corners to goal in the center, and evaluated on a start state in red circle.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "While the effectiveness of Cold Diffusion is still debatable in general application (https://openreview.net/forum?id=slHNW9yRie0), the authors have tried to justify why it makes more sense to use the data points in the dataset to degrade the data rather than adding gaussian noise. While, unlike diffusion models, there is no “theoretical” evidence or guarantee for such a phenomenon, the authors find cold diffusion to work better. \n\nThe authors refer Diffuser as DDPM across the paper, however, that is not the case. Diffuser incorporates classifier guidance by incorporating the expected return into the sampling process of obtaining the state-action sequence while keeping the start and goal state fixed via in-painting. Such a procedure is not yet established for cold diffusion.\n\nIf the authors plan to use an inverse dynamics model, why is state-action sequence sampled from the diffusion process? One can also sample state-only sequence (https://anuragajay.github.io/decision-diffuser/). The authors must include this as a baseline too in that case.\n\nFurther, recent works also show addition of safety in the reverse sampling process (https://arxiv.org/pdf/2306.00148.pdf). While the concern of the authors is justified that the samples might fall in the infeasible region, practically, several candidate solutions are sampled and the best one of them is chosen according to some metric (Q function, discounted return of the planned sequence). This weakens the overall motivation, however, the fact that cod diffusion is actually sampling efficiently is an important result that cannot be overlooked.\n\nFrom the limitations mentioned by the authors, the proposed approach seems unstable in terms of several hyperparameters. Also, because only the provided data is used, it is definitely going to suffer from not modeling multi-modality as mentioned by the authors as well.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method to improve the feasibility of motion plans generated by conditional diffusion models. The method builds on the recently-developed cold diffusion technique and proposes to replace the gaussian noising process with a process that degrades an expert trajectory with randomly-selected samples in a replay buffer. The denoising process is thus to predict the original trajectory based on random samples from the replay buffer. Since the process happens over experience data, the generated trajectory is assumed (not guaranteed) to be feasible. The method is evaluated on a 2D maze environment and three simulated robotics tasks, with varying difficulties.

The reviewers have raised concerns regarding whether the method can generalize to out of distribution states and whether the generated plans are truly guaranteed to be collision-free. There are also concerns regarding missing reference to closely-related prior research and the lack of real-world experiment. At the same time, the reviewers found the problem of improving the feasibility of plans from a generative model to be well-motivated, and that the presented method to be interesting and sound. 

Based on the reviews and the comments above, I recommend accepting the paper as a poster presentation. Please incorporate the writing and clarification suggestions in the next revision of the paper.
Decision: Accept (Poster)