Adaptive Online Replanning with Diffusion Models
Authors: Siyuan Zhou
Yilun Du
Shun Zhang
Mengdi Xu
Yikang Shen
Wei Xiao
Dit-Yan Yeung
Chuang Gan
Venue: NeurIPS 2023 poster (2023)
Keywords: Decision making
Robotics
Planning-based
TL;DR: We introduce a new data-driven replanning approach based on diffusion models.

Abstract: Diffusion models have risen a promising approach to data-driven planning, and have demonstrated impressive robotic control, reinforcement learning, and video planning performance. Given an effective planner, an important question to consider is replanning -- when given plans should be regenerated due to both action execution error and external environment changes.  Direct plan execution, without replanning, is problematic as errors from individual actions rapidly accumulate and environments are partially observable and stochastic. Simultaneously, replanning at each timestep incurs a substantial computational cost, and may prevent successful task execution, as different generated plans prevent consistent progress to any particular goal. In this paper, we explore how we may effectively replan with diffusion models. We propose a principled approach to determine when to replan, based on the diffusion model's estimated likelihood of existing generated plans. We further present an approach to replan existing trajectories to ensure that new plans follow the same goal state as the original trajectory, which may efficiently bootstrap off previously generated plans.  We illustrate how a combination of our proposed additions significantly improves the performance of diffusion planners leading to 38\% gains over past diffusion planning approaches on Maze2D and further enables handling of stochastic and long-horizon robotic control tasks.

Adaptive Online Replanning with Diffusion Models
Siyuan Zhou', Yilun Du?, Shun Zhang', Mengdi Xu', Yikang Shen', Wei Xiao², Dit- Yan Yeung', Chuang Gan
Hong Kong University of Science and Technology 2 Massachusetts Institute of Technology, 'MIT-IBM Watson AI Lab Carnegie Mellon University, *UMass Amherst
Abstract
Diffusion models have risen as a promising approach to data-driven planning, and have demonstrated impressive robotic control, reinforcement learning, and video planning performance. Given an effective planner, an important question to consider replanning when given plans should be regenerated due both action execution error and external environment changes. Direct plan execution, without replanning. is problematic as errors from individual actions rapidly accumulate and environments are partially observable and stochastic. Simultancously. replanning at each timestep incurs a substantial computational cost, and may prevent successful task execution, as different generated plans prevent consistent progress to any particular goal. In this paper, we explore how we may effectively replan with diffusion models. We propose principled approach to determine when to replan, based on the diffusion model's 's estimated likelihood of existing generated plans. We further present approach to replan existing trajectories to ensure that new plans follow the same goal state as the original trajectory, which may efficiently bootstrap off previously generated plans. We illustrate how combination of our proposed additions significantly improves the performance of diffusion planners leading to 38% gains over past diffusion planning approaches Maze2D, and further enables the handling of stochastic and long-horizon robotic control tasks. Videos can be found on the anonymous website: https: : // //vis- www. umass. edu/repl cita/replandiii ffuser/.
Introduction
Planning is a pivotal component in effective decision-makingg facilitating tasks such as collision avoidance in autonomous vehicles [29], efficient and long-horizon object manipulation [12] and strategic reasoning for extended periods [32, 25]. Recently, diffusion has risen as a promising ap- proach to data-driven planning [17, 14, 40], enabling impressive advancements in robot control [6] and the generation of video-level plans [9].
In many planning settings, an important variable consider is not only how to plan, but also when time to replan. Environments are often stochastic in nature, making previously constructed plans un-executable given unexpected environmental change. Moreover, inaceurate execution of individual actions in an environment will often also cause states to deviate from those in the expected plan, further necessitating replanning. Finally, environments will often be partially observable, and the optimal action or sequence of actions may deviate substantially from those originally planned, especially when information comes to light.
Previous works in planning with diffusion models have not explored how to tackle these factors, and typically generate new plans at each iteration. Such process is computationally expensive as
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Initial Plan
(b) Fast Replanning
(c) Complete Replanning
Figure 1: Online planning procedure. (a) The agent first generates a plan from scratch the start of the task. (b) The agent finds first door inaccessible and replans a new trajectory that deviates slightly from the original plan. (c) The agent finds the yellow door also locked, and replans completely different trajectory from scratch. generated plan in diffusion models requires hundreds of sampling steps. Frequent replanning may also be detrimental to task completion, as rapid switching between plans impedes persistent progress towards single final goal [18]. this paper, we propose a principled approach to adaptively online Replanning with Diffusion Models, called RDM. In RDM, we propose to leverage the internal estimated likelihood of the currently executed plan as a gauge on when is a good time to replan. At the start of plan execution, predicted plan is freshly sampled from the diffusion model and has high estimated likelihood. As the plan gets executed, or when unexpected state changes happen, the underlying transitions expected in the plan get increasingly less likely, and the internally estimated likelihood of the diffusion model tells us to replan.
Given previously generated plan, how do we effectively regenerate new plan? Directly resampling the entire plan can be computationally expensive, as all the previous computations put in the previous plan would be wasted. Furthermore, newly generated plan might erase all the progress previously toward the goal from a previous plan. Instead, based on the feasibility of the current plan being executed, which is estimated by likelihood determined by the diffusion model, we propose to partially replan based on our plan, or fully replan from scratch. We initialize our new plan with the old one and add small amount of noise to the plan before then denoising the added noise. This light amount of denoising serves as a planning refinement operator.
made
We illustrate our online planning procedure in Figure where a learned agent first generates full plan for the goal as in Figure 1a. When the agent reaches the first door, the likelihood of the plan drops, as the agent finds it locked, and the corresponding solution becomes infeasible due to the door's inaccessibility. However, the agent can slightly adjust the initial plan by instead entering the left yellow door and taking a short detour, after which the rest of the plan remains the same. Based on the estimated likelihood of the plan, we may replan and regenerate a new solution based on the previous solution as shown in Figure lb. In another situation where the yellow door is also locked, and there is no nearby solution, our likelihood plummets, telling us that we need to regenerate a new and completely different solution from scratch shown in Figure Ic.
In summary, our contributions are three-fold. (1) We introduce RDM, principled approach to determining when to replan with diffusion models. (2) We illustrate how to effectively replan, using a previously generated plan as a reference.
(3) We empirically validate that RDM obtains strong empirical performance across set of different benchmarks, providing a 38% boost over past diffusion planning approaches on Maze2D and further enabling effective handling of stochastic and long-horizon robotic control tasks.
Background
2.1 Problem Setting
We consider trajectory optimization problem similar to Diffuser [17]. trajectory represented as (Su. op, $1, ... Sr, or), where St, a are the state and action at the f-th time step, and T' is the episode length. The dynamics function of the environment formulated as 81+1 = (8t, ar), where represents the stochastic dynamics. For domains with partial observations, the agent observes an observation og instead of the state 8,. The objective of the trajectory optimization problem is to find sequence of actions ao: that maximizes J (so, do.r), which is the expected sum of rewards

---

Single-step
Diffuser
RDM
marioraace
50-
25
Long-horizon
Stochastic
Robotics Control
Figure 2: Results overview. The performances of RDM significantly outperform the models with single-step execution (i.c. IQI.) and previous planning- based models with diffusion models (i.c. Decision Diffuser) across long-horizon planning (Maze2D), stochastic environments (D4RL. locomotion) and robotic control (RIBencch). The performance metrics are normalized returns for D4RL, tasks (including Maze2D and Locomotion) and success rates for RLBench.
over all the time steps,
arg max J (80, (0:r) = arg max Fosyer
r(s;, 4z).
dp:r
G0:5
Following the Decision Diffuser [1], we plan across state-space trajectories and infer actions through inverse dynamics.
2.2 Planning with Diffusion Models
In Diffuser [17], plan generation is formulated as trajectory generation T through a learned iterative denoising diffusion process po (p(r=1 r") [33]. This learned denoising process is trained to reverse forward diffusion process g(T" r²-1) that slowly corrupts the structure of trajectories by adding noise. Both processes are modeled as Markov transition probabilities:
N
((70:N) = ((79) ((((">>'
po(70:N) = p((
II po po( 1T²).
(1)
Given set of predetermined increasing noise coefficients B1, BN, the data distribution induced is given by:
q(T' N
pa(T'-1 | (No((r'), ,0),
where 7(>") is the data distribution and p(>~) is a standard Gaussian prior.
Diffuser is trained to maximize the likelihood of trajectories through a variational lower bound over the individual steps of denoising
Po(N ((71:N170) se E, (g(r t(rtt
N
Eg(+9) (eog pe (T())) 2 E log
log pa 1
(3)
where trajectory plans may then be generated by sampling iteratively from the learned denoising network for steps. We follow the formulation of planning Decision Diffuser [1], where seek to optimize objective I [(T), we directly learn and sampling from conditional trajectory density (T(T). In terms of notations, we follow Janner et al. [17] and use superscripts E [0, N] to denote diffusion steps (for example, 7'), and use subscripts t [0, T] to denote time steps of the agent's execution in the environment (for example, 8t, dr, etc.).
3 Adaptive Online Replanning with Diffusion Models
In this section, we present RDM, an approach to online replanning with diffusion models. First, we discuss different criteria for determining when solution is problematic and needs replanning. then introduce the estimated likelihood of a planned trajectory as a criterion to determine when to replan in Section 1. In Section 3.2, we discuss how we replan by comparing three distinct replanning strategies and propose an adaptive replanning approach, based on combinations of the two approaches.

---

L likelhoodd
Sz
Sk-11
Sx
Szeb
0.75
Replan from scratch (All diffusion steps)
Replan with future
(Small diffusion steps)
0.25
k Environment Step
Sta
Szah
Su
Sie+1
Seehh
Stash
A dd full steps of Gaussian Noise
Add small steps of Gaussian Noise
Figure 3: Overview of our replanning approach RDM. The first figure shows the likelihood curve of the sampled trajectory the environmental steps increase. There are two steps t and l with low probabilities. Step corresponds to Replan from scratch in the middle figure, where RDM regenerates completely new trajectory based only on the current state and from Gaussian noise. Step & corresponds to Replan with future the right-most figure, where RDM shifts the current state as the first state and repeats the last state to fill the rest. RDM also introduces some timesteps of noise to the trajectory and denoises.
3.1 When to Replan
When the agent sequentially executes plan т = (so, Qo, 01, QT), different factors in the environment such as unexpected interventions or imperfect transitions may cause future states and actions in the plan to deviate substantially from the desired trajectory. We denote with T-sk the trajectory executed in the environment up to time step Formally, T-t = (so, 4g. 81, Al.............................. ................................ 5g. ag, Sk+1..... ........,,yy. ST, ar), where St, are planned states and actions generated by the diffusion model; s, is a state observed in the environment, and ā an action that is computed using the inverse dynamics during the execution process (which is the action transiting to 8141 from 8.). Continuing to execute the current plan will result in more significant discrepancies and potentially catastrophic failures. It may not even be feasible to transit to the next planned state ($K+1) from the current state that the agent at (5k). In such cases, we need to update the planned trajectory based on T-k. process we refer to replanning.
Unfortunately, determining the need for replanning is non-trivial. Directly replanning at a fixed interval is straightforward but inflexible, and may come too late when the environment suddenly changes. can also make the planning algorithm computationally inefficient when the planned trajectory is executed perfectly and does not require replanning. Instead, it is essential to establish criterion that automatically determines when replanning is necessary. An alternative approach is to replan when the distance between the actual state observed from the environment and the desired state in the plan exceeds certain threshold. However, in many cases, even if the actual state the agent is in matches with the state in the plan, new information about the environment may make the current plan infeasible. Thus, is desirable to come up with comprehensive and adaptive approach to determine the appropriate timing for replanning.
In this paper, we propose a novel method that utilizes the inherent estimated likelihood of the trajectory in diffusion models as a criterion for replanning. As the diffusion model is trained on different successful trajectories in an environment, its learned likelihood function implicitly captures the executability of an underlying plan. A feasible trajectory will be assigned high probability, with increasingly less feasible trajectories assigned decreasing probabilities. Thus, when the likelihood assessment of a diffusion model on the plan has low probability, it is good indicator of the need for replanning.
The key challenge is to estimate the likelihood of partially-executed trajectory T Recall that we optimize the variational lower bound defined in Equation 3. We propose to use the same objective trajectory. Intuitively, we assess the degree to
to determine the likelihood of partially-execute which a diffusion model recovers original plan after the plan has been corrupted by Gaussian noise. Concretely, given T we added different magnitudes of Gaussian noise to the trajectory, corresponding to sampling based on the diffusion process q(7" | T->x) a diffusion step We then compute the KI divergence between the posterior q(7"=1 | and the denoising step po(r--1 r²) at the diffusion step i. We estimate likelihood by computing this KI. divergence at several fixed different diffusion steps i, which we discuss in detail in the appendix.

---

Algorithm When to Replan.
1: Input:
Diffusion model pe, Planning horizon H. Episode length T. Threshold la of Replanning from scratch, Threshold i of Replanning with future and 4 L,, Diffusion step N, of Replan from scratch, Diffusion step N2 of Replan with future and Ny < N,, Diffusion steps I for computing likelihood
2: Environment step t
3: Generate a trajectory 7t (St, .e,,.., S14 Ot4
4: for in 0, o,.... do
5: Extract (8r; St- (4)) from Tr
6: Execute and obtain 8441 from the environment
7: Replace the st+1 in Tt.
8:
TO +
9:
Sample T q(T' 79) for all E Compute average likelihood L, = [7 Eiel Dxu(g(T' ~r(p>)))pp( (())
10:
if (t 1)% 0 or Le <= I then
12:
Sample trajectory with Replanning from scratch else if Lt <= le then
14:
Sample new trajectory based previous trajectory with Replanning with future
15: else
Te+1 + Tt end if
17:
end for
Algorithm 2 Replanning from scratch
Algorithm 3 Replanning with future
1: Input: Diffusion model po, Diffusion steps
1: Input: Diffusion model Diffusion pro- cess q. Diffusion steps Ny, Conditions C, Previous sampled trajectory 7
Ns. Conditions C
2: Initialize plan 7 ~ ~ I)
for in Ng...., do
2: NN + 9N,
4: ---1 < pa(
3: for i in Ny...., do
Constraint 7'-1 with conditions
4: T + p
6: end for
5: Constraint --1 with conditions C'
7: Return:
6: end for
7: Return: 70
Given this likelihood estimate, we set a threshold for the likelihood the planning trajectories. the likelihood drops below the threshold, it indicates the current plan deviates significantly from the optimal plan and can cause failure of the plan, which suggests that replanning is necessary. We illustrate this replanning process in Algorithm 1.
3.2 How to Replan
Given a previous partially executed plan executed to time step k, T-*k, and suppose we decide to replan at this current time step using one of the criteria defined the Section 3.1, how do we construct a new plan r'? We consider three separate replanning strategies, which are the following:
1. Replan from scratch. In this strategy, we regenerate a completely new planning trajectory based only on the current state. This involves completely resampling the trajectory starting from Gaussian noise and running all steps (defined as N,) of iterative denoising. Although such an approach fully takes advantage of the diffusion model to generate the most likely trajectory, it is computationally expensive and discards all information in the previously generated plan (Algorithm 2).
Replan from previous context. Instead, we may replan by considering both the current and historical states when executing the previously sampled plan. Given a aartiall-executee trajectory T-ik and current time step we sample new trajectory 7' based on T Specifically, we run N diffusion steps to add noise to T-ik at all the timesteps that are larger than k by sampling from the forward process (T T. We then run N, steps denoising. Here, Np is much smaller than Replan from scratch. We will fix the timesteps of T that are smaller than & to their observed
5

---

value, as we have already executed them in the environment. This approach can be seen as locally repairing a predicted plan, and thus requires a substantially smaller number of diffusion steps. 3. Replan with future context. In a similar way to replan with previous context, in replan with future context, we initialize our new plan 7' based on our partially-executed plan T--k- However, in con- trast to the previous approach, we do not add the previously executed states of the plan. We remove the executed states in T-k and repeatedly add the last predicted state in T- on the right fill in the remaining plan context. Namely, i = ($k, ag. 11++1: Ag+11.... ST, ar, ....... ST , ar), where ST, dr are repeated match the imput length of the diffusion model. To replan, similar to replan from previous context, we add Ny steps of noise to F by sampling from the forward process q(TN/ T=++) and then denoise for Nj steps (Algorithm 3), where Ny is much smaller than N,.
Replanning from scratch is highly time-consuming and inefficient for real-time tasks since it requires all diffusion steps to be run to generate a plan from Gaussian noise. However, in cases of significant interventions, it becomes necessary to regenerate the plan from scratch. Both replanning for future context or past context are efficient and locally repair the previously generated plan. Replanning with past context remains more consistent with the original goal of the previous plan, while replanning for future context further extends the current plan future context.
We empirically find that replanning with future context is a more effective approach. In our proposed approach, based on the estimated likelihood of a current plan, we either replan from scratch replan with future context. We illustrate the psuedocode for our approach in Algorithm 1. We provide a detailed ablation analysis of the different replanning strategies in the experiment section (Figure 7).
4 Experiments
In this section, we empirically use our proposed RDM algorithm for replanning in multiple decision- making tasks (as illustrated in Figure 2) and evaluate the performance of the generated trajectories. We empirically answer the following questions: Q1: In domains with long horizon plans, does RDM effectively and efficiently fix the initial plan? Q2: domains with unexpected interventions, which makes the initial plan no longer feasible during the execution, does RDM efficiently find a feasible plan? Q3: How do when to replan and different replanning strategies affect the agent's performance? In all the experiments below, we report the results using five random seeds.
4.1 Long Horizon Planning
We first evaluate the long-horizon planning capacity of RDM and the baseline methods in the Maze2D environment. In this environment, the large Maze2D domain is especially challenging, as it may require hundreds of steps to reach the target and we found that baseline algorithms struggled to find perfect trajectories given the long horizon and sparse reward setting.
Maze2D is goal-conditioned planning task, where given the starting location and the target location, the agent expected to find a feasible trajectory that reaches the target from the starting location avoiding all the obstacles. The agent receives a reward of 1 when it reaches the target state and receives a reward of otherwise. The starting and target locations are randomly generated, consistent with the setting in the Diffuser [17]. We evaluate replanning using our RDM algorithm and multiple competing baseline algorithms, including model reinforcement learning algorithms such as Batch- Constrained deep Q-learning (BCQ) [1]]. Constrained Q Q-learning (CQL) [22], Implicit Q-learning (IQL) [21], ], and planning- based algorithms like Diffuser [17].
We report the results in Table 1 where numbers for all baselines are taken from Diffuser [17]. As expected, model-free reinforcement learning algorithms (BCQ, CQL, and IQL) struggle to solve long-horizon planning tasks. On the other hand, Diffuser generates a full plan and outperforms these approaches, but does not replan online. As a result, still does not perform well when the size of the domain increases. As the planning horizon increases, it is likely that some of the transitions generated by the diffusion model are infeasible as shown in Figure 4c. We find that our replanning approach, RDM, significantly outperforms all the baseline algorithms. Specifically, it achieves a score of 190% in the Maze2D Large setting, which is almost 63% higher than the Diffuser. We also visualize the sampled trajectory and the executed trajectories in Figure 4. We can observe that infeasible transitions and collisions with the wall will drop the likelihood substantially. RDM is able to find these infeasible transitions and replan accordingly.

---

Likelihood of the sample
RR
R
0.8
LRggthoo
0.4
105
200
225
240
250
280
300
Environmental Stop
(a) Likelihood
(b) Sample
(c) Diffuser
(d) RDM
Figure 4: Visualization on Maze2D. (a) illustrates the likelihood curve of the sampled trajectory as the environmental steps increase, with noticeable drop when the agent collides with the wall. (h) presents the sampled trajectory at the onset of the task. (c) demonstrates the actual trajectory of Diffuser, which follows fixed interval replanning strategy. trajectory, leading it to a less favorable state and resulting in wasted environmental steps. (d) shows the actual trajectory of RDM. In contrast to (c), the agent successfully detects collisions and replans feasible trajectory.
evident that agent does not recognize the flawed part the sampled
Environment
BCQ
CQL
IQL
Diffuser
RDM (Ours)
Maze2D
U-Maze
12.8
5.7
17.4
113.9
133.6 =3.0
Maze2D
Medium
8.3
5.0
31.9
121.5
159.3 =3.1
Maze2D
Large
6.2
12.5
58.6
123.0
185.3 #11.7
Average
9.1
7.7
47.0
119.5
159.4
Multi2D
U-Maze
24.8
128.9
152.1
::2.31
Multi2D
Medium
12.1
127.2
155.1 #4.0
Multi2D
Large
13.9
132.1
195.4 =7.6
Average
16.9
129.4
167.5
Table 1: Long horizon planning. The normalized returns RDM and other baselines on Maze2D. RDM achieves 38% gains beyond Diffuser for all tasks and even 63% gains for large maps.
4.2 Stochastic Environments
Next, we investigate whether RDM can tackle enviromments with stochastic dynamics. We consider stochastic variation of the D4RL locomotion benchmark [10] by adding randomness to the transition function. Stochasticity can make the trajectory optimization problem more challenging, as the executed trajectory can be significantly different from the initially planned trajectory. We omitted the Halfcheetah environment as we found all algorithms performed poorly with added stochasticity. In this setting. consider competing baseline methods including state-of-the-art model-free algorithm. IQL [21]. a Transformer- based RL algorithm, Decision Transformer (DT) [5], and a diffusion model- based algorithm, Decision Diffuser (DD) [1].
As shown in Table although the performance of all the methods drops significantly compared with one without stochasticity, RDM performs comparatively well among all baselines. RDM with replanning strategy is able to detect such stochasticity and quickly regenerate a new plan, resulting in improved performance. We compare DD and RDM using the same number of total diffusion resampling steps.
4.3
Robotic Control
We further evaluate the RDM's capacity in complex domains with visual observations. We consider the RLBench domain [15], which consists of 74 challenging vision-based robotic learning tasks. The agent's observations are RGB images captured by multi-view cameras. The agent controls 7 DoF robotic arm. The domain designed to reflect a real-world-like environment with high dimensionality in both the observation space and the action space, making it challenging solve (Figure 5b).
Most previous works tackle the RLBench tasks by taking actions macro-steps [13], where the agent switches to different macro-step when the state of the gripper the velocity changes. However, such an approach has several limitations: (1) The longest trajectory length achievable using macro steps is relatively short, which is slightly over 30 steps. Such limitation hinders the agent's ability
7

---

Dataset
Environment
IQL
DT
DD
RDM (Ours)
Med-Expert
Hopper
48.9
52.5
49.4
59.4 #3.4
Med-Expert
Walker2d
90.7
90.4
71.1
92.5 12.2
Medium
Hopper
47.6
49.9
45.3
53.2 #122
Medium
Walker2d
71.8
67.9
60.7
70.7 #3.0
Med-Replay
Hopper
55.3
40.8
47.2
57.9
1:3.2
Med-Replay
Walker2d
54.4
52.6
46.4
57.3
13.11
Average
61.5
59.0
53.4
65.2
Table 2: Stochastic environments. The normalized returns of RDM and other baselines Locomotion with randomness, RDM exhibits comparable or superior performance.
Fail to open
Replan and relry
(a) With replanning approach
Fail to open box Continue the plan
(b) Without replanning approach
Figure Comparison between trajectories with without replanning. (a) Our replanning approach allows the agent to detect when the robotic arm fails to open the box and instructs it to retry. (b) Without our replanning approach, the failure goes unnoticed, leading the arm persist in following the previous, unsuccessful plan.
solve long-horizen tasks. (2) Macro-steps are pre-defined, which usually require expert knowledge or demonstrations. (3) Macro-steps constrain the agent's 's action space, which makes it impossible to optimize a finer granularity (for example, avoiding an intermediate collision) and prevent the agent from doing fine-grat fine-grained control. Therefore, we do not use macro-step in our experiments.
We choose the following three tasks for evaluation: open box, close box, and close fridge. The experimental results are in Table 3. We find that baseline algorithms fail to accomplish these three tasks. In contrast, RDM outperforms all the baselines by large margin and achieves rate of 77.3%. We visualize the execution of RDM in Figure 5. RDM successfully detects failures in the initial plan and efficiently replans during the execution.
Environment
BC
IQL
DT
DD
RDM (Ours)
open box
9.3
17.6
9.8
52.9
72.0 47.5
close box
5.9
10.3
7.8
46.0
61.3 17.7
close fridge
13.0
23.1
21.7
84.6
98.6 +2.7
Average
9.4
17.0
13.1
61.2
77.3
Table Robotic control. The success rate of RDM and other baselines for three tasks: open box, close box and close fridge on RLBench. RDM outperforms baselines by large margin.
4.4 Ablative Study
Different Levels of Stochasticity. We investigate how the algorithms respond to different levels of stochasticity by setting different noise values the task Hopper-medi uopppr- where each time step, the transition is the same the deterministic version of the domain with probability 1

---

Normalized Returns
80
IQL
Nomalized Returns
60
70
DD
RDM
60
50
0.01
0.02
Probability
0.03
0.04
DD
SDM
RPDM
RDM
Figure 7: Replan analysis. We analyze replan- ning at fixed intervals (DD), replanning based on state distance (SDM), and replanning based on past context (RPDM). With the same number of diffusion steps Hopper RDM obtains the hest performance.
Figure 6: Different levels stochastic- ity. The normalized returns with different probabilities e of randomness in the task Hopper- medium pordium-exper expeer.. By adaptive plan- ning. RDM more effective than other base- lines.
and is random (as if a random action taken) with probability 6. The results are in Figure We find that RDM consistently outperforms other baselines under all values.
Different Replanning Strategies. To analyze the different replan approaches, we consider the following baselines. (1) Decision Diffuser (DD) replans using two fixed intervals, for deciding when to replan from scratch and when to plan with future context, respectively. (2) Replan on State Distance (SDM) replans when the error between the observed states (5)) and desired states in the plan (s,) exceeds threshold. It replans with the future context. The following two algorithms determine when to plan according to the diffusion model likelihood. (3) Replan from previous context (RPDM) replans with on past context, and (4) RDM replans with future context. Figure 7, we find that SDM and RPDM both have better performance than DD, among which RDM achieves the best performance, All methods use the same number total diffusion steps.
We provide additional ablation studies and visualization analysis in the Appendix.
5 Related Work
Diffusion Models for Decision Making. The use of diffusion models for decision-makinn tasks has
seen increased interest in recent years [17, 9, 6, 30, 40, 23, 14, 38, 24] and have been applied to tasks such planning [17, IJ. robotic control [6]. behavior cloning [30], and video generation [9]. In [17, 1] introduce an approach to planning with diffusion models, but both methods typically replan at each iteration. In this paper, propose a method to adaptively determine when to replan with diffusion models and illustrate different approaches to replan.
Model-Based Planning. Our work is also related to existing work in model-based planning [35, 4, 19, 3, 36, 2, 27, 34, 8, 37, 28]. For domains with finite states, traditional methods use dynamic programming to find an optimal plan, or use Dyna-algorithms to learn the model first and then generate a plan [35]. For more complex domains, a learned dynamies model be autoregressively rolled out [7, 19], though gradient planning through these models often lead to adversarial examples 36, 2]. To avoid this issue approaches have tried gradient-free planning methods such as random shooting [27], beam search [34], and MCMC sampling [4]. Our work builds on recent trend in madel-based planning which folds the planning process into the generative model [16].
6 Limitations and Conclusion
Conclusion. In this paper, we propose RDM, an adaptive online replanning method, with a principled approach to determine when to replan and an adaptive combined strategy for how replan. We demonstrate RDM's remarkable proficiency in managing long-horizon planning, unpredictable dynamics, and intricate robotic control tasks. An interesting line of future work involves investigating the scalability of RDM to more complex and larger-scale problems and extending to real robot tasks and autonomous driving.
Limitations. There are several limitations to our approach. Firstly, the effectiveness of planning and replanning is heavily dependent on the generative capabilities of the diffusion models. If the diffusion

---

model is unable to accurately represent the current task, its plans may not be viable and RDM may fail to identify problematic trajectories. Secondly, despite RDM's significant improvement on the performance of previous diffusion planning methods, we observed that tikelihood-based replanning was insensitive to some planning failures. This highlights the potential benefit of combining likelihood- based replanning with other indicators for replanning.
Acknowledgment
This work has been made possible by Research Impact Fund project (R6003-21) funded by the Research Grants Council of the Hong Kong Govermment.
References
[1] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 2022.
[2] K. Asadi, D. Misra, and Littman. Lipschitz continuity in model-based reinforcement learning.
In International Conference on Machine Learning, 2018.
[3] Bengio, O. Vinyals, N. Jaitly, and Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, 2015.
[4] H. Chen, Y. Du, Chen, Tenenbaum, and P. Vela. Planning with sequence models through iterative energy minimization. arXiv preprint arXiv:2303. 16189, 2023.
[5] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084-15097, 2021.
[6] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 20233 2023.
[7] S. Chiappa, S. Racaniere, D. Wierstra, and S. Mohamed. Recurrent environment simulators. arXiv preprint 17Xh::104.00225. 2017.
[8] Y. Du, Lin, and Mordatch. Model based planning with energy based models. In Conference on Robot Learning. 2019.
[9] Y. Du, M. Yang, Dai, H. Dai, O. Nachum, B. Tenenhaum, Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. arXiv e-prints, pages arXiv-2302, 2023.
[10] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004 (0ti:2200400219, 2020.
[11] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[12] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, Silver, L. P. Kaelbling, and T. Lozano-Péres Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:365-293, 2021.
[13] P.L. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driveen history-aware policies for robotic manipulations. In Conference on Robot Learning, pages 175-187. PMLR, 2023.
[14] S. Huang, Z. Wang, P. Li, B. Jia, T. Liu, Y. Zhu, W. Liang, and S.-C. Zhu. Diffusson- based generation, optimization, and planning in 3d scenes. arXiv preprint arXiv:2301.0600 2023,
[15] James, Z. Ma, Arrojo, and A. Davison. Rlbench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3322 2020.
10

---

[16] M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. International Conference on Machine Learning, 2022.
[17] M. Janner, Y. Du, J. B. Tenenbaum, and Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint vrXiv:2205.099)) 2022.
[18] L.. P. Kaelbling and T. Lozano-Péred Hierarchical task and motion planning in the now. In 20/ IEEE International Conference on Robotics and Automation, pages 1470-1477. IEEE, 2011 1.
[19] N. R. Ke, A. Singh, A. Touati, A. Goyal, Y. Bengio, D. Parikh, and D. Batra. Modeling the long term future in model-based reinforcement learning. International Conference on Learning Representations, 2018.
[20] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[21] I. Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021.
[22] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
[23] Z. Liang, Y. Mu, M. Ding, F. Ni, M. Tomizuka, and P. Luo. Adaptdiffuser: Diffusion models as adaptive self- evolvvvol planners. arXiv preprint arXiv:2302.01887, 2023.
[24] W. Liu, Y. Du, T. Hermans, S. Chernova, and C. Paxton. Structdiffusion: Language-guided creation of physically-valid structures using unseen objects. arXiv preprint arXiv:2211 1.12221111004 2022.
[25] S. Maharaj, N. Polson, and A. Turk. Chess ai: competing paradigms for machine intelligence. Entropy, 24(4):550, 2022.
[26] D. Misra. Mish: A self regularized non-monotonic neural activation function. In British Machine Vision Conference, 2019.
[27] A. Nagabandi, G. Kahn, R. Fearing, and Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559-7566. IEEE, 2018
[28] Oh, S. Singh, and H. Lee. Value prediction network. Advances in neural information processing systems, 30, 2017.
[29] B. Paden, M. Čáp. S. Z. Yong, D. Yershov, and E. Frazzoli. survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on intelligent vehicles, 1 (1):33-55, 2016.
[30] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, et al. Imitating human behaviour with diffusion models, arXiv preprint arXiv:2301..066 2023.
[31] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018
[32] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[33] J. Sohl-Dickstein, F. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning. pages 2256-2265 PMLR, 2015.
[34] Sun, D.-A. Huang, B. Lu, Y.H. Liu, B. Zhou, and A. Garg. Plate: Visually-grounded planning with transformers procedural tasks. IEEE Robotics and Automation Letters, 7(2):4924 4930,, 2022.

---

[35] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[36] E. Talvitie. Model regularization for stable sample rollouts. In UAI, pages 780-789, 2014.
[37] A. Tamar, Y. Wu, G. Thomas, S. Levine, and Abbecl. Value iteration networks. Advances in neural information processing systems, 29, 2016.
[38] Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offiine reinforcement learning. arXiv preprint arXiv:2208. 06193, 2022.
[39] Y. Wu and K. He. Group normalization. European Conference on Computer Vision, 2018.
[40] Zhang, Lu, W. Wang, and A. Zhang. Lad: Language augmented diffusion for reinforcement learning. arXiv preprint arXiv:2210./500 2022.
12

---

G
G
Sut
(a) Problematic Sample
(b) = 40
(c) N 80
(d) N 256
Figure 8: Samples replanning with different difflusion steps. (a) demonstrates the problematic trajectory. (b) presents the sampled trajectory after replanning with N 40. The sampled trajectory still leads to the collision. (c) presents the sampled trajectory after replanning with N = 80. The trajectory becomes fcasible and avoids collision. (d) shows the sampled trajectory after replanning with N 256. The trajectory improves quality but needs much more time.
In the supplementary, we first discuss the experimental details and hyperparameters in Section A. Next, we analyze the impact of different numbers of diffusion steps N the replanning process in Section B, and further present the visualization in RLBench in Section C. Finally, we discuss how to compute the likelihood in Section D.
A Experimental Details
1. Our architecture built based on Diffuser [17] and Decision Diffuser [1]. In detail, our architecture comprises temporal U-Net structure with six repeated residual networks. Each network consists of two temporal convolutions followed by GroupNorm [39], and a final Mish nonlinearity [26]. Additionally, We incorporate timestep and conditions embeddings, which are both 128-dimensional vectors produced by MLP, within each block.
In RLBench, we exploit the image encoder from the pre-trained CLIP [31], followed by a 2-layered MLP with 512 hidden units and Mish activations.
The model is trained using Adam optimizer [20] with learning rate of 20-04 04 and a batch size of 64 for 1e6 training steps.
4. The planning horizon is set to 128 in Maze2D//Multi2D U-Maze, 256 in Maze2D//Mult2DD Medium, 256 in Maze2D//Mult2DD Large, 64 in Stochastic Environments, and 64 RLBench.
5. We a threshold of 0.7 for Replan from scratch and threshold of 0.5 for Replan with future.
6. The probability e of random actions is set to 0.03 in Stochastic Environments.
The diffusion steps for computing likelihood set to (5, 10, 15) Maze2D and Stochastic Environments and (10, 20, 30} in RlBench.
8. The total number of diffusion steps, corresponding to the number of diffusion steps for Replan from scratch set to 256 in Maze2D, 200 in Stochastic Environments, and 400 in RLBench. And the number of diffusion steps for Replan with future set to 80 in Maze2D, 40 in Stochastic Environments, and 100 in RLBench.
9. We perform the whole experiment with total of three Tesla V100 GPUs.
Environment
Diffuser
RDM (N = 40)
RDM (Ours)
RDM (N = 256)
Multi2D Large
129.4
160.9
195.4
197.7
Table 4: Replanning with different diffusion steps. RDM with N = 40 performs better than Diffuser. Our method RDM with N 80 further improves performance and is much more efficient than RDM with N 256.
B Different Diffusion Steps for Replanning
In this section, we visualize the impact of using different numbers diffusion steps to replan an existing trajectory. Figure illustrates problematic sampled trajectory after exccution. When
13

---

G
G
SDE
(a) Problematic Sample
(b) N== 40
(c) N 80
(d) N 256
Figure 9: Samples replanning with different difflusion steps. (a) demonstrates the problematic trajectory. (b)(c) presents the sampled trajectory after replanning with 40 and 80. The previously sampled trajectory so troublesome that it can't recover from fast replanning. So the sampled trajectories still lead to the collision. (d) shows the sampled trajectory after replanning with N 256. After Replan from scratch, the resampled trajectory becomes feasible.
Fail to clase
Replan and reiry
With replanning approach
Fail to cloe box
Continue the plan
(b) Without replanning approach
Figure 10: Comparison between trajectories with and without replanning. (a) Our replanning approach allows the agent to detect when the robotic arm fails to close the box and instructs it to retry. (b) Without our replanning approach, the failure goes unnoticed, leading the arm persist in following the previous, unsuccessful plan.
N 40 of replanning, the resampled trajectory still leads to the collision with the wall. However, with N 80, which aligns with the number of diffusion steps used in Replan with future, we observe that the sampled trajectory becomes feasible. For N 256, corresponding the number of diffusion steps used in Replan from scratch, the resampled trajectory shows additional improvement, but such a procedure is substantially more expensive than our replanning procedure.
We illustrate another example plan which based off likelihood would need to be planned from scratch in Figure 9. After replanning with N 40 and N 80, the trajectories still encounter issues with colliding with the wall. However, With N = 256, RDM successfully regenerates completely different and feasible trajectory through replanning from scratch.
We further evaluate the performance with different replanning steps in Table 4. The results clearly demonstrate that that RDM consistently outperforms Diffuser in all cases. Remarkably, RDM with N 80, corresponding to Replan with future, achieves performance close to RDM with = 256, which corresponds Replan from scratch which is substantially more expensive.
C Comparison between trajectories with and without replanning
In this section, visualize the execution in the task close box in Figure 10. In the second frame of Figure 10, we observe that the robotic arm fails to close the box due insufficient contact between the gripper and the lid. In Figure 10a, RDM successfully detects the failures and retries to establish
14

---

2.5
KL Divergence
3
KL Diveergnnee
2.0
KL Divergence
1.5
1.0
N
0.5
100 150
200
250
300
250
100 200 Environmental Stop
300
Environmental Step
Environmental Step
(a) i 60
(b) 30
(c) = 15
Figure 11: Different diffusion steps i for computing KL divergence, We estimate the likelihood of trajectory
by computing KL divergence between q(r r"), T-n1) and (o(T-11 I'). Here, we illustrate the curve of KL divergence at different environmental steps in Maze2D where the number of total diffusion steps is 256. (a) the KL divergence
When large amounts of noise are added at 60, the agent fails to detect any infeasibility remains low and consistent. (b) When intermediate amounts noise are added at = 30, the agent has the ability to notice infeasibility but the value of KL divergence is similar to that of feasible trajectories. (c) When small amounts of noise are added at i 15, the agent can effectively discern infeasibilities, as the KL divergence exhibits a clear boundary between feasible and infeasible trajectories.
proper contact with the lid. As a result, the gripper successfully closes the box this time. In contrast, Figure 10b demonstrates the scenario where the arm fails to notice the failures, making the arm persist in the unsuccessful plan. These results demonstrate the capability of RDM to detect failures and replan successful trajectory in challenging robotic control tasks.
D How to Compute Likelihood
In this section, we analyze the number of noising steps we should use to estimate the likelihood. The likelihood estimation is performed by computing the KL divergence between g(r'=1 | and po(r"=1 r²) with different magnitudes of noise added at diffusion timesteps i. We analyze the impact of different diffusion steps and the corresponding magnitude of noise added for computing KL divergence in Figure 11. We observe that (1) When large amounts of noise are added at = 60 the agent fails detect any infeasibility, as the KL divergence remains low and consistent. (2) When intermediate amounts of noise are added at = 30, the agent has the ability to notice infcasibilities, but the KL divergence value similar to that of feasible trajectory. (3) When small amounts of noise are added at i = 15, the agent can effectively discern infcasibilities, as the KL divergence exhibits clear boundary between feasible and infeasible trajectories. Based on these findings, we choose smaller values of i for computing the likelihood of trajectories and estimating when to replan.
T-*)
E Additional Experimental Results
We show the additional experimental results in the following.
1. We show the results of the comparison between our RDM algorithm and the replanning- based baseline algorithms (SDM and PRDM) across more tasks. We run experiments using different intervals or thresholds and measure their computational costs by the total number of diffusion steps. Notably, RDM consistently outperforms all the baseline algorithms under the same total diffusion steps (that is, with the same computational budget).
2. We investigate different intervals for replanning for Diffuser and Decision-Diffuser. We observe that as the interval decreases (that is replanning is done more frequently), the performance improves as we expect. However, when the interval smaller than a certain value, the performance decreases significantly. For example, in the Maze2D Large domain shown in Figure VI, the return increases when the interval decreases from 250 to 100, while it drops when the interval decreases from 100 to 1. The ablation results confirm our statement that replanning with an interval that is too small (for example, replanning at every time step) may prevent successful task execution.
15

---

3. We also analyze the impact of different thresholds / for the baseline algorithms. SDM and RDM. As we expect, when decreases (that is, when replanning is done more frequently). the performance of all the methods has improved. Despite the performance differences of the baseline algorithms, we want to emphasize again that under the same computational budget, our RDM algorithm outperforms all the baseline algorithms.
16

---

Maze2D
MultiDD Large
Hopper Med-Expert
145
Nomellzee Reture
380
Commlllee Resuens tetten
Nermalized Returs
163
270
150
50M
160
5DM FROM
SDM
PRD2
7380
130
13
14
18
1.0
124
Diffusion
100
121 1-0
2.00 Step.
7.25
2.0
Figure 12: Performance To- Figure 13: Performance To- Figure 14: Performance vs. To-
tal Diffusion Steps on Maze2D tal Diffusion Steps on Multi2D tal Diffusion Steps on Hopper Large.
Large.
Medium-Expert.
Walker2d Med-Ent Med-Expert
95.
Close Box
Maze2D Large
170
80- 85-
Nommel 1odd Retumns
8
Nomalizd Ratums
Iso
150
Normallee
144
53-
130
PRIM RDM
3D-M
map
120
110
Total Difusion Stop
14
18
Diflusion Step
17
1.6
Interval
Figure 15: Performance vs.
To-
Figure 16: Performance Total
Figure 17: Different intervals for Diffuser Maze2D Large.
tal Diffusion Steps on Walker2d Medium Medium-Expert.
Diffusion Steps on Close Box.
Hopper Med-Expert
Close Box
Maze20 Large
Normaltzed Rotume
56
kormalles Returts
Normalized Returns
8o-
50-
20
150
909
10-
60
125 State Distance Threshold
25
Interval
Threshald
Figure 20: Different thresholds for SDM Hopper Medium- Expert.
Figure 18: Different intervals
Figure 19:
Different thresholds
Decision Diffuser on Close Box,
for SDM on Maze2D Large.
Maze2D Large
Close Box
Hopper Medium
8 B d
Normalzze Returns
Normalzeo Retumss
Retums
180
3CM
370
81
50
660
150
Replain Libeshood Threshald 30-3
02
Replan
0.4
02
03
1.0
10
15
Probability
5.0
335
60
Figure 21: Different thresholds for RDM Maze2D Large.
Figure 22: Different thresholds for RDM on Close Box.
Figure 23: Different levels of stochasticity on Hopper Medium.
S
(a) interval=255
(b) interval=50
(c) interval=1
Figure 24: Visualization for Diffuser with different intervals. Replanning too frequently will cause some sub-optimal plans.
17


Reviews:
[{"review_id": "kecQVcLWVE", "reviewer": "Reviewer_G16t", "summary": "This paper studies how to effectively replan with diffusion models. The authors propose an adaptive online replanning strategy using diffusion models. This strategy employs the estimated likelihood of a plan's success to determine when replanning is needed, avoiding frequent, computationally expensive replanning. It also ensures new plans align with the original trajectory's goal, leveraging previously generated plans efficiently. This method led to a 38% performance improvement over previous diffusion planning approaches on Maze2D and enabled the handling of stochastic and long-horizon robotic control tasks.\n\nMain Contributions:\n\n1. A method to determine when to replan with diffusion\n2. A method to generate new plans while utilizing the existing plan", "questions": "- It would be great to provide the number of \"replan from scratch\" and the number of \"replan with future context\" used in each experiment, since this would help the audience further understand how the replanning helps the agent.\n- Why does \"replan with future context\" work better than \"replan from previous context\"? Could you give any intuitive explanations?\n- See the weakness section for other questions.", "limitations": "- This method only applies to diffusion models as it adds noises at some diffusion steps. It is unclear how to utilize this method in other planning methods.\n- See the weakness section for other limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Movitation**: Online re-planning is a natural and important complement to many planning methods, thus the motivation of this work is important.\n\n- **Methodology**: This paper proposes two methods: one is used to determine when to replan while the other one is for how to replan. The two methods worked together to conduct effective online replanning with diffusion models. As far as I know, the proposed methods are novel.\n\n- **Performance Improvement**: The proposed method significantly improves the performance of diffusion planners, with a reported 38% gain over past diffusion planning approaches on Maze2D. And the authors also showcased that the proposed method enable the handling of stochastic and long-horizon robotic control tasks.", "weaknesses": "- It is still unclear to me why replaning at every time step does not work. In the figure 7, the Dcesion Diffuser replans at fixed intervals and it performs much worse than the proposed method. However, the authors did not mention how large is the interval is. If the replaning frequency is very low than it should apprently works worse than the proposed method. And in the abstract, the author mentioned that \"replanning at each timestep may prevent successful task execution, as different generated plans prevent consistent progress to any particular goal\". I am not convinced by this argument without any exmaples or further explanations.\n\n- It is also unclear to me why \"Replan on State Distance (SDM)\" works so badly according to Figure 7. In Section 3.1, the authors mentioned that \"However, in many cases, even if the actual state the agent is in matches with the state in the plan, new information about the environment may make the current plan infeasible.\" This does not make too much sense to me. I appreciate it if the authors could provide further details about this point.\n\n- Computation budget is an important factor to consider when comparing different methods. (If we have an unlimited computation budget, I feel it is a good idea to replan at each timestep.) However, the numbers of replanning used in each baseline are not present in the paper. If the numbers of replanning vary a lot among different methods, then I would not think it is a fair comparison. I wish the authors could provide the numbers of replanning used in each method. \n\n- Although the proposed method aims to avoid frequent, computationally expensive replanning, it's not clear how computationally efficient the new method is, and whether the computational cost of determining when to replan offsets the benefits gained from less frequent replanning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe appreciate the reviewer for the detailed comments and insightful suggestions.\n\n**Q1. About replanning at every time step.**\n> It is still unclear to me why replaning at every time step does not work.\n\nWe investigate different intervals $I$ for replanning for Diffuser and Decision-Diffuser. The results are shown in Figure VI and VII. We observe that as the interval decreases (that is replanning is done more frequently), the performance improves as we expect. However, when the interval is smaller than a certain value, the performance decreases significantly. For example, in the Maze2D Large domain shown in Figure VI, the return increases when the interval decreases from $250$ to $100$, while it drops when the interval decreases from $100$ to $1$. The ablation results confirm our statement that replanning with an interval that is too small (for example, replanning at every time step) may prevent successful task execution.\n\n\n**Q2. Performance of SDM.**\n> It is also unclear to me why \"Replan on State Distance (SDM)\" works so badly according to Figure 7.\n\nThanks for the question. In our rebuttal PDF, we compare RDM and SDM in most tasks in Figure I-V and observe that RDM performs better than SDM given the same total diffusion steps. The reason might be that the distance is only computed on a state level, not on a trajectory level. For example, in Figure 1 of the paper, the agent follows the initial plan perfectly. However, it finds the first door inaccessible only when it reaches in front of the door. Although the current observed state is close to the planned state, it still needs to replan as the current plan is infeasible.\n\nAnother example is shown in Figure 5. The agent tries to open a box, but it fails to open the box although it follows the initial plan perfectly. In this case, the state distance between the current state and the planned state is still small. If we use Replan on State Distance (SDM), the agent would not replan. On the other hand, RDM computes the likelihood based on the current environment observation and finds that the likelihood of the original plan is low (since the box doesn't open). It will replan although the current state appears similar to the planned state.\n\n\n**Q3. Computation budget (the number of replanning).**\n> I wish the authors could provide the numbers of replanning used in each method. \n\n> It would be great to provide the number of \"replan from scratch\" and the number of \"replan with future context\" used in each experiment\n\nThanks for the suggestion. We show the results of the comparison between our RDM algorithm and the replanning-based baseline algorithms (SDM and PRDM) across more tasks. We run experiments using different intervals or thresholds and measure their computational costs by the total number of diffusion steps. In the table below, we choose one task from each of the three domains and report the total diffusion steps and the performance, and show more evaluation results in Figure I-V in the rebuttal PDF. Notably, RDM consistently outperforms all the baseline algorithms under the same total diffusion steps (that is, with the same computational budget).\n\nTable A. The total diffusion steps and performance.\n|Environment | Method   | Total Diffusion Steps | Normalized Returns |\n| ----------------------------------------- | -------- | --------------------- | ------------------ |\n| Maze2D Large             | Diffuser | 2304.0     | 165.9 (±3.8)        |\n|          | SDM      | 1925.1   | 175.0 (±5.3)        |\n|          | PRDM     | 1916.4     | 169.3 (±1.4)        |\n|      | RDM      | 1894.38      | 185.4 (±3.0)        |\n| Hopper Medium-Expert                      | DD       | 17100   | 47.4 (±3.6)         |\n|        | SDM      | 18500    | 54.4 (±3.5)         |\n|     | PRDM     | 16200     | 57 (±4.3)           |\n|      | RDM      | 15900       | 59.7 (±3.8)         |\n| Close Box              | DD       | 1968      | 46.0 (±7.5)         |\n|    | SDM      | 1813    | 52.4 (±6.6)         |\n|      | PRDM     | 1653           | 50 (±7.2)           |\n|     | RDM      | 1600          | 63.5 (±7.0)         |\n\n**Q4. Replan with future context.**\n> Why does \"replan with future context\" work better than \"replan from previous context\"? Could you give any intuitive explanations?\n\nThanks for the question. \"Replan from preview context\" serves as a baseline method to confirm that, during the execution of a plan, replanning by conditioning on the past states does not help generate better plans. On the other hand, including only the future states starting from the current state to the input to the diffusion model (our design choice in \"replan with future context\") generates the best plan. The reason might be that past states can be distracting to the diffusion model and do not help generate a better plan.\n\n**Q5. Limitations.**\n> This method only applies to diffusion models as it adds noises at some diffusion steps. It is unclear how to utilize this method in other planning methods.\n\nOur replanning strategy, based off likelihood, can be applied to any diffusion-based planner, which has seen a variety of different applications across different planning settings such as robotics policies [1] and video [2]. Broadly, we believe the idea of likelihood-based replanning can also be applied to many other likelihood-based planning methods (e.g. trajectory transformer[3]). \n\n[1] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. arXiv preprint arXiv:2303.04137, 2023.\n\n[2] Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, Pieter Abbeel. Learning Universal Policies through Text-Conditioned Video Generation. arXiv preprint arXiv:2302.00111, 2023.\n\n[3] Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem. arXiv preprint arXiv:2106.02039, 2021."}, {"review_id": "AXz9WILfty", "reviewer": "Reviewer_kWoD", "summary": "This paper introduces Replanning with Diffusion Models (RDM), which utilizes an internally estimated likelihood of the current plan to determine when to perform replanning. The authors propose various strategies for replanning in different scenarios.", "questions": "1. I am concerned about the computational resource requirements introduced by the Diffusion Models. For achieving similar performance levels, could you provide insights into the training time and resources needed for IQL, DT, and RDM?\n2. While I understand the advantages of replanning, which reduce unnecessary exploration, in RL learning, accurate judgments cannot be made without visiting certain states. How can you ensure that the states not visited after replanning are indeed redundant for learning?\n3. The direct introduction of noise into the D4RL dataset (Sec. 4.2) seems unreasonable. Why not collect data directly from random environments to validate this point?\n", "limitations": "\nN/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of Replanning with Diffusion Models (RDM) and the use of an internally estimated likelihood of the current plan for replanning strategies is novel.\n2. The experimental results demonstrate impressive performance in robot control and stochastic environments, surpassing baselines like IQL.", "weaknesses": "1. The validation of the proposed method is not fully comprehensive. In RDM, a key aspect is the estimation of likelihood, but the authors did not discuss the accuracy of the estimation or its impact on final performance.\n2. The paper contains some language issues and typos that should be carefully reviewed. For example, in line 178, it should refer to Figure 7 instead of Table 7. Additionally, in line 37, \"we propose a principled approach to xxx\" has a grammatical error.\n3. In the robot control experiments using the RLBench domain, comparing IQL or DT as baselines might not be sufficient. It would be beneficial to compare against other agents known for performing well in the RLBench domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe thank the reviewer for the comments and insightful suggestions.\n\n**Q1. Estimation of likelihood.**\n> In RDM, a key aspect is the estimation of likelihood, but the authors did not discuss the accuracy of the estimation or its impact on final performance.\n\nThe accuracy of the estimation of likelihood depends on the number of diffusion steps to compute the likelihood. We run 3 diffusion steps in the experiment. In the table below, we compare the performance under different diffusion steps of computing. We find that 3 steps are sufficient to estimate the likelihood accurately. While using more diffusion steps indeed improves the normalized returns, the improvement is marginal.\n\nTable B. Comparison under different diffusion steps of computing likelihood.\n| Different diffusion steps | 1 | 3 (Ours) |     9   | 15 |\n| --------- | --------- | --------- | --------- | --------- |\n| Normalized Returns    | 179.1 (±4.9) | 185.4 (±3.0)  | 187.0 (±4.0) | 187.2 (±2.2) |\n\n**Q2. RLBench baselines.**\n> It would be beneficial to compare against other agents known for performing well in the RLBench domain.\n\nThanks for the suggestion. Note that our method plans using preliminary actions in RLBench. As we have discussed in Line 233-238 in the paper, most other algorithms evaluated on RLBench use *micro-steps*, which do not plan using preliminary actions and cannot be directly compared with our method. To the best of our knowledge, IQL and DT are state-of-the-art offline RL algorithms that plan using preliminary actions in RLBench. So we use them as baselines in our paper.\n\n**Q3. Training time.**\n> For achieving similar performance levels, could you provide insights into the training time and resources needed for IQL, DT, and RDM?\n\nThanks for the suggestion. We list the training time in the table below and each model is trained with only one Tesla V100 GPU. It's hard to report the training time with exactly the same performance level. So we instead report the performance under similar training times in the table below. Our model with 4 hours of training time still outperforms IQL and DT.\n\nTable C. Training Time of all models on Close Box.\n| Model           | IQL  | DT    | RDM*       | RDM          |\n|-----------------|------|-------|-----------|--------------|\n| Training Time (hours)   | 3.3 | 4    | 4        | 8           |\n| Performance     | 10.3 | 7.8   | 58.3 (±8.3)| 61.5 (±7.7)   |\n\n\\* We use an earlier checkpoint that has a similar computation time to the baseline algorithms.\n\n\n**Q4. Exploration.**\n> How can you ensure that the states not visited after replanning are indeed redundant for learning?\n\nThanks for the insightful question. We can agree that in general, we can not ensure that the states not visited will be redundant for learning. One way to integrate exploration into our planning procedure is to measure the uncertainty of the state in a similar way to RND [1] and then determine the timing for replanning based on both the likelihood and the uncertainty of the unvisited states. We leave it as future work and will add this to the discussion of the paper.\n\n[1] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.\n\n\n**Q5. Stochastic Environments.**\n> The direct introduction of noise into the D4RL dataset (Sec. 4.2) seems unreasonable. Why not collect data directly from random environments to validate this point?\n\nWe believe that in practice, in many environments, there is unexpected noise in control, due to either an inaccurate controller or external environment perturbances and believe that the addition of stochasticity into the D4RL environment serves to represent this. We are happy to run additional evaluations on an environment with random environments in the final version of the paper.\n\n\n**Q6. Language issues and typos.**\n> For example, in line 178, it should refer to Figure 7 instead of Table 7. Additionally, in line 37, \"we propose a principled approach to xxx\" has a grammatical error.\n\nThanks for pointing these out. We will address them in the revision."}, {"review_id": "B4c8yWsTqa", "reviewer": "Reviewer_rt81", "summary": "The manuscript introduces a technique for enhance the motion planner rooted in diffusion models, encompassing decisions concerning the timing of replanning and plan trajectories upon the existing path. The strategy for timing replanning utilizes the inherent estimated likelihood of the trajectory in diffusion models as a criterion for replanning. In terms of forming new trajectories, the authors either completely reconstruct the plan or modify it based on future contexts determined by specific guidelines. The concept is clear-cut, and the outcomes appear to be quite promising.", "questions": "- For long horizon planning tasks and robotic control tasks, is using likelihood better than using state distance deviation as replanning criteria?\n- Comparing baselines with different replanning timing decision methods and RDM, how much efficiency improvement will RDM achieve?\n- For stochastic environment, the advantage of methods implementing replanning strategy comparing to non-replan methods should be increased when stochasticity arises, but the performance gets closer shown on Figure 6. Why?", "limitations": "The author addressed the limitations well. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Replanning is important for robots to execute trajectories robustly, deciding when to replan and replanning algorithm are the core challenges of replanning problem. Tackling these problems can help a lot with diffusion based robot motion planning methods. Moreover, the overall writing quality is good, and the core idea is presented well. ", "weaknesses": "It's not surprise that the performance gets better by adding replanning strategy into diffusion model. For long horizon planning problem in Section 4.1, the authors compared performance of RDM and baselines, however, none of the baselines contains replanning strategy, making the effectiveness of replanning timing decision method proposed by the author hard to evaluate. This also happens to Robotic Control tasks in Section 4.3. \n\nAs for stochastic environment in Section 4.2, the author demonstrates that RDM outperforms baseline models in environment with stochastic transition models. How different levels of stochasticity will affect the performance is later proved in Section 4.4. However, only one environment is tested, and the performance of RDM and other baselines are relatively close, which is inadequate to evaluate the effectiveness of replanning strategy in RDM.\n\nIn Section 4.4, the author compares the performance for models using different replanning strategy, and comparison of different fixed interval/different threshold of state distance deviation is missing, which makes the performance of DD/SDM not convincing enough.\n\nThere are also some points which are not demonstrated very well:\n- The detailed implementation of how the replanned trajectory is generated is missing.\n- The theoretical proof of the effectiveness of likelihood function for partially-executed trajectory is missing, and the threshold of likelihood to choose how to replan is selected empirically which may significantly impact the performance. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe thank the reviewer for the detailed reviews and insightful suggestions.\n\n**Q1. Replanning baselines.**\n> For long horizon planning tasks and robotic control tasks, is using likelihood better than using state distance deviation as replanning criteria?\n> \n> Comparing baselines with different replanning timing decision methods and RDM, how much efficiency improvement will RDM achieve?\n\nWe show the results of the comparison between our RDM algorithm and the replanning-based baseline algorithms (SDM and PRDM) across more tasks. We run experiments using different intervals or thresholds and measure their computational costs by the total number of diffusion steps. In the table below, we choose one task from each of the three domains and report the total diffusion steps and the performance, and show more evaluation results in Figure I-V in the rebuttal PDF. Notably, RDM consistently outperforms all the baseline algorithms under the same total diffusion steps.\n\nTable A. The total diffusion steps and performance.\n|Environment | Method   | Total Diffusion Steps | Normalized Returns |\n| ----------------------------------------- | -------- | --------------------- | ------------------ |\n| Maze2D Large                              | Diffuser | 2304.0    | 165.9 (±3.8)        |\n|        | SDM      | 1925.1        | 175.0 (±5.3)        |\n|         | PRDM     | 1916.4                | 169.3 (±1.4)        |\n|              | RDM      | 1894.38               | 185.4 (±3.0)        |\n| Hopper Medium-Expert                      | DD       | 17100        | 47.4 (±3.6)         |\n|          | SDM      | 18500                 | 54.4 (±3.5)         |\n|             | PRDM     | 16200                 | 57 (±4.3)           |\n|                | RDM      | 15900                 | 59.7 (±3.8)         |\n| Close Box                                 | DD       | 1968                  | 46.0 (±7.5)         |\n|            | SDM      | 1813                  | 52.4 (±6.6)         |\n|                 | PRDM     | 1653                  | 50 (±7.2)           |\n|              | RDM      | 1600                  | 63.5 (±7.0)         |\n\n\n**Q2. Stochasticity levels.**\n> However, only one environment is tested, and the performance of RDM and other baselines are relatively close, which is inadequate to evaluate the effectiveness of replanning strategy in RDM.\n\nWe show the comparative results about different levels of stochasticity in Figure XII in our rebuttal PDF. The reason why all planning-based methods do not perform very well is that randomness will sometimes make the agent reach the out-of-distribution states, which leads to the performance drop.\n\n**Q3. Unclear points.**\n> The detailed implementation of how the replanned trajectory is generated is missing.\n\nWe have presented how to replan in Section 3.2 and provided the pseudocode in Algorithm 3. It requires a partially executed plan, $\\tilde{\\tau}$, as input. It adds noises to it by running the forward process of the diffusion model for $N_f$ steps (where $N_f$ is a pre-defined parameter), and then denoise the trajectory by running the denoising step of the diffusion model for $N_f$ steps. We will make the implementation clearer in the revision.\n\n> The theoretical proof of the effectiveness of likelihood function for partially-executed trajectory is missing, and the threshold of likelihood to choose how to replan is selected empirically which may significantly impact the performance.\n\nWe investigate the different thresholds in Figure X and XI and empirically show that our model performs better given the same total diffusion steps.\n\nAlso note that our paper is focused on *empirically* validating the effectiveness of using the likelihood function to determine when to replan. We do not aim to claim that this method is theoretically-optimal or the return of the replaned trajectories has any theoretical guarantees, but do note that replanning based off likelihood is principled as it occurs precisely when states / plans fall outside the distribution learned by the original trajectory model.\n\n**Q4. Stochastic environment.**\n> For stochastic environment, the advantage of methods implementing replanning strategy comparing to non-replan methods should be increased when stochasticity arises, but the performance gets closer shown on Figure 6. Why?\n\nThe reason is that randomness will sometimes cause the agent to reach the out-of-distribution states, which leads to the drop of the performance of all offline RL methods. Our method RDM consistently outperforms other baselines under different levels of stochasticity. However, at high stochasticity levels, all methods will tend to fail."}, {"review_id": "XENHk37S1g", "reviewer": "Reviewer_bzBf", "summary": "The submission describes a method for deciding when and how to \"replan\" when using diffusion models for inferring plans, as in Decision Diffuser [1]. The task is essentially imitation learning (IL): given a training dataset of plans and features, the task is to predict a new plan given novel features. As in [1], the IL method consists of training a diffusion model (DM) to predict state sequences given contexts, and the features happen to include the reward of the plan. A high-reward plan can therefore be inferred by conditioning the model on a high reward value.\n\nThe replanning aspect of this work is useful because running the diffusion model may be expensive, making it difficult to use diffusion-model-based planning to do receding-horizon control. Replanning only when necessary saves computation compared to planning every cycle.", "questions": "How were the intervals / state distance thresholds set for the DD and SDM baselines in figure 7?\n\nHave you tried evaluating the trade-off between solution quality and planning time for different replanning strategies (e.g., for different possible threshold values in DD and SDM)?", "limitations": "The paper adequately addresses the limitations of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Originality: \n\nThe ideas of trying to infer when and how to replan when planning with diffusion models seem novel. I'm not aware of other work on inferring when to replan in general, so that general idea may be novel as well. A nice feature of applying probabilistic methods to planning is that it becomes quite natural to ask such questions.\n\nQuality:\n\nThere are some promising aspects to the experiments. I appreciated that the experiments were targeted towards answering specific questions that were clearly articulated in lines 182-185. The robotic control (RLBench) results were also promising in that the success rate was significantly higher than that of a baseline without replanning.\n\nSignificance:\n\nThe results show that simply replanning based on some simple heuristics is a viable technique for boosting the performance of diffusion-based planning methods. This is a potentially useful and interesting observation for anyone interested in applying diffusion models to solving planning problems in practice.", "weaknesses": "Quality:\n\nI believe the weakest part of the submission is that the experiments are focused mainly on \"apples-to-oranges\" comparisons of RDM to model-free RL and decision diffusion models without replanning. None of the baselines have the ability to respond to new information at test time, as far as I can tell. In the case of the model-free RL algorithms, this is a bit unclear—theoretically, the learned policies could condition on \"real-time\" information, but it's unclear whether this is the case. Still, even if the RL policies do condition on novel information, comparing a policy learned offline to a planner that can be evaluated online, seems a bit unfair.\n\nIt would make more sense to me if the experiments were focused more on comparisons between RDM and other replanning methods. Since the goal of intelligent replanning is to reduce computation with respect to replanning every cycle, I think it would also be fair to evaluate the net effect of different replanning strategies on the tradeoff between solution time and solution quality. For example, if we replan at fixed intervals, but evaluate that for a range of intervals, what trade-off do we see between solution time and quality compared to RDM? If we replan every cycle, is the solution quality much higher than RDM? \n\nAlthough the ablation study in section 4.4 does analyze the effects of different replanning strategies, this analysis feels unsatisfying for a few reasons. First, it is unclear how the threshold / interval for replanning were chosen for DD and SDM, which simply replan based on time or state distance thresholds. Ideally, these thresholds would be chosen to maximize some metric evaluated on a validation set, but I could find no details about this. \n\nI also note that the baseline for figure 7 is not zero, which gives a very misleading impression about the relative performance gap between (e.g.) SDM (replan triggered by state distance) and RDM, which is actually pretty small. There are also no error bars on this plot, so it is hard to tell whether this result is significant. This experiment was also only run on a single task (hopper-medium-expert)—the results would be much more interesting if these simple baselines (DD, SDM) were run on the entire suite of tasks. If it turns out that SDM performs almost as well as RDM on most tasks, then that would make RDM significantly less appealing.\n\nClarity:\n\nThe problem statement is  unclear. On my initial read-through of the paper, I was under the impression that RDM (the proposed method) was an imitation learning (IL) method—until I reached the experiments, which compared RDM to RL methods. I then backtracked to Section 2.1 (problem setting), which states:\n\n\"We consider a trajectory optimization problem similar to Diffuser[17]... the objective of the trajectory optimization problem is to find a sequence of actions … that maximizes J… which is the expected sum of rewards over all timesteps.\"\n\nAfter reading the reference for Diffuser[17], this paragraph convinced me that my original belief was wrong, and that RDM is in fact an (online) RL method that uses a diffusion model as a planner—just like Diffuser[17]. I then at some point encountered a Decision Diffuser[1] reference in the paper and decided to revisit that reference to refresh my memory. I was very confused because the paper was not what I remembered it being, and in fact Decision Diffuser[1] is an IL method.\n\nI eventually realized that Diffuser[17] and Decision Diffuser[1] are different methods: the first is an RL method, and the second is an IL method. However, they both share common authors, use similar methods, and have similar names. I then re-read lines 89-91, (\"we follow the formulation of planning in Decision Diffuser[1]…\") which convinced me that my original impression was correct: RDM is an IL method, not an RL method.\n\nDespite the unfortunately similar names, I believe the real root of my confusion is that the paper lacks a crisp problem statement—the reader should not have to guess as to whether the problem addressed is IL, or offline/online off-policy/on-policy RL. It's unfortunate that Section 2.1 (Problem Setting) introduces the method in RL terms and explicitly states that the problem is similar to that of Diffuser[17], which is an RL method. Instead, that paragraph should probably state something like \"Our work addresses a problem similar to that described in Decision Diffuser[1]: given a dataset of training state trajectories and rewards, our task is to predict a state trajectory similar to what was observed in the training dataset, while conditioning on a query reward value. This is essentially an Imitation Learning problem where we observe and condition on rewards, but it is also similar to RL, in the sense that the goal is to produce a trajectory with high reward.\"\n\nThe comparison to RL methods also confuses things—the submission should be clearer about why RDM is compared to RL methods. It would also be beneficial to call Diffuser by a different name (e.g., Planning Diffuser) to disambiguate it from Decision Diffuser.\n\nThe exposition of the method could also be significantly improved. The explanation of when to replan in section 3.1 is a bit too verbose and doesn't convey the basic idea well, which is simply to evaluate the likelihood of a plan where states/actions at previous timesteps are replaced with those observed during plan execution. A simple figure would be very helpful here.\n\nSignificance:\n\nOne factor that may limit the significance of this work is that the topic is relatively niche—it attempts to solve a particular problem (performance of replanning) with applying a particular planning method (Decision Diffuser), which itself is still relatively immature. Solving niche problems is ok, but the potential for significant impact would be greater if there were  some interesting take-away for a more general audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe thank the reviewer for the detailed comments and questions.\n\n**Q1. Comparison.**\n> It would make more sense to me if the experiments were focused more on comparisons between RDM and other replanning methods.\n> What trade-off do we see between solution time and quality compared to RDM?\n\nWe show the results of the comparison between our RDM algorithm and the replanning-based baseline algorithms across more tasks. We run experiments using different intervals or thresholds and measure their computational time by the total number of diffusion steps. In the table below, we report the total diffusion steps and the performance and show more evaluation results in Figure I-V in the rebuttal PDF. Notably, RDM consistently outperforms all the baseline algorithms under the same total diffusion steps.\n\nTable A. The total diffusion steps and performance.\n\n|Environment|Method|Total Diffusion Steps|Normalized Returns|\n|-----|-----|-----|-----|\n|Maze2D Large|Diffuser|2304.0|165.9 (±3.8)|\n| |SDM|1925.1|175.0 (±5.3)|\n| |PRDM|1916.4|169.3 (±1.4)|\n| |RDM|1894.38|185.4 (±3.0)|\n|Hopper Medium-Expert|DD|17100|47.4 (±3.6)|\n| |SDM|18500|54.4 (±3.5)|\n| |PRDM|16200|57 (±4.3)|\n| |RDM|15900|59.7 (±3.8)|\n|Close Box|DD|1968|46.0 (±7.5)|\n| |SDM|1813|52.4 (±6.6)|\n| |PRDM|1653|50 (±7.2)|\n| |RDM|1600|63.5 (±7.0)|\n\n**Q2. Different replanning strategies.**\n> How the threshold/interval for replanning were chosen for DD and SDM.\n> If we replan every cycle, is the solution quality much higher than RDM?\n\nWe investigate different intervals $I$ for replanning for Diffuser and Decision-Diffuser. The results are shown in Figure VI and VII. We observe that as the interval decreases (that is replanning is done more frequently), the performance improves as we expect. However, when the interval is smaller than a certain value, the performance decreases significantly. For example, in the Maze2D Large domain shown in Figure VI, the return drops when the interval decreases from $100$ to $1$. The ablation results confirm our statement that replanning with an interval that is too small (for example, replanning at every time step) may prevent successful task execution.\n\nWe also analyze the impact of different thresholds $l$ for the baseline algorithms, SDM and RDM. The results are shown in Figure VIII-XI. As we expect, when $l$ decreases (that is, when replanning is done more frequently), the performance has improved. Despite the performance differences of the baseline algorithms, we want to emphasize again that as shown in Figure I-V, under the same computational budget, our RDM algorithm outperforms all the baseline algorithms.\n\n**Q3. Clarity**\n> The real root of my confusion is that the paper lacks a crisp problem statement.\n> The comparison to RL methods also confuses things.\n\nWe apologize that our problem statement might be confusing, we would like to clarify that Diffuser [1], Decision-Diffuser [2], and our work can be seen as following the same setting, which is offline reinforcement learning (offline RL) and generate plans conditioned on maximizing reward (where for Long-Horizon and Robotic Control tasks we use the reward function that corresponds to either reaching the conditioned goal or solving the robotics task respectively). To solve this offline reinforcement learning task, Diffuser, Decision-Diffuser and our work sample a trajectory of states (a plan) that maximizes the conditioned reward, where Diffuser samples from the composition of a diffusion model and value function, while Decision Diffuser and our work sampling from conditional diffusion model conditioned on the desired reward function. This plan is then transformed into a policy.\n\nSince the setting we consider can be considered an offline RL setting, our other baselines are then naturally offline RL algorithms. It's important to note that Decision-Diffuser is not a behavioral cloning method, as in the Mujoco settings, it does not fit a model to all trajectories in a dataset, but rather learns a reward-conditioned trajectory model (similar to Decision Transformer) in order to construct trajectories that maximize reward in an environment. \n\nOur approach towards replanning can be applied to any Diffusion model that synthesizes a trajectory of actions to optimize a reward function and can be applied to either Diffuser or Decision-Diffuser.\n\nWe will revise our problem statement in our revision to make this clearer, please let us know if you have any additional questions.\n\n[1] M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022.\n\n[2] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.\n\n**Q4. Significance.**\n> It attempts to solve a particular problem (performance of replanning) with applying a particular planning method (Decision Diffuser)\n\nThanks for the comment. Our replanning strategy, based off likelihood, can be applied to any diffusion-based planner, which has seen a variety of different applications across different planning settings such as robotics policies [4] and video [5]. Broadly, we believe the idea of likelihood-based replanning can also be applied to many other likelihood-based planning methods (e.g. trajectory transformer[3]). \n\n[3] Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem. arXiv preprint arXiv:2106.02039, 2021.\n\n[4] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. arXiv preprint arXiv:2303.04137, 2023.\n\n[5] Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, Pieter Abbeel. Learning Universal Policies through Text-Conditioned Video Generation. arXiv preprint arXiv:2302.00111, 2023."}, {"review_id": "LpU8ri8OjC", "reviewer": "Reviewer_XopQ", "summary": "The authors address the online replanning problem within diffusion-based models by introducing their method, RDM, which uses the likelihood of trajectories to decides when and how to replan. ", "questions": "- How does this method compare to a fixed, fully replanning strategy at different intervals in the main scenarios compared in the experiments?\n- How are the thresholds in Algorithm 1 tuned, and how does the method perform for different $l_1$/$l_2$ values?", "limitations": "The authors address some limitations of their method in Section 6 of the paper, and there are no concerns regarding potential negative societal impact for this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The topic of when and how to replan is an important one, particularly within the diffusion model community where it has not, to the best of my knowledge, been addressed before.\n- The methodology section appears to be sound, and the idea of choosing to replan and what type of replanning should happen based on the trajectory likelihoods is compelling.\n- The paper is clear, well-written and easy to follow.", "weaknesses": "- The experimental results are lacking in terms of comparison with other simple baselines that are able to replan. While the authors partially address this in the ablation study of Section 4.4, it is unclear why these are also not used in the main experiments. Given the aim of the paper is to provide a good replanning strategy, I do not believe this should be an ablation only, but rather that most results should include “no replanning baselines” (i.e., the ones already provided), and “replanning baselines” as proposed in the ablation. This would allow a reader to take conclusions regarding the actual improvement from performing no replanning to a basic strategy to the proposed method.\n- The absence of replanning in the baselines could justify the significant drop of performance in the results in Section 4.1, 4.2 and 4.3. A comparison with replanning baselines (both fully and, for example, with a randomly selected number of steps) strengthens the case that the “when” and “how” to replan are key to the success of the results.\n- The paper is also lacking an ablation with a variable number of replanning steps. In Section 4.4 the authors compare DD which replans “using two fixed intervals”, but how does the performance change for a higher replanning frequency? \n\n**Minor comments**:\n- Typo in abstract, line 1, should be “have risen as a promising”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe thank the reviewer for the reviews and insightful suggestions.\n\n**Q1. Comparison.**\n> The experimental results are lacking in terms of comparison with other simple baselines that are able to replan.\n\nWe show the results of the comparison between our RDM algorithm and the replanning-based baseline algorithms (SDM and PRDM) across more tasks. We run experiments using different intervals or thresholds and measure their computational costs by the total number of diffusion steps. In the table below, we choose one task from each of the three domains and report the total diffusion steps and the performance, and show more evaluation results in Figure I-V in the rebuttal PDF. Notably, RDM consistently outperforms all the baseline algorithms under the same total diffusion steps.\n\nTable A. The total diffusion steps and performance.\n\n|Environment | Method   | Total Diffusion Steps | Normalized Returns |\n| ----------------------------------------- | -------- | --------------------- | ------------------ |\n| Maze2D Large                              | Diffuser | 2304.0                | 165.9 (±3.8)        |\n|                                           | SDM      | 1925.1                | 175.0 (±5.3)        |\n|                                           | PRDM     | 1916.4                | 169.3 (±1.4)        |\n|                                           | RDM      | 1894.38               | 185.4 (±3.0)        |\n| Hopper Medium-Expert                      | DD       | 17100                 | 47.4 (±3.6)         |\n|                                           | SDM      | 18500                 | 54.4 (±3.5)         |\n|                                           | PRDM     | 16200                 | 57 (±4.3)           |\n|                                           | RDM      | 15900                 | 59.7 (±3.8)         |\n| Close Box                                 | DD       | 1968                  | 46.0 (±7.5)         |\n|                                           | SDM      | 1813                  | 52.4 (±6.6)         |\n|                                           | PRDM     | 1653                  | 50 (±7.2)           |\n|                                           | RDM      | 1600                  | 63.5 (±7.0)         |\n\n\n**Q2. Replan interval.**\n> How does the performance change for a higher replanning frequency?\n\nWe investigate different intervals $I$ for replanning for Diffuser and Decision-Diffuser. The results are shown in Figure VI and VII. We observe that as the interval decreases (that is replanning is done more frequently), the performance improves as we expect. However, when the interval is smaller than a certain value, the performance decreases significantly. For example, in the Maze2D Large domain shown in Figure VI, the return increases when the interval decreases from $250$ to $100$, while it drops when the interval decreases from $100$ to $1$. The ablation results confirm our statement that replanning with an interval that is too small (for example, replanning at every time step) may prevent successful task execution.\n\n**Q3. Replan threshold.**\n> How are the thresholds in Algorithm 1 tuned, and how does the method perform for different $l_1$/$l_2$ values?\n\nWe analyze the impact of different thresholds $l_s$ for RDM. The results are shown in Figure X-XI in the rebuttal PDF. As we expect, when $l$ decreases (that is when replanning is done more frequently), the performance has improved. We choose the best result for different $l_s/l_f$ under the same total diffusion steps.\n\n**Q4. Minor comments.**\n> Typo in abstract, line 1, should be “have risen as a promising”.\n\nThanks for pointing this out. We will address them in the revision.\n"}]

Author rebuttal:
We thank all the reviewers for taking the time to review our paper and providing insightful and detailed feedback. We appreciate that the reviewers recognize the following contributions.

* **Significance of our problem**. Replanning is a crucial problem in many planning problems.
> Online re-planning is a natural and important complement to many planning methods. (Reviewer G16t)
> 
> Replanning is important for robots to execute trajectories robustly. (Reviewer rt81)
> 
> This is a potentially useful and interesting observation. (Reviewer bzBf)
> 
> The topic of when and how to replan is an important one. (Reviewer XopQ)

* **Methodology**. Our approach to determining when and how to replan is novel and compelling.
> As far as I know, the proposed methods are novel. (Reviewer G16t)
> 
> Replanning strategies is novel. (Reviewer kWoD)
> 
> The ideas of trying to infer when and how to replan seem novel. (Reviewer bzBf)
> 
> The methodology section appears to be sound, and compelling. (Reviewer XopQ)

* **Performance**. Our approach brings significant improvement.
> The proposed method significantly improves the performance of diffusion planners. (Reviewer G16t)
> 
> The experimental results demonstrate impressive performance surpassing baselines like IQL. (Reviewer kWoD)
> 
> There are some promising aspects to the experiments. (Reviewer bzBf)

* **Presentation**. Some reviewers acknowledge the clear presentation of the paper.
> The overall writing quality is good, and the core idea is presented well. (Reviewer rt81)
> 
> The paper is clear, well-written and easy to follow. (Reviewer XopQ)

We want to emphasize again that the **main contributions** of our work are as follows.
* We propose a principled and novel approach to determine when and how a diffusion model should replan in planning tasks. 
* We empirically validate the effectiveness of our algorithm in various domains, use comprehensive ablative studies to justify the design of our algorithm, and demonstrate that our algorithm outperforms other state-of-the-art offline RL algorithms in these domains.

In the rebuttal, we address the reviewers' questions and concerns by providing the following new results in our responses and in the rebuttal PDF as well as reviewer-specific comments in the per reviewer response.

**Comprehensive evaluations of the baseline algorithms on more domains.**

* **Replanning-based Baselines**. We show the results of the comparison between our RDM algorithm and the replanning-based baseline algorithms (SDM and PRDM) across more tasks. We run experiments using different intervals or thresholds and measure their computational costs by the total number of diffusion steps. In the table below, we choose one task from each of the three domains and report the total diffusion steps and the performance, and show more evaluation results in Figure I-V in the rebuttal PDF. Notably, RDM consistently outperforms all the baseline algorithms under the same total diffusion steps (that is, with the same computational budget).

Table A. The total diffusion steps and performance.
|Environment | Method   | Total Diffusion Steps | Normalized Returns |
| ----------------------------------------- | -------- | --------------------- | ------------------ |
| Maze2D Large                              | Diffuser | 2304.0                | 165.9 (±3.8)        |
|                                           | SDM      | 1925.1                | 175.0 (±5.3)        |
|                                           | PRDM     | 1916.4                | 169.3 (±1.4)        |
|                                           | RDM      | 1894.38               | 185.4 (±3.0)        |
| Hopper Medium-Expert                      | DD       | 17100                 | 47.4 (±3.6)         |
|                                           | SDM      | 18500                 | 54.4 (±3.5)         |
|                                           | PRDM     | 16200           | 57 (±4.3)           |
|                                           | RDM      | 15900                 | 59.7 (±3.8)         |
| Close Box                                 | DD       | 1968                  | 46.0 (±7.5)         |
|                                           | SDM      | 1813                  | 52.4 (±6.6)         |
|                                           | PRDM     | 1653                  | 50 (±7.2)           |
|                                           | RDM      | 1600                  | 63.5 (±7.0)         |


**Comprehensive ablative studies for diffusion-based algorithms.**

* **Different Intervals for Replanning**. We investigate different intervals $I$ for replanning for Diffuser and Decision-Diffuser. The results are shown in Figure VI and VII. We observe that as the interval decreases (that is replanning is done more frequently), the performance improves as we expect. However, when the interval is smaller than a certain value, the performance decreases significantly. For example, in the Maze2D Large domain shown in Figure VI, the return increases when the interval decreases from $250$ to $100$, while it drops when the interval decreases from $100$ to $1$. The ablation results confirm our statement that replanning with an interval that is too small (for example, replanning at every time step) may prevent successful task execution.
* **Different Thresholds for Replanning**. We also analyze the impact of different thresholds $l$ for the baseline algorithms, SDM and RDM. The results are shown in Figure VIII-XI. As we expect, when $l$ decreases (that is, when replanning is done more frequently), the performance of all the methods has improved. Despite the performance differences of the baseline algorithms, we want to emphasize again that as shown in Figure I-V, under the same computational budget, our RDM algorithm outperforms all the baseline algorithms.

Please see our detailed responses to all the reviewers below.

Meta-review: The paper considers the problem of replanning in settings where diffusion models are used for planning, a problem that is becoming increasinginly important and yet has received very little attention of-late. The paper proposes a method (RDM) that reasons over when to initiate replanning (based on the diffusion models estimated likelihood of generated plans) and how to perform replanning when the underlying planner is diffusion-based.

The paper was evaluated by five reviewers who largely agree that the problem of online replanning in the case of diffusion model-based planning is particularly important. They also find that the proposed framework, which involves separate methods for determining when and how to replan, is novel and shown to yield significant performance gains. However, the reviewers raised several questions/concerns regarding the computational cost of the method and deficiencies in the evaluation, including the need to compare to more apples-to-apples baselines. The authors made a concerted effort to resolve the reviewers' questions and to address their concerns. As the reviewers point ot in their follow-up comments, the additional discussion regarding experiments on the part of the authors have helped to resolve several of the reviewers' key concerns. The authors are strongly encouraged to ensure that the paper is updated based upon the discussion with the reviewers, which includes updating the paper to include the new experimental results.
Decision: Accept (poster)