Title: Adaptive Online Replanning with Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Decision making

Robotics

Planning-based
Abstract: Diffusion models have risen a promising approach to data-driven planning, and have demonstrated impressive robotic control, reinforcement learning, and video planning performance. Given an effective planner, an important question to consider is replanning -- when given plans should be regenerated due to both action execution error and external environment changes.  Direct plan execution, without replanning, is problematic as errors from individual actions rapidly accumulate and environments are partially observable and stochastic. Simultaneously, replanning at each timestep incurs a substantial computational cost, and may prevent successful task execution, as different generated plans prevent consistent progress to any particular goal. In this paper, we explore how we may effectively replan with diffusion models. We propose a principled approach to determine when to replan, based on the diffusion model's estimated likelihood of existing generated plans. We further present an approach to replan existing trajectories to ensure that new plans follow the same goal state as the original trajectory, which may efficiently bootstrap off previously generated plans.  We illustrate how a combination of our proposed additions significantly improves the performance of diffusion planners leading to 38\% gains over past diffusion planning approaches on Maze2D and further enables handling of stochastic and long-horizon robotic control tasks.

Adaptive Online Replanning with Diffusion Models

Abstract

Diffusion models have risen as a promising approach to data-driven planning, and have demonstrated impressive robotic control, reinforcement learning, and video planning performance. Given an effective planner, an important question to consider is replanning - when given plans should be regenerated due to both action execution error and external environment changes. Direct plan execution, without replanning, is problematic as errors from individual actions rapidly accumulate and environments are partially observable and stochastic. Simultaneously, replanning at each timestep incurs a substantial computational cost, and may prevent successful task execution, as different generated plans prevent consistent progress to any particular goal. In this paper, we explore how we may effectively replan with diffusion models. We propose a principled approach to determine when to replan, based on the diffusion model's estimated likelihood of existing generated plans. We further present an approach to replan existing trajectories to ensure that new plans follow the same goal state as the original trajectory, which may efficiently bootstrap off previously generated plans. We illustrate how a combination of our proposed additions significantly improves the performance of diffusion planners leading to 38% gains over past diffusion planning approaches on Maze2D, and further enables the handling of stochastic and long-horizon robotic control tasks. Videos can be found on the anonymous website: https://vis-www.cs.umass. edu/replandiffuser/ .

1 Introduction

Planning is a pivotal component in effective decision-making, facilitating tasks such as collision avoidance in autonomous vehicles [29], efficient and long-horizon object manipulation [12] and strategic reasoning for extended periods [32, 25]. Recently, diffusion has risen as a promising approach to data-driven planning [17, 1, 14, 40], enabling impressive advancements in robot control [6] and the generation of video-level plans [9].

In many planning settings, an important variable to consider is not only how to plan , but also when it is time to replan . Environments are often stochastic in nature, making previously constructed plans un-executable given an unexpected environmental change. Moreover, inaccurate execution of individual actions in an environment will often also cause states to deviate from those in the expected plan, further necessitating replanning. Finally, environments will often be partially observable, and the optimal action or sequence of actions may deviate substantially from those originally planned, especially when new information comes to light.

Previous works in planning with diffusion models have not explored how to tackle these factors, and typically generate new plans at each iteration. Such a process is computationally expensive as a

(a) Initial Plan

(b) Fast Replanning

(c) Complete Replanning

Given a previously generated plan, how do we effectively regenerate a new plan? Directly resampling the entire plan can be computationally expensive, as all the previous computations put in the previous plan would be wasted. Furthermore, a newly generated plan might erase all the progress we made previously toward the goal from a previous plan. Instead, based on the feasibility of the current plan being executed, which is estimated by likelihood determined by the diffusion model, we propose to partially replan based on our plan, or fully replan from scratch. We initialize our new plan with the old one and add a small amount of noise to the plan before then denoising the added noise. This light amount of denoising serves as a planning refinement operator.

We illustrate our online planning procedure in Figure 1, where a learned agent first generates a full plan for the goal as in Figure 1a. When the agent reaches the first door, the likelihood of the plan drops, as the agent finds it locked, and the corresponding solution becomes infeasible due to the door's inaccessibility. However, the agent can slightly adjust the initial plan by instead entering the left yellow door and taking a short detour, after which the rest of the plan remains the same. Based on the estimated likelihood of the plan, we may replan and regenerate a new solution based on the previous solution as shown in Figure 1b. In another situation where the yellow door is also locked, and there is no nearby solution, our likelihood plummets, telling us that we need to regenerate a new and completely different solution from scratch as shown in Figure 1c.

In summary, our contributions are three-fold. (1) We introduce RDM, a principled approach to determining when to replan with diffusion models. (2) We illustrate how to effectively replan, using a previously generated plan as a reference. (3) We empirically validate that RDM obtains strong empirical performance across a set of different benchmarks, providing a 38% boost over past diffusion planning approaches on Maze2D and further enabling effective handling of stochastic and long-horizon robotic control tasks.

2 Background

2.1 Problem Setting

We consider a trajectory optimization problem similar to Diffuser [17]. A trajectory is represented as τ = ( s 0 , a 0 , s 1 , a 1 , . . . , s T , a T ) , where s t , a t are the state and action at the t -th time step, and T is the episode length. The dynamics function of the environment is formulated as s t +1 = f σ ( s t , a t ) , where σ represents the stochastic dynamics. For domains with partial observations, the agent observes an observation o t instead of the state s t . The objective of the trajectory optimization problem is to find a sequence of actions a 0: T that maximizes J ( s 0 , a 0: T ) , which is the expected sum of rewards

0

50

100

150

Performace

Long-horizon

0

20

40

60

Stochastic

Single-step

Diffuser

RDM

0

25

50

75

Robotics Control

over all the time steps,

Following the Decision Diffuser [1], we plan across state-space trajectories and infer actions through inverse dynamics.

2.2 Planning with Diffusion Models

In Diffuser [17], plan generation is formulated as trajectory generation τ through a learned iterative denoising diffusion process p θ ( τ i -1 | τ i ) [33]. This learned denoising process is trained to reverse a forward diffusion process q ( τ i | τ i -1 ) that slowly corrupts the structure of trajectories by adding noise. Both processes are modeled as Markov transition probabilities:

Given a set of predetermined increasing noise coefficients β 1 , ..., β N , the data distribution induced is given by:

where q ( τ 0 ) is the data distribution and p ( τ N ) is a standard Gaussian prior.

Diffuser is trained to maximize the likelihood of trajectories through a variational lower bound over the individual steps of denoising

where trajectory plans may then be generated by sampling iteratively from the learned denoising network for N steps. We follow the formulation of planning in Decision Diffuser [1], where if we seek to optimize an objective J ( τ ) , we directly learn and sampling from conditional trajectory density p θ ( τ |J ) .

In terms of notations, we follow Janner et al. [17] and use superscripts i ∈ [0 , N ] to denote diffusion steps (for example, τ i ), and use subscripts t ∈ [0 , T ] to denote time steps of the agent's execution in the environment (for example, s t , a t , etc.).

3 Adaptive Online Replanning with Diffusion Models

In this section, we present RDM, an approach to online replanning with diffusion models. First, we discuss different criteria for determining when a solution is problematic and needs replanning. We then introduce the estimated likelihood of a planned trajectory as a criterion to determine when to replan in Section 3.1. In Section 3.2, we discuss how we replan by comparing three distinct replanning strategies and propose an adaptive replanning approach, based on combinations of the two approaches.

𝐒𝐭+𝐡

𝐒𝐭+𝟐

𝐒𝐭+𝟏

𝐒𝐭

Replan from scratch

(All diffusion steps)

𝐒𝐭+𝐡

𝐒𝐭+𝟐

𝐒𝐭

● ● ●

● ● ●

Replan with future

(Small diffusion steps)

𝐒𝐤+𝐡

𝐒𝐭+𝐡+𝟏

𝐒𝐭+𝐡

𝐒𝐤

● ● ●

● ● ●

Likelihood

Environment Step

𝐒𝐭+𝐡

𝐒𝐤+𝟏

𝐒𝐤

𝐒𝐭

● ● ●

● ● ●

𝐒𝐭+𝟏

𝐒𝐭+𝐡

● ● ●

𝐒𝐭+𝐡

𝐒𝐤+𝟏

𝐒𝐤-𝟏

1

1

2

2

t              k

1

0.75

0.5

0.25

0

3.1 When to Replan

When the agent sequentially executes a plan τ = ( s 0 , a 0 , s 1 , a 1 , . . . , s T , a T ) , different factors in the environment such as unexpected interventions or imperfect transitions may cause future states and actions in the plan to deviate substantially from the desired trajectory. We denote with τ → k the trajectory executed in the environment up to time step k . Formally, τ → k = ( s 0 , a 0 , ¯ s 1 , ¯ a 1 , . . . , ¯ s k , ¯ a k , s k +1 , . . . , s T , a T ) , where s t , a t are planned states and actions generated by the diffusion model; ¯ s t is a state observed in the environment, and ¯ a t is an action that is computed using the inverse dynamics during the execution process (which is the action transiting to s t +1 from ¯ s t ). Continuing to execute the current plan will result in more significant discrepancies and potentially catastrophic failures. It may not even be feasible to transit to the next planned state ( s k +1 ) from the current state that the agent is at ( ¯ s k ). In such cases, we need to update the planned trajectory based on τ → k , a process we refer to as replanning .

Unfortunately, determining the need for replanning is non-trivial. Directly replanning at a fixed interval is straightforward but inflexible, and may come too late when the environment suddenly changes. It can also make the planning algorithm computationally inefficient when the planned trajectory is executed perfectly and does not require replanning. Instead, it is essential to establish a criterion that automatically determines when replanning is necessary. An alternative approach is to replan when the distance between the actual state observed from the environment and the desired state in the plan exceeds a certain threshold. However, in many cases, even if the actual state the agent is in matches with the state in the plan, new information about the environment may make the current plan infeasible. Thus, it is desirable to come up with a comprehensive and adaptive approach to determine the appropriate timing for replanning.

In this paper, we propose a novel method that utilizes the inherent estimated likelihood of the trajectory in diffusion models as a criterion for replanning. As the diffusion model is trained on different successful trajectories in an environment, its learned likelihood function implicitly captures the executability of an underlying plan. A feasible trajectory will be assigned a high probability, with increasingly less feasible trajectories assigned decreasing probabilities. Thus, when the likelihood assessment of a diffusion model on the plan has a low probability, it is a good indicator of the need for replanning.

The key challenge is to estimate the likelihood of a partially-executed trajectory τ → k . Recall that we optimize the variational lower bound defined in Equation 3. We propose to use the same objective to determine the likelihood of a partially-executed trajectory. Intuitively, we assess the degree to which a diffusion model recovers its original plan after the plan has been corrupted by Gaussian noise. Concretely, given τ → k , we added different magnitudes of Gaussian noise to the trajectory, corresponding to sampling based on the diffusion process q ( τ i | τ → k ) at a diffusion step i . We then compute the KL divergence between the posterior q ( τ i -1 | τ i , τ → k ) and the denoising step p θ ( τ i -1 | τ i ) at the diffusion step i . We estimate likelihood by computing this KL divergence at several fixed different diffusion steps i , which we discuss in detail in the appendix.

Algorithm 1 When to Replan.

1: Input: Diffusion model p θ , Planning horizon H , Episode length T , Threshold l s of Replanning from scratch , Threshold l f of Replanning with future and l s < l f , Diffusion step N s of Replan from scratch , Diffusion step N f of Replan with future and N f ≪ N s , Diffusion steps I for computing likelihood 2: Environment step t = 0 3: Generate a trajectory τ t = ( s t , a t , . . . , s t + H , a t + H ) 4: for t in 0 , . . . , T do 5: Extract ( s t , s t +1 , a t ) from τ t 6: Execute a t and obtain s t +1 from the environment 7: Replace the s t +1 in τ t 8: τ 0 ← τ t 9: Sample τ i ∼ q ( τ i | τ 0 ) for all i ∈ I 10: Compute average likelihood L t = 1 | I | ∑ i ∈ I D KL ( q ( τ i -1 | τ i , τ 0 ) ∥ p θ ( τ i -1 | τ i )) 11: if ( t +1)% H = 0 or L t < = l 1 then 12: Sample a trajectory τ t with Replanning from scratch 13: else if L t < = l 2 then 14: Sample a new trajectory τ t based on previous trajectory τ t -1 with Replanning with future 15: else 16: τ t +1 ← τ t 17: end if 18: end for

Algorithm 2 Replanning from scratch

1: Input: Diffusion model p θ , Diffusion steps N s , Conditions C 2: Initialize plan τ N s ∼ N ( 0 , I ) 3: for i in N s , . . . , 1 do 4: τ i -1 ← p θ ( τ i ) 5: Constraint τ i -1 with conditions C 6: end for 7: Return: τ 0

Algorithm 3 Replanning with future

1: Input: Diffusion model p θ , Diffusion process q , Diffusion steps N f , Conditions C , Previous sampled trajectory ˜ τ

2: τ N f ← q N f (˜ τ )

3: for i in N f , . . . , 1 do

4: τ i -1 ← p θ ( τ i )

5: Constraint τ i -1 with conditions C

6: end for

7: Return: τ 0

Given this likelihood estimate, we set a threshold for the likelihood of the planning trajectories. If the likelihood drops below the threshold, it indicates the current plan deviates significantly from the optimal plan and can cause the failure of the plan, which suggests that replanning is necessary. We illustrate this replanning process in Algorithm 1.

3.2 How to Replan

Given a previous partially executed plan executed to time step k , τ → k , and suppose we decide to replan at this current time step using one of the criteria defined the Section 3.1, how do we construct a new plan τ ′ ? We consider three separate replanning strategies, which are the following:

Replan from scratch. In this strategy, we regenerate a completely new planning trajectory based only on the current state. This involves completely resampling the trajectory starting from Gaussian noise and running all steps (defined as N s ) of iterative denoising. Although such an approach fully takes advantage of the diffusion model to generate the most likely trajectory, it is computationally expensive and discards all information in the previously generated plan (Algorithm 2).

Replan from previous context. Instead, we may replan by considering both the current and historical states when executing the previously sampled plan. Given a partially-executed trajectory τ → k and current time step k , we sample a new trajectory τ ′ based on τ → k . Specifically, we run N p diffusion steps to add noise to τ → k at all the timesteps that are larger than k by sampling from the forward process q ( τ N p | τ → k ) . We then run N p steps of denoising. Here, N p is much smaller than Replan from scratch. We will fix the timesteps of τ → k that are smaller than k to their observed

value, as we have already executed them in the environment. This approach can be seen as locally repairing a predicted plan, and thus requires a substantially smaller number of diffusion steps.

Replan with future context. In a similar way to replan with previous context, in replan with future context, we initialize our new plan τ ′ based on our partially-executed plan τ → k . However, in contrast to the previous approach, we do not add the previously executed states of the plan. We remove the executed states in τ → k and repeatedly add the last predicted state in τ → k on the right to fill in the remaining plan context. Namely, ˜ τ = (¯ s k , ¯ a k , s k +1 , a k +1 , . . . , s T , a T , s T , a T , . . . , s T , a T ) , where s T , a T are repeated to match the input length of the diffusion model. To replan, similar to replan from previous context, we add N f steps of noise to ˜ τ by sampling from the forward process q ( τ N f | τ → k ) and then denoise for N f steps (Algorithm 3), where N f is much smaller than N s .

Replanning from scratch is highly time-consuming and inefficient for real-time tasks since it requires all diffusion steps to be run to generate a plan from Gaussian noise. However, in cases of significant interventions, it becomes necessary to regenerate the plan from scratch. Both replanning for future context or past context are efficient and locally repair the previously generated plan. Replanning with past context remains more consistent with the original goal of the previous plan, while replanning for future context further extends the current plan to future context.

We empirically find that replanning with future context is a more effective approach. In our proposed approach, based on the estimated likelihood of a current plan, we either replan from scratch or replan with future context. We illustrate the psuedocode for our approach in Algorithm 1. We provide a detailed ablation analysis of the different replanning strategies in the experiment section (Figure 7).

4 Experiments

In this section, we empirically use our proposed RDM algorithm for replanning in multiple decisionmaking tasks (as illustrated in Figure 2) and evaluate the performance of the generated trajectories. We empirically answer the following questions: Q1: In domains with long horizon plans, does RDM effectively and efficiently fix the initial plan? Q2: In domains with unexpected interventions, which makes the initial plan no longer feasible during the execution, does RDM efficiently find a feasible plan? Q3: How do when to replan and different replanning strategies affect the agent's performance? In all the experiments below, we report the results using five random seeds.

4.1 Long Horizon Planning

We first evaluate the long-horizon planning capacity of RDM and the baseline methods in the Maze2D environment. In this environment, the large Maze2D domain is especially challenging, as it may require hundreds of steps to reach the target and we found that baseline algorithms struggled to find perfect trajectories given the long horizon and sparse reward setting.

Maze2D is a goal-conditioned planning task, where given the starting location and the target location, the agent is expected to find a feasible trajectory that reaches the target from the starting location avoiding all the obstacles. The agent receives a reward of 1 when it reaches the target state and receives a reward of 0 otherwise. The starting and target locations are randomly generated, consistent with the setting in the Diffuser [17]. We evaluate replanning using our RDM algorithm and multiple competing baseline algorithms, including model-free reinforcement learning algorithms such as BatchConstrained deep Q-learning (BCQ) [11], Constrained Q-learning (CQL) [22], Implicit Q-learning (IQL) [21], and planning-based algorithms like Diffuser [17].

We report the results in Table 1 where numbers for all baselines are taken from Diffuser [17]. As expected, model-free reinforcement learning algorithms (BCQ, CQL, and IQL) struggle to solve long-horizon planning tasks. On the other hand, Diffuser generates a full plan and outperforms these approaches, but does not replan online. As a result, it still does not perform well when the size of the domain increases. As the planning horizon increases, it is likely that some of the transitions generated by the diffusion model are infeasible as shown in Figure 4c. We find that our replanning approach, RDM, significantly outperforms all the baseline algorithms. Specifically, it achieves a score of 190% in the Maze2D Large setting, which is almost 63% higher than the Diffuser. We also visualize the sampled trajectory and the executed trajectories in Figure 4. We can observe that infeasible transitions and collisions with the wall will drop the likelihood substantially. RDM is able to find these infeasible transitions and replan accordingly.

180

200

220

240

260

280

300

Environmental Step

0.0

0.2

0.4

0.6

0.8

1.0

Likelihood

Likelihood of the sample

Diffuser

RDM

(a) Likelihood

(b) Sample

(c) Diffuser

(d) RDM

4.2 Stochastic Environments

Next, we investigate whether RDM can tackle environments with stochastic dynamics. We consider a stochastic variation of the D4RL locomotion benchmark [10] by adding randomness to the transition function. Stochasticity can make the trajectory optimization problem more challenging, as the executed trajectory can be significantly different from the initially planned trajectory. We omitted the Halfcheetah environment as we found all algorithms performed poorly with added stochasticity. In this setting, we consider competing baseline methods including a state-of-the-art model-free algorithm, IQL [21], a Transformer-based RL algorithm, Decision Transformer (DT) [5], and a diffusion model-based algorithm, Decision Diffuser (DD) [1].

As shown in Table 2, although the performance of all the methods drops significantly compared with one without stochasticity, RDM performs comparatively well among all baselines. RDM with replanning strategy is able to detect such stochasticity and quickly regenerate a new plan, resulting in improved performance. We compare DD and RDM using the same number of total diffusion resampling steps.

4.3 Robotic Control

We further evaluate the RDM's capacity in complex domains with visual observations. We consider the RLBench domain [15], which consists of 74 challenging vision-based robotic learning tasks. The agent's observations are RGB images captured by multi-view cameras. The agent controls a 7 DoF robotic arm. The domain is designed to reflect a real-world-like environment with high dimensionality in both the observation space and the action space, making it challenging to solve (Figure 5b).

Most previous works tackle the RLBench tasks by taking actions in macro-steps [13], where the agent switches to a different macro-step when the state of the gripper or the velocity changes. However, such an approach has several limitations: (1) The longest trajectory length achievable using macro steps is relatively short, which is slightly over 30 steps. Such a limitation hinders the agent's ability

Fail to open box Replan and retry

心

Fail to open box Continue the plan

9

to solve long-horizon tasks. (2) Macro-steps are pre-defined, which usually require expert knowledge or demonstrations. (3) Macro-steps constrain the agent's action space, which makes it impossible to optimize at a finer granularity (for example, avoiding an intermediate collision) and prevent the agent from doing fine-grained control. Therefore, we do not use macro-step in our experiments.

We choose the following three tasks for evaluation: open box , close box , and close fridge . The experimental results are in Table 3. We find that baseline algorithms fail to accomplish these three tasks. In contrast, RDM outperforms all the baselines by a large margin and achieves a success rate of 77.3%. We visualize the execution of RDM in Figure 5. RDM successfully detects failures in the initial plan and efficiently replans during the execution.

4.4 Ablative Study

Different Levels of Stochasticity . We investigate how the algorithms respond to different levels of stochasticity by setting different noise values ϵ in the task Hopper-medium-expert , where at each time step, the transition is the same as the deterministic version of the domain with probability 1 -ϵ ,

0.01

0.02

0.03

0.04

Probability

50

60

70

80

Normalized Returns

IQL

DT

DD

RDM

DD

SDM

RPDM

RDM

50

55

60

Normalized Returns

and is random (as if a random action is taken) with probability ϵ . The results are in Figure 6. We find that RDM consistently outperforms other baselines under all ϵ values.

Different Replanning Strategies . To analyze the different replan approaches, we consider the following baselines. (1) Decision Diffuser (DD) replans using two fixed intervals, for deciding when to replan from scratch and when to plan with future context, respectively. (2) Replan on State Distance (SDM) replans when the error between the observed states ( ¯ s t ) and desired states in the plan ( s t ) exceeds a threshold. It replans with the future context. The following two algorithms determine when to plan according to the diffusion model likelihood. (3) Replan from previous context (RPDM) replans with on past context, and (4) RDM replans with future context. In Figure 7, we find that SDM and RPDM both have a better performance than DD, among which RDM achieves the best performance. All methods use the same number of total diffusion steps.

We provide additional ablation studies and visualization analysis in the Appendix.

5 Related Work

Diffusion Models for Decision Making . The use of diffusion models for decision-making tasks has seen increased interest in recent years [17, 1, 9, 6, 30, 40, 23, 14, 38, 24] and have been applied to tasks such as planning [17, 1], robotic control [6], behavior cloning [30], and video generation [9]. In [17, 1] introduce an approach to planning with diffusion models, but both methods typically replan at each iteration. In this paper, we propose a method to adaptively determine when to replan with diffusion models and illustrate different approaches to replan.

Model-Based Planning . Our work is also related to existing work in model-based planning [35, 4, 7, 19, 3, 36, 2, 27, 34, 8, 37, 28]. For domains with finite states, traditional methods use dynamic programming to find an optimal plan, or use Dyna-algorithms to learn the model first and then generate a plan [35]. For more complex domains, a learned dynamics model be autoregressively rolled out [7, 19], though gradient planning through these models often lead to adversarial examples [3, 36, 2]. To avoid this issue approaches have tried gradient-free planning methods such as random shooting [27], beam search [34], and MCMC sampling [4]. Our work builds on a recent trend in model-based planning which folds the planning process into the generative model [16].

6 Limitations and Conclusion

Conclusion . In this paper, we propose RDM, an adaptive online replanning method, with a principled approach to determine when to replan and an adaptive combined strategy for how to replan. We demonstrate RDM's remarkable proficiency in managing long-horizon planning, unpredictable dynamics, and intricate robotic control tasks. An interesting line of future work involves investigating the scalability of RDM to more complex and larger-scale problems and extending to real robot tasks and autonomous driving.

Limitations . There are several limitations to our approach. Firstly, the effectiveness of planning and replanning is heavily dependent on the generative capabilities of the diffusion models. If the diffusion

model is unable to accurately represent the current task, its plans may not be viable and RDM may fail to identify problematic trajectories. Secondly, despite RDM's significant improvement on the performance of previous diffusion planning methods, we observed that likelihood-based replanning was insensitive to some planning failures. This highlights the potential benefit of combining likelihoodbased replanning with other indicators for replanning.

Acknowledgment

This work has been made possible by a Research Impact Fund project (R6003-21) funded by the Research Grants Council of the Hong Kong Government.

References

A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 , 2022.

K. Asadi, D. Misra, and M. Littman. Lipschitz continuity in model-based reinforcement learning. In International Conference on Machine Learning , 2018.

S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems , 2015.

H. Chen, Y. Du, Y. Chen, J. Tenenbaum, and P. A. Vela. Planning with sequence models through iterative energy minimization. arXiv preprint arXiv:2303.16189 , 2023.

L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 , 2023.

S. Chiappa, S. Racaniere, D. Wierstra, and S. Mohamed. Recurrent environment simulators. arXiv preprint arXiv:1704.02254 , 2017.

Y. Du, T. Lin, and I. Mordatch. Model based planning with energy based models. In Conference on Robot Learning , 2019.

Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. arXiv e-prints , pages arXiv-2302, 2023.

J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems , 4:265-293, 2021.

P.-L. Guhur, S. Chen, R. G. Pinel, M. Tapaswi, I. Laptev, and C. Schmid. Instruction-driven history-aware policies for robotic manipulations. In Conference on Robot Learning , pages 175-187. PMLR, 2023.

S. Huang, Z. Wang, P. Li, B. Jia, T. Liu, Y. Zhu, W. Liang, and S.-C. Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. arXiv preprint arXiv:2301.06015 , 2023.

S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters , 5(2):3019-3026, 2020.

M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , 2022.

M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991 , 2022.

L. P. Kaelbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation , pages 1470-1477. IEEE, 2011.

N. R. Ke, A. Singh, A. Touati, A. Goyal, Y. Bengio, D. Parikh, and D. Batra. Modeling the long term future in model-based reinforcement learning. In International Conference on Learning Representations , 2018.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

Z. Liang, Y. Mu, M. Ding, F. Ni, M. Tomizuka, and P. Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877 , 2023.

W. Liu, Y. Du, T. Hermans, S. Chernova, and C. Paxton. Structdiffusion: Language-guided creation of physically-valid structures using unseen objects. arXiv preprint arXiv:2211.04604 , 2022.

S. Maharaj, N. Polson, and A. Turk. Chess ai: competing paradigms for machine intelligence. Entropy , 24(4):550, 2022.

D. Misra. Mish: A self regularized non-monotonic neural activation function. In British Machine Vision Conference , 2019.

A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for modelbased deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 7559-7566. IEEE, 2018.

J. Oh, S. Singh, and H. Lee. Value prediction network. Advances in neural information processing systems , 30, 2017.

B. Paden, M. ˇ Cáp, S. Z. Yong, D. Yershov, and E. Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on intelligent vehicles , 1 (1):33-55, 2016.

T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.10677 , 2023.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

J. Sun, D.-A. Huang, B. Lu, Y.-H. Liu, B. Zhou, and A. Garg. Plate: Visually-grounded planning with transformers in procedural tasks. IEEE Robotics and Automation Letters , 7(2):4924-4930, 2022.

R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction . MIT press, 2018.

E. Talvitie. Model regularization for stable sample rollouts. In UAI , pages 780-789, 2014.

A. Tamar, Y. Wu, G. Thomas, S. Levine, and P. Abbeel. Value iteration networks. Advances in neural information processing systems , 29, 2016.

Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193 , 2022.

Y. Wu and K. He. Group normalization. In European Conference on Computer Vision , 2018.

E. Zhang, Y. Lu, W. Wang, and A. Zhang. Lad: Language augmented diffusion for reinforcement learning. arXiv preprint arXiv:2210.15629 , 2022.

(a) Problematic Sample

(b)

N

= 40

(c)

N

= 80

(d)

N

= 256

In the supplementary, we first discuss the experimental details and hyperparameters in Section A. Next, we analyze the impact of different numbers of diffusion steps N on the replanning process in Section B, and further present the visualization in RLBench in Section C. Finally, we discuss how to compute the likelihood in Section D.

A Experimental Details

Our architecture is built based on Diffuser [17] and Decision Diffuser [1]. In detail, our architecture comprises a temporal U-Net structure with six repeated residual networks. Each network consists of two temporal convolutions followed by GroupNorm [39], and a final Mish nonlinearity [26]. Additionally, We incorporate timestep and conditions embeddings, which are both 128-dimensional vectors produced by MLP, within each block.

In RLBench, we exploit the image encoder from the pre-trained CLIP [31], followed by a 2-layered MLP with 512 hidden units and Mish activations.

The model is trained using Adam optimizer [20] with a learning rate of 2e -04 and a batch size of 64 for 1e6 training steps.

The planning horizon is set to 128 in Maze2D//Multi2D U-Maze, 256 in Maze2D//Multi2D Medium, 256 in Maze2D//Multi2D Large, 64 in Stochastic Environments, and 64 in RLBench.

We use a threshold of 0 . 7 for Replan from scratch and a threshold of 0 . 5 for Replan with future .

The probability ϵ of random actions is set to 0 . 03 in Stochastic Environments.

The diffusion steps i for computing likelihood is set to { 5 , 10 , 15 } in Maze2D and Stochastic Environments and { 10 , 20 , 30 } in RLBench.

The total number of diffusion steps, corresponding to the number of diffusion steps for Replan from scratch is set to 256 in Maze2D, 200 in Stochastic Environments, and 400 in RLBench. And the number of diffusion steps for Replan with future is set to 80 in Maze2D, 40 in Stochastic Environments, and 100 in RLBench.

We perform the whole experiment with a total of three Tesla V100 GPUs.

B Different Diffusion Steps for Replanning

In this section, we visualize the impact of using different numbers of diffusion steps N to replan an existing trajectory. Figure 8 illustrates a problematic sampled trajectory after execution. When

(a) Problematic Sample

(b)

N

= 40

(c)

N

= 80

(d)

N

= 256

Fail to close box Replan and retry

Fail to close box Continue the plan

N = 40 of replanning, the resampled trajectory still leads to the collision with the wall. However, with N = 80 , which aligns with the number of diffusion steps used in Replan with future , we observe that the sampled trajectory becomes feasible. For N = 256 , corresponding to the number of diffusion steps used in Replan from scratch , the resampled trajectory shows additional improvement, but such a procedure is substantially more expensive than our replanning procedure.

We illustrate another example plan which based off likelihood would need to be planned from scratch in Figure 9. After replanning with N = 40 and N = 80 , the trajectories still encounter issues with colliding with the wall. However, With N = 256 , RDMsuccessfully regenerates a completely different and feasible trajectory through replanning from scratch.

We further evaluate the performance with different replanning steps in Table 4. The results clearly demonstrate that that RDM consistently outperforms Diffuser in all cases. Remarkably, RDM with N = 80 , corresponding to Replan with future , achieves performance close to RDM with N = 256 , which corresponds to Replan from scratch which is substantially more expensive.

C Comparison between trajectories with and without replanning

In this section, we visualize the execution in the task close box in Figure 10. In the second frame of Figure 10, we observe that the robotic arm fails to close the box due to insufficient contact between the gripper and the lid. In Figure 10a, RDM successfully detects the failures and retries to establish

0

50

100

150

200

250

300

Environmental Step

1

2

3

4

5

6

7

KL Divergence

1e

4

(a)

i

= 60

0

50

100

150

200

250

300

Environmental Step

0.5

1.0

1.5

2.0

2.5

KL Divergence

1e

3

(b)

i

= 30

0

50

100

150

200

250

300

Environmental Step

0.2

0.4

0.6

0.8

1.0

1.2

KL Divergence

1e

2

(c)

i

= 15

proper contact with the lid. As a result, the gripper successfully closes the box this time. In contrast, Figure 10b demonstrates the scenario where the arm fails to notice the failures, making the arm persist in the unsuccessful plan. These results demonstrate the capability of RDM to detect failures and replan a successful trajectory in challenging robotic control tasks.

D How to Compute Likelihood

In this section, we analyze the number of noising steps we should use to estimate the likelihood. The likelihood estimation is performed by computing the KL divergence between q ( τ i -1 | τ i , τ → k ) and p θ ( τ i -1 | τ i ) with different magnitudes of noise added at diffusion timesteps i . We analyze the impact of different diffusion steps i and the corresponding magnitude of noise added for computing KL divergence in Figure 11. We observe that (1) When large amounts of noise are added at i = 60 the agent fails to detect any infeasibility, as the KL divergence remains low and consistent. (2) When intermediate amounts of noise are added at i = 30 , the agent has the ability to notice infeasibilities, but the KL divergence value is similar to that of a feasible trajectory. (3) When small amounts of noise are added at i = 15 , the agent can effectively discern infeasibilities, as the KL divergence exhibits a clear boundary between feasible and infeasible trajectories. Based on these findings, we choose smaller values of i for computing the likelihood of trajectories and estimating when to replan.

E Additional Experimental Results

We show the additional experimental results in the following.

We show the results of the comparison between our RDM algorithm and the replanningbased baseline algorithms (SDM and PRDM) across more tasks. We run experiments using different intervals or thresholds and measure their computational costs by the total number of diffusion steps. Notably, RDM consistently outperforms all the baseline algorithms under the same total diffusion steps (that is, with the same computational budget).

We investigate different intervals I for replanning for Diffuser and Decision-Diffuser. We observe that as the interval decreases (that is replanning is done more frequently), the performance improves as we expect. However, when the interval is smaller than a certain value, the performance decreases significantly. For example, in the Maze2D Large domain shown in Figure VI, the return increases when the interval decreases from 250 to 100 , while it drops when the interval decreases from 100 to 1 . The ablation results confirm our statement that replanning with an interval that is too small (for example, replanning at every time step) may prevent successful task execution.

We also analyze the impact of different thresholds l for the baseline algorithms, SDM and RDM. As we expect, when l decreases (that is, when replanning is done more frequently), the performance of all the methods has improved. Despite the performance differences of the baseline algorithms, we want to emphasize again that under the same computational budget, our RDM algorithm outperforms all the baseline algorithms.

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Total Diffusion Step

1e3

130

140

150

160

170

180

190

Normalized Returns

Maze2D Large

Diffuser

SDM

PRDM

RDM

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Total Diffusion Step

1e3

140

150

160

170

180

190

200

Normalized Returns

Diffuser

SDM

PRDM

RDM

1.00

1.25

1.50

1.75

2.00

2.25

2.50

Total Diffusion Step

1e4

45

50

55

60

Normalized Returns

Hopper Med-Expert

DD

SDM

PRDM

RDM

1.4

1.6

1.8

2.0

2.2

2.4

2.6

2.8

Total Diffusion Step

1e4

65

70

75

80

85

90

95

Normalized Returns

Walker2d Med-Expert

DD

SDM

PRDM

RDM

1.4

1.5

1.6

1.7

1.8

1.9

Total Diffusion Step

1e3

10

20

30

40

50

60

70

Normalized Returns

Close Box

DD

SDM

PRDM

RDM

0

50

100

150

200

250

Interval

110

120

130

140

150

160

170

Normalized Returns

Maze2D Large

Diffuser

20

40

60

80

100

Interval

10

20

30

40

50

Normalized Returns

Close Box

DD

1

2

3

4

5

6

State Distance Threshold

140

150

160

170

180

Normalized Returns

Maze2D Large

SDM

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

State Distance Threshold

44

46

48

50

52

54

56

Normalized Returns

Hopper Med-Expert

SDM

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

1e

2

150

160

170

180

190

Normalized Returns

Maze2D Large

RDM

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1e

1

35

40

45

50

55

60

65

70

Normalized Returns

Close Box

RDM

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Probability

1e

2

35

40

45

50

55

60

65

Normalized Returns

Hopper Medium

DD

SDM

PRDM

RDM

(a) interval=255

(b) interval=50

(c) interval=1

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the online replanning problem within diffusion-based models by introducing their method, RDM, which uses the likelihood of trajectories to decides when and how to replan.\n\nSTRENGTHS:\n- The topic of when and how to replan is an important one, particularly within the diffusion model community where it has not, to the best of my knowledge, been addressed before.\n- The methodology section appears to be sound, and the idea of choosing to replan and what type of replanning should happen based on the trajectory likelihoods is compelling.\n- The paper is clear, well-written and easy to follow.\n\nWEAKNESSES:\n- The experimental results are lacking in terms of comparison with other simple baselines that are able to replan. While the authors partially address this in the ablation study of Section 4.4, it is unclear why these are also not used in the main experiments. Given the aim of the paper is to provide a good replanning strategy, I do not believe this should be an ablation only, but rather that most results should include “no replanning baselines” (i.e., the ones already provided), and “replanning baselines” as proposed in the ablation. This would allow a reader to take conclusions regarding the actual improvement from performing no replanning to a basic strategy to the proposed method.\n- The absence of replanning in the baselines could justify the significant drop of performance in the results in Section 4.1, 4.2 and 4.3. A comparison with replanning baselines (both fully and, for example, with a randomly selected number of steps) strengthens the case that the “when” and “how” to replan are key to the success of the results.\n- The paper is also lacking an ablation with a variable number of replanning steps. In Section 4.4 the authors compare DD which replans “using two fixed intervals”, but how does the performance change for a higher replanning frequency? \n\n**Minor comments**:\n- Typo in abstract, line 1, should be “have risen as a promising”\n\nQUESTIONS:\n- How does this method compare to a fixed, fully replanning strategy at different intervals in the main scenarios compared in the experiments?\n- How are the thresholds in Algorithm 1 tuned, and how does the method perform for different $l_1$/$l_2$ values?\n\nLIMITATIONS:\nThe authors address some limitations of their method in Section 6 of the paper, and there are no concerns regarding potential negative societal impact for this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission describes a method for deciding when and how to \"replan\" when using diffusion models for inferring plans, as in Decision Diffuser [1]. The task is essentially imitation learning (IL): given a training dataset of plans and features, the task is to predict a new plan given novel features. As in [1], the IL method consists of training a diffusion model (DM) to predict state sequences given contexts, and the features happen to include the reward of the plan. A high-reward plan can therefore be inferred by conditioning the model on a high reward value.\n\nThe replanning aspect of this work is useful because running the diffusion model may be expensive, making it difficult to use diffusion-model-based planning to do receding-horizon control. Replanning only when necessary saves computation compared to planning every cycle.\n\nSTRENGTHS:\nOriginality: \n\nThe ideas of trying to infer when and how to replan when planning with diffusion models seem novel. I'm not aware of other work on inferring when to replan in general, so that general idea may be novel as well. A nice feature of applying probabilistic methods to planning is that it becomes quite natural to ask such questions.\n\nQuality:\n\nThere are some promising aspects to the experiments. I appreciated that the experiments were targeted towards answering specific questions that were clearly articulated in lines 182-185. The robotic control (RLBench) results were also promising in that the success rate was significantly higher than that of a baseline without replanning.\n\nSignificance:\n\nThe results show that simply replanning based on some simple heuristics is a viable technique for boosting the performance of diffusion-based planning methods. This is a potentially useful and interesting observation for anyone interested in applying diffusion models to solving planning problems in practice.\n\nWEAKNESSES:\nQuality:\n\nI believe the weakest part of the submission is that the experiments are focused mainly on \"apples-to-oranges\" comparisons of RDM to model-free RL and decision diffusion models without replanning. None of the baselines have the ability to respond to new information at test time, as far as I can tell. In the case of the model-free RL algorithms, this is a bit unclear—theoretically, the learned policies could condition on \"real-time\" information, but it's unclear whether this is the case. Still, even if the RL policies do condition on novel information, comparing a policy learned offline to a planner that can be evaluated online, seems a bit unfair.\n\nIt would make more sense to me if the experiments were focused more on comparisons between RDM and other replanning methods. Since the goal of intelligent replanning is to reduce computation with respect to replanning every cycle, I think it would also be fair to evaluate the net effect of different replanning strategies on the tradeoff between solution time and solution quality. For example, if we replan at fixed intervals, but evaluate that for a range of intervals, what trade-off do we see between solution time and quality compared to RDM? If we replan every cycle, is the solution quality much higher than RDM? \n\nAlthough the ablation study in section 4.4 does analyze the effects of different replanning strategies, this analysis feels unsatisfying for a few reasons. First, it is unclear how the threshold / interval for replanning were chosen for DD and SDM, which simply replan based on time or state distance thresholds. Ideally, these thresholds would be chosen to maximize some metric evaluated on a validation set, but I could find no details about this. \n\nI also note that the baseline for figure 7 is not zero, which gives a very misleading impression about the relative performance gap between (e.g.) SDM (replan triggered by state distance) and RDM, which is actually pretty small. There are also no error bars on this plot, so it is hard to tell whether this result is significant. This experiment was also only run on a single task (hopper-medium-expert)—the results would be much more interesting if these simple baselines (DD, SDM) were run on the entire suite of tasks. If it turns out that SDM performs almost as well as RDM on most tasks, then that would make RDM significantly less appealing.\n\nClarity:\n\nThe problem statement is  unclear. On my initial read-through of the paper, I was under the impression that RDM (the proposed method) was an imitation learning (IL) method—until I reached the experiments, which compared RDM to RL methods. I then backtracked to Section 2.1 (problem setting), which states:\n\n\"We consider a trajectory optimization problem similar to Diffuser[17]... the objective of the trajectory optimization problem is to find a sequence of actions … that maximizes J… which is the expected sum of rewards over all timesteps.\"\n\nAfter reading the reference for Diffuser[17], this paragraph convinced me that my original belief was wrong, and that RDM is in fact an (online) RL method that uses a diffusion model as a planner—just like Diffuser[17]. I then at some point encountered a Decision Diffuser[1] reference in the paper and decided to revisit that reference to refresh my memory. I was very confused because the paper was not what I remembered it being, and in fact Decision Diffuser[1] is an IL method.\n\nI eventually realized that Diffuser[17] and Decision Diffuser[1] are different methods: the first is an RL method, and the second is an IL method. However, they both share common authors, use similar methods, and have similar names. I then re-read lines 89-91, (\"we follow the formulation of planning in Decision Diffuser[1]…\") which convinced me that my original impression was correct: RDM is an IL method, not an RL method.\n\nDespite the unfortunately similar names, I believe the real root of my confusion is that the paper lacks a crisp problem statement—the reader should not have to guess as to whether the problem addressed is IL, or offline/online off-policy/on-policy RL. It's unfortunate that Section 2.1 (Problem Setting) introduces the method in RL terms and explicitly states that the problem is similar to that of Diffuser[17], which is an RL method. Instead, that paragraph should probably state something like \"Our work addresses a problem similar to that described in Decision Diffuser[1]: given a dataset of training state trajectories and rewards, our task is to predict a state trajectory similar to what was observed in the training dataset, while conditioning on a query reward value. This is essentially an Imitation Learning problem where we observe and condition on rewards, but it is also similar to RL, in the sense that the goal is to produce a trajectory with high reward.\"\n\nThe comparison to RL methods also confuses things—the submission should be clearer about why RDM is compared to RL methods. It would also be beneficial to call Diffuser by a different name (e.g., Planning Diffuser) to disambiguate it from Decision Diffuser.\n\nThe exposition of the method could also be significantly improved. The explanation of when to replan in section 3.1 is a bit too verbose and doesn't convey the basic idea well, which is simply to evaluate the likelihood of a plan where states/actions at previous timesteps are replaced with those observed during plan execution. A simple figure would be very helpful here.\n\nSignificance:\n\nOne factor that may limit the significance of this work is that the topic is relatively niche—it attempts to solve a particular problem (performance of replanning) with applying a particular planning method (Decision Diffuser), which itself is still relatively immature. Solving niche problems is ok, but the potential for significant impact would be greater if there were  some interesting take-away for a more general audience.\n\nQUESTIONS:\nHow were the intervals / state distance thresholds set for the DD and SDM baselines in figure 7?\n\nHave you tried evaluating the trade-off between solution quality and planning time for different replanning strategies (e.g., for different possible threshold values in DD and SDM)?\n\nLIMITATIONS:\nThe paper adequately addresses the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript introduces a technique for enhance the motion planner rooted in diffusion models, encompassing decisions concerning the timing of replanning and plan trajectories upon the existing path. The strategy for timing replanning utilizes the inherent estimated likelihood of the trajectory in diffusion models as a criterion for replanning. In terms of forming new trajectories, the authors either completely reconstruct the plan or modify it based on future contexts determined by specific guidelines. The concept is clear-cut, and the outcomes appear to be quite promising.\n\nSTRENGTHS:\nReplanning is important for robots to execute trajectories robustly, deciding when to replan and replanning algorithm are the core challenges of replanning problem. Tackling these problems can help a lot with diffusion based robot motion planning methods. Moreover, the overall writing quality is good, and the core idea is presented well.\n\nWEAKNESSES:\nIt's not surprise that the performance gets better by adding replanning strategy into diffusion model. For long horizon planning problem in Section 4.1, the authors compared performance of RDM and baselines, however, none of the baselines contains replanning strategy, making the effectiveness of replanning timing decision method proposed by the author hard to evaluate. This also happens to Robotic Control tasks in Section 4.3. \n\nAs for stochastic environment in Section 4.2, the author demonstrates that RDM outperforms baseline models in environment with stochastic transition models. How different levels of stochasticity will affect the performance is later proved in Section 4.4. However, only one environment is tested, and the performance of RDM and other baselines are relatively close, which is inadequate to evaluate the effectiveness of replanning strategy in RDM.\n\nIn Section 4.4, the author compares the performance for models using different replanning strategy, and comparison of different fixed interval/different threshold of state distance deviation is missing, which makes the performance of DD/SDM not convincing enough.\n\nThere are also some points which are not demonstrated very well:\n- The detailed implementation of how the replanned trajectory is generated is missing.\n- The theoretical proof of the effectiveness of likelihood function for partially-executed trajectory is missing, and the threshold of likelihood to choose how to replan is selected empirically which may significantly impact the performance.\n\nQUESTIONS:\n- For long horizon planning tasks and robotic control tasks, is using likelihood better than using state distance deviation as replanning criteria?\n- Comparing baselines with different replanning timing decision methods and RDM, how much efficiency improvement will RDM achieve?\n- For stochastic environment, the advantage of methods implementing replanning strategy comparing to non-replan methods should be increased when stochasticity arises, but the performance gets closer shown on Figure 6. Why?\n\nLIMITATIONS:\nThe author addressed the limitations well.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Replanning with Diffusion Models (RDM), which utilizes an internally estimated likelihood of the current plan to determine when to perform replanning. The authors propose various strategies for replanning in different scenarios.\n\nSTRENGTHS:\n1. The introduction of Replanning with Diffusion Models (RDM) and the use of an internally estimated likelihood of the current plan for replanning strategies is novel.\n2. The experimental results demonstrate impressive performance in robot control and stochastic environments, surpassing baselines like IQL.\n\nWEAKNESSES:\n1. The validation of the proposed method is not fully comprehensive. In RDM, a key aspect is the estimation of likelihood, but the authors did not discuss the accuracy of the estimation or its impact on final performance.\n2. The paper contains some language issues and typos that should be carefully reviewed. For example, in line 178, it should refer to Figure 7 instead of Table 7. Additionally, in line 37, \"we propose a principled approach to xxx\" has a grammatical error.\n3. In the robot control experiments using the RLBench domain, comparing IQL or DT as baselines might not be sufficient. It would be beneficial to compare against other agents known for performing well in the RLBench domain.\n\nQUESTIONS:\n1. I am concerned about the computational resource requirements introduced by the Diffusion Models. For achieving similar performance levels, could you provide insights into the training time and resources needed for IQL, DT, and RDM?\n2. While I understand the advantages of replanning, which reduce unnecessary exploration, in RL learning, accurate judgments cannot be made without visiting certain states. How can you ensure that the states not visited after replanning are indeed redundant for learning?\n3. The direct introduction of noise into the D4RL dataset (Sec. 4.2) seems unreasonable. Why not collect data directly from random environments to validate this point?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to effectively replan with diffusion models. The authors propose an adaptive online replanning strategy using diffusion models. This strategy employs the estimated likelihood of a plan's success to determine when replanning is needed, avoiding frequent, computationally expensive replanning. It also ensures new plans align with the original trajectory's goal, leveraging previously generated plans efficiently. This method led to a 38% performance improvement over previous diffusion planning approaches on Maze2D and enabled the handling of stochastic and long-horizon robotic control tasks.\n\nMain Contributions:\n\n1. A method to determine when to replan with diffusion\n2. A method to generate new plans while utilizing the existing plan\n\nSTRENGTHS:\n- **Movitation**: Online re-planning is a natural and important complement to many planning methods, thus the motivation of this work is important.\n\n- **Methodology**: This paper proposes two methods: one is used to determine when to replan while the other one is for how to replan. The two methods worked together to conduct effective online replanning with diffusion models. As far as I know, the proposed methods are novel.\n\n- **Performance Improvement**: The proposed method significantly improves the performance of diffusion planners, with a reported 38% gain over past diffusion planning approaches on Maze2D. And the authors also showcased that the proposed method enable the handling of stochastic and long-horizon robotic control tasks.\n\nWEAKNESSES:\n- It is still unclear to me why replaning at every time step does not work. In the figure 7, the Dcesion Diffuser replans at fixed intervals and it performs much worse than the proposed method. However, the authors did not mention how large is the interval is. If the replaning frequency is very low than it should apprently works worse than the proposed method. And in the abstract, the author mentioned that \"replanning at each timestep may prevent successful task execution, as different generated plans prevent consistent progress to any particular goal\". I am not convinced by this argument without any exmaples or further explanations.\n\n- It is also unclear to me why \"Replan on State Distance (SDM)\" works so badly according to Figure 7. In Section 3.1, the authors mentioned that \"However, in many cases, even if the actual state the agent is in matches with the state in the plan, new information about the environment may make the current plan infeasible.\" This does not make too much sense to me. I appreciate it if the authors could provide further details about this point.\n\n- Computation budget is an important factor to consider when comparing different methods. (If we have an unlimited computation budget, I feel it is a good idea to replan at each timestep.) However, the numbers of replanning used in each baseline are not present in the paper. If the numbers of replanning vary a lot among different methods, then I would not think it is a fair comparison. I wish the authors could provide the numbers of replanning used in each method. \n\n- Although the proposed method aims to avoid frequent, computationally expensive replanning, it's not clear how computationally efficient the new method is, and whether the computational cost of determining when to replan offsets the benefits gained from less frequent replanning.\n\nQUESTIONS:\n- It would be great to provide the number of \"replan from scratch\" and the number of \"replan with future context\" used in each experiment, since this would help the audience further understand how the replanning helps the agent.\n- Why does \"replan with future context\" work better than \"replan from previous context\"? Could you give any intuitive explanations?\n- See the weakness section for other questions.\n\nLIMITATIONS:\n- This method only applies to diffusion models as it adds noises at some diffusion steps. It is unclear how to utilize this method in other planning methods.\n- See the weakness section for other limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}