Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans
Authors: Kyowoon Lee
Seongun Kim
Jaesik Choi
Venue: NeurIPS 2023 poster (2023)
Keywords: Offline Reinforcement Learning
Trajectory Optimization
Diffusion Models
Sequential Decision Making

Abstract: Diffusion-based planning has shown promising results in long-horizon, sparse-reward tasks by training trajectory diffusion models and conditioning the sampled trajectories using auxiliary guidance functions. However, due to their nature as generative models, diffusion models are not guaranteed to generate feasible plans, resulting in failed execution and precluding planners from being useful in safety-critical applications. In this work, we propose a novel approach to refine unreliable plans generated by diffusion models by providing refining guidance to error-prone plans. To this end, we suggest a new metric named restoration gap for evaluating the quality of individual plans generated by the diffusion model. A restoration gap is estimated by a gap predictor which produces restoration gap guidance to refine a diffusion planner. We additionally present an attribution map regularizer to prevent adversarial refining guidance that could be generated from the sub-optimal gap predictor, which enables further refinement of infeasible plans. We demonstrate the effectiveness of our approach on three different benchmarks in offline control settings that require long-horizon planning. We also illustrate that our approach presents explainability by presenting the attribution maps of the gap predictor and highlighting error-prone transitions, allowing for a deeper understanding of the generated plans.

Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans
Kyowoon Lee"
Seongun Kim
Jaesik Choi KAIST, INEEJI
UNIST
KAIST
Leekwoon@unist.en ac.
seongun@kaist.ac.kk
jaesik. chosik.chiieee @kwikkaaitt...moo kr
Abstract
Diffusion-based planning has shown promising results in long-horizon, sparse- reward tasks by training trajectory diffusion models and conditioning the sampled trajectories using auxiliary guidance functions. However, due to their nature generative models, diffusion models are not guaranteed to generate feasible plans, resulting failed execution and precluding planners from being useful in safety- critical applications. In this work, we propose novel approach to refine unreliable plans generated by diffusion models by providing refining guidance error- prone plans. To this end, we suggest new metric named restoration gap for evaluating the quality of individual plans generated by the diffusion model. A restoration gap is estimated by gap predictor which produces restoration gap guidance refine a diffusion planner. We additionally present an attribution map regularizer to prevent adversarial refining guidance that could be generated from the sub-optimal gap predictor, which enables further refinement of infeasible plans. We demonstrate the effectiveness of our approach on three different benchmarks in offline control settings that require long-horizon planning. We also illustrate that our approach presents explainability by presenting the attribution maps of the gap predictor and highlighting error- pronn transitions, allowing for a deeper understanding the generated plans.
error-prone
Introduction
Planning plays a crucial and efficient role in tackling decision-making problems when the dynamics are known, including board games and simulated robot control (Tassa et al. 20122 Silver et al.) 2016
2017 Lee al.) 2018). To plan for more general tasks with unknown dynamies, the agent needs to learn the dynamics model from experience. This approach is appealing since the dynamics model independent of rewards, enabling
adapt new tasks in the same environment, while also taking advantage of the latest advancements from deep supervised learning to employ high-capacity models.
The most widely used techniques for learning dynamics models include autoregressive forward models (Deisenroth & Rasmussen 2011 [affner et al., 2019: Kaiser et al., 2020), which make predictions based future time progression. Although an ideal forward model would provide significant benefits, there is key challenge that the accuracy of the model directly affects the quality of the plan. As model inaccuracies accumulate over time (Ross & Bagnell| 2012 Talvitie 2014
Luo et al., 2019; Janner et al.] 2019 Voelcker et al.) 2022), long-term planning using imprecise models might yield sub-optimal performances compared to those achievable through model-free techniques. Building upon the latest progress generative models, recent studies have shown promise in transforming reinforcement learning (RL) problems into conditional sequence modeling, through the modeling of the joint distribution of sequences involving states, actions, and rewards
Equual Contribution
Codes are available at http:p////gghhhub..cc///////////
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Low Restoration Gap
8
Restoration Gap
High
Input Plan
Perturbed Plan
Restored Plan
Figure 1: Illustration of two plans with low/high restoration gaps with a specified start and goal For each imput plan, first perturb it using Gaussian noise. We then remove the noise from the perturbed plan by simulating the reverse SDE which progressively transforms the perturbed plan into the initial plan by utilizing the score function (Section 2.2 The restoration gap is then computed as the expected L2 distance between the input plan and the plan restored from noise corruption (Section 3). The top example exhibits smaller restoration gap because of its successful restoration close to the original plan, while the bottom example has a larger restoration gap due to its poor restoration performance, Plans restored from various noise corruptions are differentiated by distinct colors.
For instance, Diffuser (Janner et al. 2022) introduces an effective framework for generating trajectories using a diffusion model with flexible constraints the resulting trajectories through reward guidance the sampling phase. Although these approaches have achieved notable performance on long-horizon tasks, they still face challenges generating outputs with unreliable trajectories, referred limited performance and unsuitability for deployment in safety-critical applications.
(Lambert al. 2021 Chen al. 2021 Janner al. 2021
2022
artifacts, resulting in
This paper presents orthogonal approach aimed enhancing the plan quality of the diffusion model. We first propose a novel metric called restoration gap that can automatically detect whether generated plans feasible error probabilities under regularity conditions. The restoration gap directly evaluates the quality of generated plans measuring their restorability through diffusion models in which plans exposed to certain degree of noise, as illustrated in Figure [. A restoration gap is estimated by a function approximator which we name gap predictor. The gap predictor provides an additional level of flexibility to the diffusion model, and we demonstrate its ability to efficiently improve low-quality plans by guiding the reduction of the estimated restoration gap through process, which we call Restoration Gap Guidance (RGG). Furthermore, we propose regularizer that prevents adversarial restoration gap guidance by utilizing an attribution map of the gap predictor. effectively mitigates the risk of the plan being directed towards an unreliable plan, enabling further improvement in the planning performance.
We theoretically analyze that it could detect artifacts with bounded
The main contributions of this paper are summarized as follows: (1) We provide a novel metric to the quality of individual plans generated by the diffusion model with theoretical justification. (2) We propose a new generative process, Restoration Gap Guidance (RGG) which utilizes a gap predictor that estimates the restoration gap. (3) We show the effectiveness of approach across three different benchmarks in offline control settings.
Background
2.1 Planning with Diffusion Probabilistic Models
We consider the reinforcement learning problem which aims to maximize the expected discounted sum of rewards Ex, IT 7'r(s;, a:)] where is a policy that defines distribution actions

---

represents the states that undergo transition according to unknown discrete-time dynamics
S1+1 fs, az), S x A + I is a reward function, and y (0, 1] is the discount factor.
Trajectory optimization solves this problem by finding the sequence of actions dò.r that maximizes the expected discounted sum of rewards over planning horizon T:
T
T = arg max [TT = arg max dorr
y y'r(s,,))) ar);
ao:?
t=U
where т = (8o: Agg 81, 01, 8t, at) represents trajectory and (T) denotes an objective value of that trajectory. This trajectory be viewed as particular form of two-dimensional sequence data:
80
81
8T ar
Q1
Diffuser (Janner et al. 2022) a trajectory planning model, which models a trajectory distribution by employing diffusion probabilistic models (Sohl-Dickstein al. 2015 Ho al. 2020):
Po( = D D(T II Po dr
/
where p(r-)) is a standard Gaussian prior, 7 is a noiseless trajectory, and pa(T<-1(7') is denoising process which is reverse of forward process q(-')=-11) that gradually deteriorates the data structure by introducing noise. The denoising process is often parameterized as Gaussian with fixed üineste--depndden covariances: Po (-² = N((''>1lup("" i), E"). Diffuser recasts the trajectory optimization problem as a conditional sampling with the conditional diffusion process under smoothness condition on IO = 1|T) (Sohl-Dickstein et al. 2015):
(T) p(T 1) < ((TT)((((()))))) = 1|T), po .1.)) N(T ; Eg, E
where E are the parameters of the denoising process pe pe( (T'), Or is the optimality of timestep t of trajectory with p(O, 1) exx((""((x_ az)) and
T
g = V
Σ 1=0
7' V ( = VJ(M).
(5)
Therefore, a separate model Jg can be trained predict the cumulative rewards of trajectory samples T'. By utilizing the gradients of Jg. trajectories with high cumulative rewards can be generated.
As part of the training procedure, Diffuser trains an e-model to predict the source noise instead of training Mg it turns out that learning eœ enables the of simplified objective, where M4 is easily recovered in a closed form (Ho et al. 2020):
( == E, |||e co((",
where i E (0. 1,.., trajectory corrupted by the noise from the noiseless trajectory 70.
N} is the diffusion timestep, 6 ~ (0 I) is the target noise, and 7 is the
2.2 Generalizing Diffusion Probabilistic Models Stochastic Differential Equation (SDE)
The forward process diffusion probabilistic models perturbs data structure by gradually adding Gaussian noises. Under infinite number of noise scales, this forward process over continuous time can be represented as a stochastic differential equation (SDE) (Song et al. (202T):
(7) where t (0, 1] is continuous time variable for indexing diffusion timestep, f(T, t) is the drift coefficient, g() is the diffusion coefficient, and w is the standard Wiener process. Similarly, the denoising process be defined by the following reverse-time SDE:
dr f(T, t) di + g()
dr = [f(T,t) t) - " t) dt + g(t) dw.
(8)
3

---

where W is the infinitesimal noise in the reverse time direction and sø(T,t) is the learned score network which estimates the data score V V log (.)). This score network can be replaced by the e-model:
CO(T, t) C2
EO(T, C
tt V ( = E V log (()7()))] 1
(9)
where C2 is a constant determined by the chosen perturbation strategies.
The solution of forward SDE is time-varying random variable r², Using the reparameterization trick (Kingma & Welling 2014), it is achieved by sampling random noise E from standard
Gaussian distribution which is scaled by the target standard deviation or and shifted by the target mean:
T'= aat + oie, c~N(O,I).
(10)
where or : 1] + [0.1] 1] denotes scalar function indicating the magnitude of the noiseless data T9, and o+ : [0, - [0, oo) denotes scalar function that determines the size of the noise c. Depending on perturbation strategies for a and or, two types of SDEs are commonly considered: the Variance Exploding SDE (VE-SDE) has ag = 1 for all I; whereas the Variance Preserving (VP) SDE satisfies for all L. Both VE and VP SDE change the data distribution to random Gaussian noise as moves from to I. In this work, we describe diffusion probabilistic models within the continuous- timee framework using VE-SDE simplify notation, as VE/VP SDEs are mathematically equivalent under scale translations (Song et al. 2021).
of + of
For VE SDE, the forward process and denoising process are defined by the following SDEs:
dio dw
(Forward SDE) dr
(11)
di
dioi'
diož
(Reverse SDE) dr
(T sp (T, t)
dt
+
dw.
(12)
df
df
3 Restoration Gap
To assess the quality of plans generated by diffusion probabilistic models, we propose novel metric named restoration gap. It aims to automatically detect infeasible plans that violate system constraints. We hypothesize that for feasible plans, even if certain amount of noise perturbs them, they can be closely restored to their initial plans by diffusion models. It is attributed to the property of temporal compositionality in diffusion planners (Janner et al. 2022) that encourages them compose feasible trajectories by stitching together any feasible plan subsequences. However, for infeasible plans that obviously fall out of the training distribution as they violate physical constraints shown in Figure [5] restoring them to a state near their original conditions is challenging. Based on this intuition, we define the restoration gap of the generated plan as follows:
perturb;(T) = + ogEi
Ei ~ N(0.1
(13)
[ dio
dio dw
restore = +
- dt so (T, t)
dt +
(14)
dt
restoration gapi,o = Ee, [||T - restore; (perturb;(T))lla]
(15)
where i E (0, 1] indicates the magnitude of applied perturbation. The restoration gap measures the expected Le distance between the generated plan and the plan restored from noise corruption, which estimated by the Monte Carlo approximation.
Figure 2 provides empirical evidence supporting our hypothesis. In order to analyze the effectiveness of the restoration gap, we define artifact plans generated by Diffuser (Janner et al.) 2022) that involve transitions of passing through walls for which is impossible for the agent to follow. We compare the distribution of the restoration gap for both groups, normal plans and artifact plan|]] The histogram of
the restoration gap for normal and artifact plans demonstrates that infeasible artifact plans have larger
'The purpose of detining artifacts in this manner is solely to validate our hypothesis. Artifacts are not explicitly defined beyond the scope of this validation.

---

Artifacts
Normal
Artifacts Nocuaa
Density
-1 14 122 -4
70
100
106
2000
3000 Restoration Gap (ours)
1000
8000
10000
1200
2000
Negative Realism Score
Rarity Score
Figure 2: The first and second rows show examples artifact and normal plans, respectively, gener- ated by Diffuser (Janner al. 2022) in the Maze2D-Large environment, including predetermined start and goal ( The third row presents the density of realism score (Kynkäänniemi et al. 2011))
rarity score (Han al. 2023), and restoration gap illustrate the differences distribution between artifacts and normal plans. Detailed explanation of other metrics is described in Appendix C.2
restoration gap values compared to normal plans. Therefore, the detection of infeasible artifact plans can be automated by incorporating a statistical test that utilizes the restoration gap and thresholding with threshold value of b 0:
restoration gT);,e(T) b.
(16)
To bound the probability of making errors by choosing the specific threshold b, we provide Proposition I Let Hy represent the null hypothesis which assumes that the trajectory belongs the normal set normal. and let HI, represent the alternative hypothesis which assumes that the trajectory belongs to the artifact set Tartifacts- The following proposition suggests how to choose the threshold in order to bound the error probabilities.
Proposition Given E [0, 1] and a positive constant C, A, assume that (So(T,)))) < C2 for all Toormal C Rdd, and ||So (T, 1) Z (C A)2 for all T Tartifacts C Rdd If
2vd 2 d + 2 log 2 log
(17)
oi
then setting
b b2 oi Co,+ Vd++ Vd 2 - log 2log
(18)
guarantees both type I and type Il errors at most 26.
Proof Sketch. We begin by deriving thresholds by and bTz control type I and type II errors at most respectively. This is done by decomposing the restoration gap into the outcomes of the score and Gaussian noise. To ensure the control of both type and type II errors, examine the condition bi < b and obtain the conclusion. For the complete proof, see Appendix
According to Proposition to achieve low probabilities for both type (false positives, where normal trajectories are incorrectly classified as artifacts) and type II (false negatives, where artifact trajectories are wrongly identified as normal) errors, is essential properly satisfy the condition, which implies having large enough f. In practice, we find that setting f = 0.9 works well.
have large enough o; to

---

Refining Diffusion Planner
4.1 Restoration Gap Guidance
Although Diffuser (Janner et al., 2022) has demonstrated competitive performance against previous non-diffusion-basse planning methods by utilizing gradients of return To guide trajectories during the denoising process:
dr = d - (et))-(()P((o(77) aVV g(t) dw.
(19)
it entirely relies on the ability of a generative model and assumes perfect data score estimation. For plans with inaccurately estimated scores, the diffusion models could generate unreliable plans that are infeasible to execute and lead to limited performance. To address this, it is essential to construct an adjusted score refine the generative process of the diffusion planner. Therefore, estimate the restoration gap by training a gap predictor Gy on synthetic diffused data generated through the diffusion process, taking full advantage of its superior generation ability with conditional guidance from gradients of return. Parameters of the gap predictor & are optimized by minimizing the following objective:
(vv) Ex, [|resstrrtooo g Gw(T', ,)]1)
(20)
where E (0, 1] denotes a continuous time variable for indexing the diffusion timestep, and T' is the diffused trajectory resulting from 70 at diffusion timestep With this gap predictor, we define the Restoration Gap Guidance (RGG) as follows:
dr [f(TT g(t)² ( (V + +g(t dw,
(21)
where a is a positive coefficient that scales the overall guidance and /3 is positive coefficient that can be adjusted enforce small restoration gap for the generated trajectory.
4.2 Attribution Map Regularization
Although guiding the diffusion planner to minimize the restoration gap effectively refines low- low-quality plans (more details in Section(5). this refining guidance could push the plan in an undesirable direction due to the estimation егтоr of the gap predictor during the denoising process. As a result of this estimation error, guiding plans with the sub-optimal gap predictor may result in model exploitation (Kurutach et al. 2018 Janner et al.) 2019 Rajeswaran et al. 2020), yielding sub-optimal results.
To mitigate the issue of adversarial guidance, we present a regularization method that prevents the gap predictor from directing plans in the wrong direction. Inspired by the prior studies which improve the model performance by utilizing at- tribution maps (Nagisetty et al.) 22020] Bertoin al. 2022). we measure total variation of the attribution map M obtained from any input attribution methods
Plannng Performance
2-10
2
(TG t)). Each el- ement of the attribution map indicates the extent to which the final prediction influenced by the corresponding input feature. The rationale of employing the total variation of M lies in the hypothesis that transitions with excessively high attribution scores are more likely to be outliers. This because a sequence of transitions within a planned trajectory, rather than a single one, causes a plan to have high restoration gap. By adding this attribution map regularization, Equation 21 becomes:
Plamnng Budget (# of Phmns)
Figure 3: Planning performance of RGG+ on M-ze2D-Large single-task with varying values.
dr [f(T, 9())² (s) ((t()
+ (JT(TT) (V. (VIVVMI)) di g(t) dw,
(22)
where X is a control parameter given by positive constant, encouraging the attribution map to have simple, organized structure while preventing the occurrence of adversarial artifacts. We refer to this modification as RGG+.

---

Table The performance of RGG, RGG+, and various previous algorithms, measured as normalized
average return, is presented on the D4RL locomotion benchmark (Fu et al. 20200). Results for RGG and RGG+ show the mean and standard error over 15 planning seeds. Detailed sources for the
performance of prior methods are provided in Appendix F.
Dataset
Environment
BC
CQI.
IQI.
17T
TT
MOPO
MORel.
MBOP
Diffuser
RGG
RGG+
Med Med-Expert
HallCheetsh
55.2
91.6
86.7
86.8
95.0
63.3
53.3
105.9
79,8
90.8 4 0.3
91.2 0,3
Med- Expert
Hupper
52.5
110.0
55.1
109.9
Mod--Espent
Walker2d
107.5
108.8
109.6
108.1
101.9
44.6
95.6
70.2
108.4
107.8 + 0.1
107.7 0.2
Medium
HaltCheetah
42.6
44.0
47.4
42.6
42.3
42.1
44.6
44.2
44.0 +
44.2 0.3
Medium
Hopper
52.9
58.5
66.3
67.6
61.1
28.0
95.4
48.8
58.5
82.5 +:
84,9
Medium
Walker2d
75.3
72.5
78.3
74.0
79.0
17.8
77.8
41.0
79.7
81.7 +: 0.5
82.0 0.4
Read-Replay
HallCheetah
36.6
36.6
42.3
42.2
41.0
0.2
Mod-Replay
Hopper
18.1
95.0
94.7
82.7
91.5
67.5
93.6
12.4
96.8
95.2
95.8 0.5
Med Replay
Walker2d
26.0
77.2
73.9
66.5
82.6
39.0
49.8
9.7
61.2
78.3 1: 4.4
77.5 4.7
Average
74.7
47.8
Experiments
We present the analytical results of approaches to improve planning performance by leveraging guidance from our proposed metric, the restoration gap, for a wide range of decision-making tasks in offline control settings. Specifically, we demonstrate (1) the relationship between high restoration gap and poor planning performance, (2) the enhancement of planning performance in the diffusion planner by leveraging restoration gap guidance, and (3) explainability by presenting the attribution maps of the learned gap predictor, highlighting infeasible transitions. More information about experimental setup and implementation details can be found in Appendix C and Appendix [ respectively.
5.1 Relationship between Restoration Gap and Planning Performance
We evaluate how effectively the restoration gap can identify infeasible plans by comparing our metric with realism score [Kynkäänniee
Planning Porformance
155
et al. 2019) and rar- ity score (Han et al. 2023). Both prior metrics are designed assess the quality of generated samples by examining the discrepancy between the generated sample and the real data manifold in the feature space. Figure 4 iillustrates the performance of plans which are chosen up top-K% from each metric. As illustrated in Figure 4 the higher the restoration gap of the plan is, the poorer the performance is, which implies that the restoration gap captures the quality of the plan well compared to other metrics.
150
145
Negativ Realiem
140
Raill Store
Restoratoon on Gap
20
40
60
100
Top-K %
Figure Performance of plans chosen from the top-K % considering various metrics, in the Maze2D-Large single-task.
5.2 Planning Performance Enhancement
Maze2D Experiments Maze2D en- vironments (Fu al. 2020) involve a navigation task that requires an agent to exhibit long-horizon planning abilities to reach a target goal location. Maze2D environments consist of two tasks: a single-tast task where the goal location is fixed, and a multi-task which we refer to as Multi2D where the goal location randomized at the beginning of ev- ery episode. We compare our methods with the model-free offline reinforcement learning algorithms CQL (Kumar et al.) 2020) and IQL
Table 2: Diffuser with RGG and Diffuser with RGG+ out- perform baselines. We report the mean and the standard error over 1000 planning seeds.
Environment
CQL
MPPI
IQL
Dillaser
RGG
RGG+
Mace2D
U-Muze
5.7
33.2
47.4
1086 1 1.4
1088 =
109.5 1. 1.3
MaxeID
Medium
50
Large
12.5
123.5
1354-10
Single-task Average
7.7
16.2
47.0
120.6
125.3
128.5
127.9
1283
1 0.8
MuliiD Mali2n
Mediumm
15.4
12.1
130.1
4:
1300
130.0
Large
KD
13.9
141.2 + 1.6
148.3
150.9 + 1.3
Multi-task Average
21.5
16.9
133.1
135.5
136.4
(Kostrikov et al. 2022); conventional trajectory optimizer MPPI (Williams et 2015)) and sequence modeling approach Diffuser (Janner al. 2002). As shown in Table RGG improves the planning performance of Diffuser in 5 out of 6 tasks, with notable improvements in the Maze2D- envi-
7

---

ronments where the complexity of the obstacle maps is higher than in U-Maze or Medium layouts, leading to a higher occurrence of infeasible plans. RGG+ performs on par with or better than RGG. In contrast, model-free algorithms fail to reliably achieve the goal, as Maze2I) environments require hundreds of steps to arrive at the goal location.
Locomotion Experiments Gym-MuloCo locomotion tasks (Fu et al. 2020) are standard bench- marks in evaluating algorithms on heterogeneous data with varying quality. We compare our methods with the model-free algorithms CQL (Kumar et al. 2020) and IQL (Kostrikov et al. 2022); model- based algorithms MOPO (Yu et al. 2020), MOReL (Kidambi et al. 2020), and MBOP (Argenson
& Dulac-Arold 2021): sequence modeling approach Decision Transformer (DT) (Chen et al.
2021p. Trajectory Transformer (TT) (Janner et al. 2021 and Diffuser (Janner et al., 2022); and pure
umitation-based approach vehavior-cloning (BC). As indicated in Table our approach of refining
Diffuser with RGG either matches surpasses most of the offline RL baselines when considering the average score across various tasks. Additionally, it significantly enhances the performance of Diffuser, particularly in the "Medium" dataset. We attribute this improvement to the sub-optimal and exploratory nature of the policy that was used to generate the "Medium" dataset, which results in a challenging data distribution learn the diffusion planner. Consequently, RGG clearly contributes to the enhancement of planning performance. However, RGG+ only brings about a marginal im- provement over RGG. This might be because we adopt the strategy of (Janner et al. 2022). using a closed-loop controller and a shorter planning horizon in locomotion environments compared to Maze2D environments, thereby simplifying the learning process of the gap predictor.
Block Stacking Experiments The block stacking task suite with a Kuka iiwa robotic arm is benchmark to evaluate the model performance for a large state space (Janner) ct al. 2022) where the offline demonstration data is achieved by PDDLStream (Garrett) ct al. 2020). It involves two tasks: an un- conditional stacking task whose goal is to maximize the height of a block tower, and a conditional stacking task whose goal is to stack towers of blocks subject to specified order of blocks. We compare our methods with model-free offline reinforcement learning algorithms BCQ (Fujimoto et al., 2019) and CQL (Kumar et al.) 2020). and Diffuser (Janner et al. (022). We present quantitative results in Table 3 where a score of 100 corre- sponds to the successful completion of the task. The results demonstrate the superior performance of RGG over all baselines, with RGG+ further enhancing this planning performance.
Table 3: The performance of RGG, RGG+, and vari- ous prior methods evaluated over 100 planning seeds, A score of 100 is desired, while a random approach would receive a score of O.
Environment
BCQ
CQL
Diffuser
RGG
RGG+
Uncoeditional Stacking
0.0
53.3
2.4
63.3
65.3
Conditional Stacking
0.0
4.2
53.0 3.3
56.7
Average
0.0 8.1
48.8
58.2
61.0
5.3 Injecting Explainability to Diffusion Planners
The explainability of decision-maker models is particularly important in control domains they could potentially harm physical objects including humans (Kim & Choil [021 Lee et al., 2023 Beechey et al. 2023; Kim al., 2023) Kenny al., (023). Training the gap predictor enables the diffusion planner to have explainability. Diffusion planners often generate trajectories with unreliable transitions resulting in execution failures. Attribution maps from the gap predictor highlight such unreliable transitions by identifying the extent to which each transition contributes to the decision of the gap predictor. Specifically, in Maze2D, the attribution maps emphasize the transitions involving wall-crossing or abrupt directional changes, as illustrated in Figure 5 In the unconditional block stacking task where the robot destroys the tower while stacking the last block, the bower-breaking transitions are highlighted. On the other hand, for successful trajectories on the second and third attribution maps, the attribution maps do not emphasize picking or stacking behaviors. Similarly, in the conditional block stacking task where the robot fails to stack the block, they spotlight the transitions of stacking behaviors.
5.4 Additional Experiments
To study the benefit of regularization on harder tasks, characterized by a larger trajectory space and a smaller fraction of the space observed in training, we explore À values (0.0, 0.5, 1.0, 3.0, 5.0] while increasing the planning budget as illustrated in Figure 3 As the planning budget increases, À = generates adversarial plans, resulting in decreased performance. In contrast, RGG+ demonstrates
8

---

Medium
Maze2D
Large
Unconditional
Block Stacking
0
Conditional
Figure 5: Attribution maps for trajectories generated by diffusion planner highlight transitions that have substantial contribution to the estimation of high restoration gap by the gap predictor, indicated in red.
effectiveness across wide range of À values, with À 0 consistently outperforming À (i.e.. better than no regularization). Further investigation into the attribution method, perturbation magnitude, and comparison with guidance approaches, including metrics such the rarity score, the negative realism score, and the discriminator, as well as the visualization of low and high restoration gap plans, be found in Appendix
6 Related Work
Metries for Evaluating Generative Model Inception score (IS) (Salimans et al. 2016) and Fréchet inception distance (FID) (Heusel et al.] 2017) are commonly used as standard evaluation metrics for generative models, assessing the quality of generated samples by comparing the discrepancy between real and generated samples in the feature space. However, these metrics do not distinguish between fidelity and diversity aspects of generated samples. To address this issue, precision and
recall variants (Sajjadi et al.) 2018 Kynkäänniemi et al. (219)) are introduced to separately evaluate these properties. Subsequently. density and coverage (Naeem al., (220)) are proposed to overcome some of the drawbacks of precision and recall, such as vulnerability to outliers and computational inefficiency. While these metries are helpful for evaluating the quality a set of generated samples, they are not suitable for ranking individual samples. In contrast, realism score (Kynkäänniemi et al. 2019) and rarity score (Han et al. 2023) offer a continuous extension of improved precision and recall, enabling the evaluation of individually generated sample quality. Despite their usefulness,

---

these methods come with limitations as they rely on real samples for precise real manifold estimation, whereas our restoration gap does not have such a constraint.
Diffusion Model in Reinforcement Learning Diffusion models have gained prominence as notable class of generative models, characterizing the data generation process through iterative denoising procedure (Sohl-Dickstein et al. 2015;Ho et al.] (020). This denoising procedure can be
viewed as way to parameterize the gradients of the data distribution (Song & Ermon 2019). linking
diffusion models to score matching (Hyvärinen & Dayan
2005) and energy-based models (EBMs)
(LeCun et al. 2006 Du & Mordatch | Nijkamp et al.
2019 Grathwohl et al. 2020). Recently,
diffusion models have been successfully applied to various control tasks (Janner et al. 2022; Urain
ct al. 2023 Ajay et al. 2023 Chi et al. 2023 Liang et al. 2023). In particular, Diffuser (Janner et al.
2022) employs an unconditional diffusion model to generate trajectories consisting of state-action pairs. The approach includes training a separate model that predicts the cumulative rewards of noisy trajectory samples, which then guides the reverse diffusion process towards high-return trajectory samples in the inference phase, analogous to classifier guided sampling (Dhariwal & Nichol 202T).
Building upon this, Decision Diffuser (Ajay al.] 2023) extends the capabilities of Diffuser by adopting a conditional diffusion model with reward or constraint guidance to effectively satisfy constraints, compose skills, and maximize return. Meanwhile, AdaptDiffuser (Liang et al.) (023) enhances generalization ability of the diffusion model to unseen tasks by selectively fine-tuning it with high-quality data, derived through the use of hand-designed reward functions and an inverse dynamics model. In contrast, in this work, we focus on evaluating the quality of individually generated samples and explore ways enhance planning performance by utilizing guidance derived from these evaluations.
Restoring Artifacts in Generative Models Recently, several studies have concentrated on in- vestigating the artifacts in Generative Adversarial Networks (GAN) model architectures for image generation tasks. GAN Dissection (Bau al., 2019) explores the internal mechanisms of GANs, focusing on the identification and removal of units that contribute to artifact production, leading to more realistic outputs. In a subsequent study, an external classifier is trained to identify regions of low visual fidelity in individual generations and to detect internal units associated with those regions (Tousi et al.) 202T). Alternatively, artifact correction through latent code manipulation based on binary linear classifier is proposed (Shen et al., (2020). Although these methods can assess the fidelity of individual samples, they still necessitate additional supervision, such as human annotation. To address this limitation, subsequent works explore unsupervised approaches for detecting and
correcting artifact generations by examining local activation (Jeong et al.] (022) and activation
frequency (Choi et al.) 2022). In contrast, our work primarily focuses on refining the generative process of diffusion probabilistic models to restore low-quality plans.
Conclusion
We have presented novel refining method that fixes infeasible transitions within the trajectory generated by the diffusion planner. This refining process is guided by a proposed metric, restoration gap, which quantifies the restorability of a given plan. Under specific regularity conditions, we prove that the restoration gap effectively identifies unreliable plans while ensuring a low eиror probability for both type I and type II errors. The experimental results, which include enhancement in quantitative planning performance and visualization of qualitative attribution maps, highlight the importance of the refinement method of the diffusion planner.
Limitations While the restoration gap guidance effectively enhances the feasibility of plans and consistently improves the planning performance of diffusion models, our method is limited in situations where an offline dataset provided. Training the diffusion model often requires transition data that uniformly covers the state-action space, the collection of which is a nontrivial and time- consuming task.
Future Work Our analysis of the effectiveness of the restoration gap is currently confined to a relatively simple task, Maze2D (see Figure where we explicitly define normal and artifact plans. The choice of Maze2D is motivated by its suitability for identifying violations of prior knowledge, such as feasible plans not passing through walls. However, as future work, would be worthwhile to explore the efficacy of restoration gap in more complex tasks, such the block stacking task.
10

---

Acknowledgements
This work was supported by the Industry Core Technology Development Project, 20005062, Devel- opment of Artificial Intelligence Robot Autonomous Navigation Technology for Agile Movement in Crowded Space, funded by the Ministry of Trade, Industry & Energy (MOTIE, Republic of Korea) and by Institute of Information & communications Technology Planning & Evaluation (IIPP) grant funded by the Korea government (MSIT) (No. 2022-0-00984, Development of Artificial Intelligence Technology for Personalized Plug-and-Pla Explanation and Verification of Explanation, No.201 2..20190--0075, Artificial Intelligence Graduate School Program (KAIST)).
References
Ajay, A., Du, Y., Gupta, A., Tenenbaum, J., Jaakkola, T., and Agrawal, P. conditional gener- ative modeling all you need for decision making? In International Conference on Learning Representations (ICLR), 2023.
Argenson, A. and Duloc-Arold, G. Model-based offline planning. International Conference on Learning Representations (ICLR), 2021.
Bau, D., Zhu, J.-Y.. Strobelt, H., Zhou, B., Tenenbaum, J. B., Freeman, W. T., and Torralba, A.
Gan dissection: Visualizing and understanding generative adversarial networks. In International Conference Learning Representations (ICLR), 2019.
Beechey, D., Smith, T. M., and Şimşek, Ö. Explaining reinforcement learning with shapley values. In International Conference on Machine Learning (ICML), 2023.
Bertoin, D., Zouitine, A., Zouitine, M., and Rachelson, E. Look where you look! saliency-guu guided q-networks for generalization in visual reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
Choi, H., Chang, W., and Choi, J. Can find neurons that cause unrealistic images in deep generative networks? In International Joint Conference on Artificial Intelligence (IJCAI), 2022.
Deisenroth, M. and Rasmussen, C. E. Pilco: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning (ICML), 2011.
Dhariwal, P. and Nichol, Diffusion models beat gans on image synthesis. In Advances Neural Information Processing Systems (icemmm(NNrlPPe (NeurIPS), 2021.
Du, Y. and Mordatch, I. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.0720 2020.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (ICML), 2019.
Garrett, C. R., Lozano-Pérez, T., and Kaclbling, L. P. Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. Proceedings of the International Conference on Automated Planning and Scheduling (ICAPS), volume 30, pp. 440 448, 2020.
Grathwohl, W., Wang, K.-C., Jacobsen, J.-H., Duvenaud, D., and Zemel, R. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In International Conference Machine Learning (ICML), 2020.

---

Haarnoja, T.. Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic. Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning (ICML), 2018.
Hafner, D., Lillicrap. T., Fischer, I,, Villegas, R., Ha, D., Lee, H.,, and Davidson, J. Learning latent dynamics for planning from pixels. International Conference on Machine Learning (ICML), 2019.
Han, J., Lee, K., Tong, A., and Choi, J. Confirmatory bayesian online change point detection in the covariance structure of gaussian processes. International Joint Conference on Artificial Intelligence (IJCAT), 2019.
Han, J., Choi, H., Choi, Y.. Kim, J., Ha, J.-W., and Choi, Rarity score: A new metric to evaluate the uncommonness of synthesized images. In International Conference on Learning Representations (ICLR), 2023.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Hyvärinen, and Dayan, P. Estimation of non-normll statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Janner, M., Li, Q., and Levine, S. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning (ICML), 2022.
Jeong, H., Han, J., and Choi, J. An unsupervised way to understand artifact generating internal units in generative neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Kaiser, t.., Babaeizadeh, M., Mitos, P., Osiński, B., Campbell, R. H., Czechowski, K., Erhan, D.,
Finn, C., Kozakowski, P., Levine, S., et al. Model-based reinforcement learning for atari. In International Conference on Learning Representations (ICLR), 2020.
Kenny, E. M., Tucker, M., and Shah, J. Towards interpretable deep reinforcement learning with human-friendly prototypes. In International Conference on Learning Representations (ICLR), 2023.
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, Morel: Model-based offline reinforce- ment learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Kim, S. and Choi, J. Explaining the decisions of deep policy networks for robotic manipulations. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2663-2669, 2021.
Kim, S., Lee, K., and Choi, J. Variational curriculum reinforcement learning for unsupervised discovery of skills. In International Conference on Machine Learning (ICML), 2023.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Kostrikov, I., Nair, A., and Levine. S. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations (ICLR), 2022.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
12

---

Kurutach, T., Clavera, I., Duan, Y., Tamar, A., and Abbeel, P. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations (ICLR), 2018.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assessing generative models. Advances Neural Information Processing Systems (NeurIPS), 2019.
Lambert, N., Wilcox, A., Zhang, H., Pister, K. S., and Calandra, R. Learning accurate long-term dynamics for model-based reinforcement learning. In 2021 IEEE Conference Decision and Control (CDC), pp. 2880-2887, 2021.
Laurent, B. and Massart, P. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics, pp. 1302-1338, 2000.
LeCun, Y., Chopra, S., Hadsell, R.,, Ranzato, M., and Huang, F. A tutorial energy baaey learning. Predicting structured data, 1()), 2006.
Lee, K., Kim, S.-A., Choi, J., and Lee, S.-W. Deep reinforcement learning in continuous action spaces: a case study in the game of simulated curling. In International Conference on Machine Learning (ICML), 2018.
Lee, K. Kim, S., and Choi, J. Adaptive and explainable deployment of navigation skills via hierarchical deep reinforcement learning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1673-1679, 2023.
Liang, Z., Mu, Y., Ding, M., Ni, F., Tomizuka, M., and Luo, P. Adaptdiffuser; Diffusion models as adaptive self-evolving planners. In International Conference Machine Learning (ICML), 2023.
Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations (ICLR), 2019.
Meng, C., He, Y., Song. Y.. Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and editing with stochastic differential equations. International Conference on Learning Representations (ICLR), 2022.
Naeem, M. F., Oh, J., Uh, Y., Choi, Y., and Yoo, J. Reliable fidelity and diversity metrics for generative models. In International Conference Machine Learning (ICML), 2020.
Nagisetty, V., Graves, L., Scott, J., and Ganesh, V. xai-gan: Enhancing generative adversarial networks via explainable ai systems. arXiv preprint arXiv:2002.10438, 2020.
Nijkamp, E., Hill, M., Zhu, S.-C., and Wu, Y. N. Learning non-convergent non non-persistent short- meme toward energy- basee model. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Rajeswaran, A., Mordatch, and Kumar, V. A game theoretic framework for model based reinforce- ment learning. In International Conference on Machine Learning (ICML), 2020.
Ross, S. and Bagnell, J. Agnostic system identification for model-based reinforcement learning. In International Conference on Machine Learning (ICML), 2012.
Sajjadi, M. S., Bachem, O., Lucic. M.. Bousquet, O., and Gelly, S. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Salimans, T., Goodfellow, I., Zaremba, W. Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference computer vision (ICCV), 2017.
13

---

Silver, D., Schrittwieser, J., Simonyan, K., Antonogiou, I., Huang, A., Guez, A., Hubert, T., Baker, Shen, Y., Gu, J., Tang, X., and Zhou, B. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Shrikumar, A., Greenside, P., and Kundaje, A. Learning important features through propagating activation differences. International Conference on Machine Learning (ICML), 2017.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam,, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, $29(7587):484-88, 2016.
L., Lai, M., Bolton, A., et al. Mastering the game go without human knowledge. nature, 550 (7676):354-3) 3399 2017.
Simonyan, K., Vedaldi, A., and Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint 1rXiv:1312.600:: 2013.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamies.. In International Conference on Machine Learning (ICML), 2015.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Song. Y., Sohl-Dickstein, J., Kingma, P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
Talvitie, Model regularization for stable sample rollouts. Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2014.
Tassa, Y., Erez, T., and Todorov, E. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 19906-4913 2012.
Tousi, A., Jeong, H., Han, J., Choi, H., and Choi, Automatic correction of internal units in generative neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Urain, J.. Funk, N., Peters, J., and Chalvatzaki, G. Seff)ddiffusionfiell:: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In 2023 IEFE International Conference on Robotics and Automation (ICRA), pp. 5923-5930, 2023.
Voelcker, C., Liao, V., Garg, A., and Farahmand, A.-m. Value gradient weighted model-based reinforcement learning. In International Conference Learning Representations (ICLR), 2022.
Williams, G., Aldrich, A., and Theodorou, E. Model predictive path integral control using covariance variable importance sampling. arXiv preprint arXiv: 1509.0 00 149, 2015.
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
14

---

Appendix A Proofs
Proposition 1. Given [0 and positive constant C, A assume that | (T,
C2 for all
T normal
RR, and [S t) (C+A)) for all Tartifacts
Rd. If
2vd 2 d+ 2
log 8 2 2og
(17)
then setting
2 oi Caz+ + vd+ + yd d+ 2/ d. log 8 2log
(18)
guarantees both type I and type II errors most 26.
Proof. We prove the proposition by employing techniques similar to those used in Theorem 3.3 from (Han al. 2019) and Proposition from (Meng al.) 2022). To guarantee that both Type and Type Il errors are at most 26, we first derive thresholds by and b77 to control Type and Type II errors at most 8, respectively.
Controlling Type Error
For the type error, we aim to bound the probability:
P'(restoration gap (T) 2 by [Ho,),
(23)
where by represents the acceptance threshold for the alternative hypothesis. The restoration g (T) can be written as
restoration (T)
(24)
Ex
T --rsttoree,o (petttr; (perturb;(T))
(25)
(r 038 1-² d(oř]
dio
Reg
so(T,
di +
dw)
(26)
de
dt
2
O7c + /.² [[
- so(T.t) () dt V
d(o?]
[[o2]
Ee
dw
(27)
dt
dt
S Re, [11e:11]] 1 dt
d
d
8o SO ()
di +
dw
(28)
dt
1.-0
d(o
d
Soy [1|e|1e]+
So
dt
dw
(29)
dt
dt
f.
d
Vvdo; + Cof +
dw
(30)
dt
where the last inequality comes from Jensen's inequality, E., [llealaa] S v Eeg Il i
vd, and
considering the assumption over So(T,t) t) under the null hypothesis. As shown (Meng et al. 2022 the last term corresponds to the La norm of random variable arising from Wiener process at time I = 0, where its marginal distribution is given by e ~ AO0 I). Dividing the squared L2 norm of by oi results in a x2 disttribuuionn with d degrees of freedom. According to Lemma from (Laurent
Massart 2000), obtain the following one-sided tail bound:
P([C 2 d 2/ d log 2 2oo 6) I exp(log 8) = 6.
(31)
Then, have,
P p(1cl2/07 v d+ 2/ d. log ē 2 2 8 S.
(32)
15

---

Therefore, under null hypothesis, with probability of at least (1 6), we have that:
restoration g S by - (Cor + Vd+ d 2/ dd log d 2 log 8
(33)
which guarantees bounded type error at most 8. Controlling Type II Error
For the type II error, we aim to bound the probability:
IP(restoration g S byr HH),
(34)
where رдд represents the acceptance threshold for the alternative hypothesis. The restoration ( can be written
restoration gapi,e(T)
(35)
Ex
restore (perturb (T))
(36)
(r + 1340 {-'
dlož] so (o(T,
Exg
dt + V
(37)
dt
oic + 1("
alo?]
d[ož]] dw
Ex
4p so( 0s
(38)
dt
Bg [llc|l12] + S
dlož)
/|oo
SO 1))
dt
dw)
(39)
dt
dt
d[o
d
2 OyLE, [lleella] +
So
t)
dt
dw
(40)
2
d
2 + A)o
(41)
dt
where we follow similar procedure as in the proof for controlling type error, except deriving the lower bound by employing the triangle inequality. Therefore, under alternative hypothesis, with probability least 6), we have that:
restoration g 2 b = of ((++))o;- Vd - d+22-dl log 2 log
(42)
which guarantee a bounded type II error at most 6.
Combining Type I and Type II Thresholds
Following Theorem 3.3 in (Han et al. 2019), when by < dıs, selecting by the threshold ensures that both type and type Il errors are at most 25. Therefore, under
2vd 2 2/-d logō 2log 6
(43)
Oi
setting
b z oi Caz+ + Va v d+ 2/ -d d. log 2 2oo
(44)
guarantees both type and type errors at most 28, which completes the proof.
Appendix B Additional Experiments
B.1 Sensitivity Analysis to Input Attribution Methods
16

---

Table 4: Performance of RGG+ in Maze2D depending on the choice of attribution maps which suggests the robustness of the attri- bution map regularizer.
We investigate how sensitive the proposed method is depending on the choice of imput attribution methods. As described in Section 4.2 RGG+ utilizes the input at- tribution method as a regularizer to prevent adversarial restoration gap guidance. We apply the input attribu- tion method to the gap predictor, where it quantifies the impact of each transition in the generated plan on the prediction the restoration gap. To validate the robust- ness of RGG+ to different input attribution methods, we compare its performance while employing three attribution methods: Grad-CAM (Selvaraju et al., 2017). Sailency (Simonyan et al. 2013).
Maze2D Large
Multi2D Large
Grad-Cam Saliency
143.9 1.50
150.9 1.25
142.7
1.56
150.1 1.29
DeepLIFT
143.8 1.48
150.9 = 1.26
and LIT (Shrikumar al. 2017). The experimental results for Maze2D tasks are presented in Table RGG+ exhibits comparable performances across the three attribution methods, which implies that proposed method is robust in terms of the choice of input attribution methods.
B.2 Sensitivity Analysis to Perturbation Magnitude
To further investigate the effect of choosing dif- ferent values on the restoration gap, we com- pare the performance of plans which are chosen
Planning Performance
up to top-K% from the restoration gap when
150
Negatàve Realiseu Score
setting f to 0.3, 0.5, and 0.7, in addition to the initial setting of 0.9. The results of this abla-
Rer ci: pert
Harity
Hestoration Gap / 0.3
tion study are illustrated in Figure The results clearly show that the restoration gap remains sig- nificantly correlated with planning performance, even with these varied f values, when consider- ing the top 10% of results. Specifically, while the correlation strongest with set to 0.9, even with at 0.3, 0.5, or 0.7, the restoration gap still demonstrates stronger correlation with plan- ning performance compared the rarity score or the negative realism score. This indicates the robustness of the restoration gap metric for assessing the quality of generated plans. Moreover, these results support our Proposition T that larger is an effective choice when using the restoration gap.
Hestoration Gap 8
Restoration Gap i 0.7
Restoration Gap
0
20
40
60 80
100
Top-K %
Figure 6: An ablation study varying i to under- stand the impact of the magnitude the applied perturbation.
B.3 Comparison to Other Metrics
Table 5: Performance Comparison with rarity score and negative realism score.
Environment
Rarity
Negative Realism
RGG
RGG+
Maze2D
Large
126.9 + 2.1
128.9 1.6
135.4 + 1.7
143.9 + 1.5
Multi2D
Large
143.4 + 1.7
143.3 + 1.5
148.3 = 1.4
150.9 + 1.3
We investigate how effectively the restoration gap evaluates the quality generated samples by comparing the histogram with various metrics in Figure and comparing the planning performance Figure To further explore how significant a guidance signal provided by the restoration gap is for refining diffusion planner, metrics including the rarity score, the negative realism score, and the restoration gap. We conduct additional experiments in the Maze2D Large and Multi2D Large environments, the results of which are presented in Table []] Consistent with the observations in the previous experiments, Table 5 clearly demonstrates that the restoration gap is a useful metric for control tasks. Unlike other metries requiring expert data for training, our restoration gap works without such constraints, making it even more practical.
compare the planning performance of plans guided by various
B.4 Comparison to Discriminator Guidance
In our work, we have demonstrated the effectiveness of utilizing the restoration gap prediction model to enhance the performance of Diffuser. However, there could be an alternative approach that is
17

---

Table Performance Comparison with discriminator guidance (DG).
Environment
Diffuser
DG
RGG
RGG+
Maze2D
Large
123.5 2.0
127.0 1.9
135.4 + 1.7
143.9 + 1.5
Multi2D
Large
141.2 + 1.6
143.6 + 1.6
148.3 + 1.4
150.9 + 1.3
Artifacts Normal
Artifacts
Normal
Density
10
15
2000
4000
6000
8000
10000
12000
14000
Negative D-value
Restoration Gap (ours)
Figure 7:
The distribution differences between Artifacts and Normal plans illustrated through the density of the negative D-value and restoration gap.
to leverage the output of the discriminator which distinguishes whether the given plan is real or generated; we refer to this discriminator output as the "D-value". It is plausible as the discriminator inherently distinguishes between real and generated trajectories.
To evaluate the ability of the discriminator accurately identify infeasible plans, we define a subset of plans generated by Diffuser (Janner et al.) (022) "artifact plans". These artifact plans consist of transitions that include passing through walls, an impossible action for the agent to follow. We then compare the distribution of the restoration gap for both the normal and artifact plans.
As illustrated Figure 7 our comparison reveals that the D-value fails distinguish between artifacts and normal groups as effectively as the restoration gap. A noteworthy observation is that the discriminator, while adept at identifying infeasible transitions within generated trajectories, tends to focus more on local transitions rather than the overall structure. This local concentration results in it being less useful than the restoration gap when comes recognizing infeasible plans.
To further illustrate the practical implications of these findings, we conduct an additional performance comparison experiment. Here, the discriminator guidance (DG) is used to refine the Diffuser, and the resulting performances are compared with those of RGG and RGG+ (see Table 6). In our comparative experiments involving Maze2D Large and Multi2D Large environments, it apparent that the restoration gap guidance methods significantly outperform both the original Diffuser and the discriminator guidance (DG) method. While DG offers slight improvement over Diffuser, is unable to match the performance enhancements provided by our proposed methods.
These results underscore the effectiveness of the restoration gap reliable metric for improving tra- jectory generation by diffusion planners. This superiority holds even when compared to discriminator guidance, which directly capitalizes the discriminator's capability to distinguish between real and generated trajectories. Consequently, is evident that the restoration gap provides a more efficient strategy for refining diffusion planners.
B.5 Visualization of Plans with Low and High Restoration Gap
To understand how restoration gap values relate to the qualities of generated plans and to validate the efficacy of the restoration gap in detecting infeasible plans, we present additional qualitative results comparing visualized plans with low and high restoration gaps.
In the Maze2D-Large environment, as depicted Figure & plans with low restoration gap values exhibit smoother and more coherent trajectories. In contrast, plans with high restoration gap values often include physically infeasible transitions, such as passing through walls or abrupt changes in direction.
18

---

In the HalfCheetah environment, as demonstrated in Figure 9 the instability of movement, particularly during landing, is more apparent plans with high restoration gap. On the other hand, plans with low restoration gap values allow for stable landings, enabling the cheetah to move farther.
In the Unconditional Block Stacking environment, illustrated in Figure TO plans with high restoration gap exhibit error-prone transitions, For example, in such plans, the robotic arm suddenly teleports from the initial joint position, magically grasps block stacked under other blocks, or unexpectedly changes the grasped block. Moreover, these plans violate physical constraints, such as having the block in the same position as the robotic arm. In comparison, plans with low restoration gap values comply with the physical constraints.
This disparity between plans with low and high restoration gap values illustrates the effectiveness of our restoration gap metric distinguishing between reliable and unreliable plans across various environments.
High Restoration Gap
Low Restoration Gap
Figure 8: Visual comparison of plans of low and high restoration gap values generated by Diffuser in the Maze2D-Large environment.
High Restoration Gap
Low Restoration Gap
Figure 9: Visual comparison of plans of low and high restoration gap values generated by Diffuser in the HalfCheetah environment.
19

---

High Restoration Gap
Low Restoration Gap
planning borizon
planning horizon
7. n.n.
Figure 10: Visual comparison of plans of low and high restoration gap values generated by Diffuser in the Unconditional Block Stacking environment.
Appendix C Experimental Setup Details
C.1 Environments
Maze2D Maze2D environments (Fu et al. 2020) involve navigation task where an agent needs to plan for long-horize to navigate toward distant target goal location. A reward is not provided except when the agent successfully reaches the target goal where it only gets reward of 1. Maze2D environments consist of three distinct maze layouts: "U-Maze", "Medium", and "Large" each of which offers different levels of difficulty. In Maze2D environments, there are two tasks: a single-task, where the goal location is fixed, and multi-task, which refer Multi2D, where the goal location randomly selected at the beginning of every episode. Details about Maze2D environments summarized in Table 7
Table 7: Environment details for Maze2D experiments.
Maze2D-Large
Maze2D-Medium
Maze2D-UMaze
State space S Action space A
R
R
E R4
IR2
IR2
E R²
Goal space G
E I
E IR2
< R²2
Episode length
800
600
300
Gym-MuJoCo locomotion tasks (uu al. 2020) are widely used benchmarks for evaluating algorithms on heterogeneous data with varying quality. The "Medium" dataset is generated
Locomotion
20

---

by collecting 1M samples from an SAC agent (Haarnoja et al. 2018) trained to approximately one-third of the performance level compared to an expert. The "Medium-Replay" dataset includes all
samples acquired during training until a "Medium" level of performance is achieved. The "Medium- Expert" dataset is composed of an equal mixture of expert demonstrations and sub-optimal data. Environmental details for Locomotion experiments are summarized in Table
Table 8: Environment details for Locomotion experiments.
Hopper-* * R
Walker2d-
Halfcheetah-
State space Action space AA Episode length
E RR7 R
IR:17
E RR3
E R
1000
1000
1000
Block Stacking The Block Stacking suite a benchmark used evaluate the model performance for a large state space, which employs a Kuka iiwa robotic arm (Janner et al. (022). The offline demonstration data required to train a policy model or a diffusion planning model is obtained through the application of PDDLStream (Garrett et al. (020).
The objective of the unconditional stacking task is to construct tower of blocks with the maximum possible height. In this task, the agent observes the state including the joint position of the robot, as well as the position and rotation of each block, and then commands the robot's desired joint position while performing the grasping action to pick up the blocks. In the conditional stacking task, where the objective is stack blocks in a specified order, the agent observes the same state as in the unconditional stacking task, but it additionally observes the index of the block which indicates the order in which the blocks should be stacked.
We employ the same diffusion model for both tasks, but in the conditional stacking task, we addition- ally utilize value function to guide the diffusion planner in stacking blocks according specified conditions. Details about Block Stacking environments are summarized in Table 9
Table 9: Environment details for Block Stacking experiments.
Unconditional Stacking
Conditional Stacking
State space S Action space A Episode length
IR 39
IR:
IR 11
E: IR 11
384
384
C.2 Other Metries for the Assessment of Individually Generated Samples
Kynkäänniemi et al. (2019) present the notions of improved precision and recall to the study of generative models. Precision is determined by examining if each generated sample falls within the estimated manifold of real samples. Symmetrically, recall is computed by checking if each real sample resides within the estimated manifold of generated samples. Real and generated sample feature vectors are represented o, and Φg. respectively, and the corresponding sets of these feature vectors are denoted by & and b To compute improved precision and recall, the manifolds real and generated samples estimated using the sets of k-NN hyperspheres for each sample:
manifold, (Ø)
U (x' Φ), Ba + {ób || - Φ112 I 1| NN&(0' 1) (45) 6'
where NN, ④ returns the kth nearest feature vector of ø' from the set Φ. (& 1) is the K-NN hyperspheres with the radius of o - NN&(0', $ 11
Although the improved precision metric provides a way to evaluate the quality of a population of generated samples, it yields only a binary result for an individual sample, making unsuitable for ranking individual samples by their quality. In contrast, realism score (Kynkäänniemi et al. 2019) and rarity score (Han et al., (023) offer continuous extension of improved precision and recall, enabling the assessment of individually generated sample quality.
21

---

Realism Score The realism score quantifies the maximum inverse relative distance of a generated sample within a k-NN hypersphere originating from real data.
11 NN (
1,)122
realism score( (dy: ..) max Pr
(46)
|0g
Ør112
A high realism score is achieved when the relative distance between generated sample and a real sample small, compared to the radius the real sample's k-NN hypersphere.
The rarity score measures the radius of the smallest nearest-neighbor sphere that contains the generated sample.
Rarity Score
rarity score(dg: 2,))
min
11
NN&(dr, Rt)1le-
(47)
r.e. r. Bx
This is grounded in the hypothesis that normal samples will be closely grouped, whereas unique and rare samples will be dispersed in the feature space.
In an attempt to compare the restoration gap against negative realism scores and rarity scores, we apply the parameters suggested in (Han et al. 22023,, making use of k = 3 and 30,000 real samples to approximate the real manifold and calculate the corresponding scores: negative realism and rarity,
Appendix D Implementation Details and Hyperparameters
D.1 Implementation of Gap Predictor
We employ temporal U-Net architecture, with repeated convolutional residual blocks, for param- eterizing G3 as introduced in Diffuser (Janner et al. (022). By using the pre-trained down blocks from Diffuser's diffussonn model as our feature extraction module and keeping fixed during training, we can achieve enhanced performance and reduce training costs. The hyperparameters for training the gap predictor are summarized Table 10
Table 10: Hyperparameters used for training the gap predictor. Values that within brackets separately tuned through a grid search.
Hyperparameter
Maze2D
Locomotion
Block Stacking
# synthetic data Observation normalization
500000
500000
500000 Yes (0.00002, 0.0002, 0.001) {32, 64, 128, 256) 2000000
Yes
Yes
Gap predictor learning rate
0.0002
0.0002
Gap predictor batch size Gap predictor train steps
32
32
2000000
2000000
for perturbation magnitude
0.9
0.9
0.9
D.2 Implementation of Restoration Gap Guidance
In this section, we document hyperparameters employed for restoration gap guidance. We adopt the hyperparameters from Diffuser (Janner et al. [2022] for determining the planning horizon and diffusion steps. Furthermore, the values of a, B, and À are determined through grid search. The hyperparameters utilized for the Maze2D experiments are presented in Table IT] while those for the Locomotion experiments are summarized in Table 12 and 14 The hyperparameters for the Block Stacking experiments are provided in Table 15
The planning performance exhibits a clear trend based on the choices of a, 3, and À parameters. This allows us to perform a grid search using the relatively minimal number of evaluation episodes. As depicted in Figure 3 the planning performance initially increases with rising values of À but begins to decline when the values become excessively large. Specifically, we conduct 10, 15, and 10 evaluation episodes for the Maze2D, Locomotion, and Block Stacking experiments, respectively.
22

---

Table 11: Specific hyperparameters for Maze2D experiments. Values that within brackets
separately tuned through a grid search.
Hyperparameter
Large
Medium
UMaze
Planning horizon
384
256
128
Diffusion steps
256
256
64
# samples for MC estimate of restoration gap
10
10
10
for scaling the overall guidance
(0.05, 0.1)
(0.05, 0.1)
(0.05, 0.1)
3 for scaling restoration gap guidance
1.0
1.0
1.0
X for scaling attribution map regularization
{0.1, 1.0, 3.0)
(0.1, 1.0, 3.0)
(0.1, 1.0, 3.0)
Table 12: Specific hyperparameters for HalfCheetah experiments. separately tuned through a grid search.
Values that are within brackets are
Hyperparameter
Med-Expert
Medium
Med-Replay
Planning horizon
32
32
Diffusion steps
20
20
20
samples for MC estimate of restoration gap
64
64
64
a for scaling the overall guidance for scaling restoration gap guidance for scaling attribution map regularization
{0.01, 0.1
{0.01. 0.1]
{0.01, 0.1]
(0.1, 1.0, 10.0)
(0.1, 1.0, 10.0)
(0.1, 1.0, 10.0)
(0.001, 0.01, 0.1, 1.0)
10.001, 0.01, 0.1, 1.0)
10.001, 0.01, 0.1, 1.0)
Table 13: Specific hyperparameters for Hopper experiments. separately tuned through a grid search.
Values that are within brackets
Hyperparameter
Med-Expert
Medium
Med-Replay
Planning horizon
32
32
32
Diffusion steps
20
20
20
samples for MC estimate of restoration gap
64
64
64
a for scaling the overall guidance
10.01, 0.1)
[0.01, 0.1]
[0.01, 0.1]
scaling restoration gap guidance
(0.1. 1.0, 10.0)
10.1, 10.0]
10.1, 1.0, 10.0)
X for scaling attribution map regularization
(0,001, 0.01, 0.1, 10]
10.001, 0.01, 0.1. 1.0]
10.001, 0.01, 0.1. 1.0)
Table 14: Specific hyperparameters for Walker2d experiments. Values that are within brackets are
separately tuned through a grid search.
Hy Hyperparameter
Med-Expert
Medium
Med-Replay
Planning horizon
32
32
32 20
Diffusion steps
20
20
samples for MC estimate of restoration gap
64
64
64
a for scaling the overall guidance
[0.01, 0.1|
(0.01, 0.1|
[0.01, 0.1|
scaling restoration gap guidance
[0.1. 1.0, 10.0)
10.1. 1.0. 10.0)
[0.1. 1.0. 10.0)
X for scaling attribution map regularization
(0.001, 0.01, 0.1, 1.0)
(0.001, 0.01, 0.1. 1.0)
(0.001, 0.01, 0.1, 1.0)
Table 15:
Specific hyperparameters for Block Stacking experiments. Values that are within brackets are separately tuned through grid search.
Hyperparameter
Unconditional Stacking
Conditional Stacking
Planning horizon
128
128 1000
Diffusion steps
1000
# samples MC estimate of restoration gap
5
a for guide scale
(0.01, 0.02, 0.05, 0.2, 0.5)
[0.1, 0.3, 0.5, 0.7] (0.01, 0.02, 0.04, 0.07]
B for scaling restoration gap guidance X for scaling attribution map regularization
1.0
(0.1. 0.3, 0.5, 1.0, 3.0)
(0.001. 0.003. 0.005. 0.008)
D.3 Implementation of Attribution Map Regularizer
For the Maze2D experiments and the Block Stacking experiments, we adopt Grad-CAM (Selvaraju et al. 2017). while we employ Deep (Shrikumar et al.] (017) for the Locomotion experiments, as the attribution method. The rationale for the choice of each method that both Grad-CAM and DeepLIFT offer simplicity and efficiency in computation. Specifically, apply the DeepLIFT for
23

---

the Locomotion experiments due to their relatively smaller planning horizon compared to the other tasks. This is because Grad-CAM compresses saliency information into a single value. However, it is worth noting that our proposed method has shown robustness across different input attribution methods, as demonstrated in Table
D.4 The Amount of Computation
Our proposed method, refining diffusion planner, requires training the gap predictor which estimates the restoration gap. For this purpose, we generate synthetic data of 500,000 plans generated by Diffuser. The generation process takes approximately 30 to 50 hours, depending the situation, on single NVIDIA Quadro 8000 GPU. The training time of the gap predictor the same GPU can range from 5 to 8 hours.
Appendix E Baseline Performance Sources
E.1
Maze2D Tasks
The scores for CQL are taken from Table in Fu et al. (2020). The scores for MPPI and IQL are taken from Table in Janner et al. (2022).
E,2 Locomotion Tasks
The scores for BC, CQL, and IQL are found in Table 1 of Kostrikov et al. (2022). while DT scores are taken from Table 2 in Chen et al. (202T). TT from Table 1 in Janner al. (202T), MOPO from Table in Yu et al. (2020), MORel. from Table 2 in Kidambi et al. (2020), MBOP from Table I in Argenson & Dulac-Arnold (021), and Diffuser from Table in Janner al. (2022).
E.3 Block Stacking Tasks
The scores corresponding to BCQ and CQL are obtained from Table 3 of Janner et al. (2022). To obtain the scores for Diffuser, the official implementation and model provided by the authors are used, which can be found at httph:://// //g thub. cooom/jannemmmafffa.. diffuser
24


Reviews:
[{"review_id": "ADcu3W720g", "reviewer": "Reviewer_iw4y", "summary": "The paper proposes to refine the predictions of a diffusion-based planner by identifying unfeasible plans prior to execution. The newly introduced \"restoration gap\" measures the reconstruction error for a given sample under some noisy perturbation, and it is argued that this gap is a reasonable proxy for feasibility (i.e. if the restoration gap is high for a sample (plan), then that plan is probably not feasible). The authors show that the proposed metric can identify \"artifacts\" with bounded error probabilities. The proposed approach trains a separate model known as the gap predictor, which simply predicts the restoration gap for a given sample. Then this gap predictor, which is differentiable, can be used to guide the diffusion sampling process away from unfeasible plans. Additionally, since errors in the gap predictor could lead to model exploitation, the authors propose to use attribution maps to perform regularization. Experiments on various offline control tasks (Maze2D, Mujoco locomotion, Block stacking) show that the proposed approaches (with/without regularization) outperform several strong offline RL baselines. The authors also show how the attribution maps can be used to identify specific parts of samples which contribute the most to the restoration gap, which is useful for identifying problematic transitions for instance.", "questions": "Is it possible to adapt the metrics compared to in 5.1 to downstream control? I think it would be helpful if the authors could show that the proposed restoration gap metric is actually better than prior metrics on relevant control benchmarks. I'm not super familiar with these other metrics so I don't know how feasible that is.\n\nI would suggest including more formal definitions for \"faithful\" and \"infeasible\" plans. I have no idea what a faithful plan is. I think I know what an infeasible plan is, but given the aforementioned confusing around the motivation for the paper, it's possible I have a different definition in mind.\n\nAlso for some reason I am unable to open the supplementary materials zip file, so nothing in there affected my review.\n\n===\nI have read the rebuttal and bumped my score up. I am in favor of acceptance.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed work seems novel and the experimental results show that the proposed work is state-of-the-art.\n\nFrom my readthrough, I didn't identify any glaring technical flaws and the approach itself seems sound.\n\nSection 5.3 is very solid and highlights a pretty interesting application of the proposed approach that I haven't seen before.", "weaknesses": "I disagree a bit on the high-level motivation for this work. I don't think there is anything about the restoration gap that specifically targets \"unfeasible\" plans (by \"unfeasible\" I mean plans that we cannot execute due to certain constraints, which is different from plans that are just bad under some reward function). For instance, I would expect that highly OOD plans have very high restoration gaps even if they are technically feasible to execute, because our diffusion model will struggle to reconstruct plans that are not in the training distribution. My suspicion is that this approach is equivalent to performing some kind of low temperature sampling to ensure predicted samples are high confidence. If that is the case, then I think the paper needs some rewriting because the approach really is not related to feasibility I think. Would be curious to hear if the authors have a counterpoint to this, but otherwise I think this should be addressed.\n\nSince I do think this is fairly major concern with the writing, my score is a bit lower, but I still think the core technical contribution of this paper is reasonable so I'm happy to bump my score if the paper is revised, or it turns out that I am incorrect in my suspicions.\n\nThe experimental results aren't terribly impressive, in most settings the proposed approach is barely better than Diffuser.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: I disagree a bit on the high-level motivation for this work. I don’t think there is anything about the restoration gap that specifically targets “unfeasible” plans.**\n\n**A:** In low-data regime, we acknowledge the scenario where even feasible plans can appear highly OOD, leading to a substantial restoration gap. However, our high-level motivation mainly targets scenarios where there is a sufficient offline dataset comprising feasible trajectories. The property of **temporal compositionality** in Diffusion Planners, as described by Janner et al. (2022), plays a significant role in this context.\n\nWith temporal compositionality, Diffusion Planners can generate globally coherent trajectories by continuously enhancing local consistency. This allows them to compose out-of-distribution trajectories by stitching together any feasible plan subsequences.\n\nGiven this property, we argue that most infeasible plans, those that cannot be executed due to system constraints, would likely be OOD, while feasible plans would generally remain within the distribution. So, the restoration gap, in our view, does hold relevance in identifying unfeasible plans. We appreciate your thoughtful feedback and we will make sure to emphasize this context in our revision. \n\n**Q: I would suggest including more formal definitions for “faithful” and “infeasible” plans.**\n\n**A:** In our context, “faithful” refers to plans that are consistent with known constraints and can be executed. Conversely, “infeasible” plans are those that cannot be executed due to system constraints. We will include formal definitions on the revised version for clarity.\n\n**Q: Also for some reason I am unable to open the supplementary materials zip file, so nothing in there affected my review.**\n\n**A:** We sincerely apologize for the inconvenience you experienced in accessing the supplementary materials. We have tested the accessibility of the zip file across various OS, including Windows, Ubuntu, and MacOS where it worked fine for all systems. It appears that the issue might be specific to your system configuration. If you provide additional details regarding the problem you encountered, we’ll address the problem.\n\nIn the meantime, with the approval of the Area Chair, we have provided an anonymized Dropbox link containing the exact files uploaded on OpenReview. You can access them directly using the following link: [Supplementary Materials](https://www.dropbox.com/scl/fo/2dwyps3kw02eveyyog9ue/h?rlkey=dv07r9vp5eccj9s172egn76j6&dl=0). In our final submission, we’ll ensure the materials are accessible.\n\n**Q: The experimental results aren’t terribly impressive.**\n\n**A:** Please refer to the global response provided above.\n\n**Q: Is it possible to adapt the metrics compared to in 5.1 to downstream control?**\n\n**A:** Please refer to the global response provided above.\n\n**References**\n\n- Janner et al., Planning with Diffusion for Flexible Behavior Synthesis. In *ICML*, 2022."}, {"review_id": "t46uwfPxsn", "reviewer": "Reviewer_N3Wq", "summary": "This paper introduces a novel approach to refining unreliable plans generated by diffusion models in long-horizon, sparse-reward tasks. While diffusion-based planning has shown promise, the generative nature of diffusion models can result in infeasible plans, limiting their usefulness in safety-critical applications. The proposed approach utilizes a metric called restoration gap to evaluate plan quality and provides refining guidance to error-prone plans. Additionally, an attribution map regularizer is presented to prevent adversarial refining guidance. The effectiveness of the approach is demonstrated on three benchmarks, and the explainability of the generated plans is highlighted through the attribution maps.", "questions": "1）How does the proposed approach address the potential issue of rare but valid trajectories being considered abnormal due to the distribution of the expert dataset? Have any measures been taken to mitigate this limitation?\n2）Could you provide more insight into how the coefficients for the opportunity cost function J's guide and the opportunity restore guide are balanced in the proposed approach? Has any experimentation been conducted to determine the optimal balance between these coefficients?", "limitations": "the article should address the issues related to the accuracy of the restoration gap metric in few-shot case and conduct experiments to explore the balance between coefficients in the opportunity cost function and the opportunity restore guide.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Detailed and comprehensive experiments: The article provides a thorough evaluation of the proposed approach on three different benchmarks in offline control settings that require long-horizon planning. This comprehensive experimentation strengthens the validity of the findings.\n2. Novel idea and solution: The article introduces a new metric, restoration gap, and a corresponding refining guidance approach to address the limitations of diffusion-based planning. This novel approach contributes to the field and expands the understanding of plan refinement in such tasks.\n3. Addressing the issue of error-prone plans: The article successfully tackles the challenge of refining unreliable plans generated by diffusion models. By providing refining guidance based on the restoration gap metric, the proposed approach improves the feasibility and quality of the plans.", "weaknesses": "1） Accuracy of restoration gap metric: This paper uses the difference between the generated trajectory and the original input trajectory to evaluate plan quality. However, this approach may be inaccurate for small-sample cases where the distribution of the expert dataset leads to rare but valid trajectories being considered abnormal. Further exploration is needed to address this limitation.\n2） Lack of experimentation on balancing coefficients: The article does not conduct an ablation study to examine the balance between the opportunity cost function J's guide and the opportunity restore guide. It would be beneficial to investigate how to strike a balance between these coefficients to achieve optimal plan refinement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: How does the proposed approach address the potential issue of rare but valid trajectories being considered abnormal due to the distribution of the expert dataset?**\n\n**A:** We appreciate the reviewer’s insightful comment regarding the accuracy of the restoration gap. Indeed, the inaccuracy comes from the two cases: 1) when the initially generated trajectory (the original input trajectory) is valid but considered invalid, and 2) when the initially generated trajectory is invalid but incorrectly considered valid. We acknowledge that the reviewer’s specific concern pertains to the first case, where valid trajectories may be misclassified.\n\nTo address this issue, we employ a strategy that involves sufficiently corrupting the given trajectory (i.e., the originally generated trajectory) during the restoration process by setting a large enough value for $\\hat{t}$, where $\\hat{t} \\in (0, 1]$ is a continuous time variable for indexing diffusion timestep. As the expert dataset does not contain invalid trajectories at all, valid trajectories inherently possess larger density compared to invalid ones, even though their density might be relatively small. Therefore, by perturbing the given trajectory significantly, we ensure that the restored trajectory from an invalid trajectory diverges more significantly in terms of the L2 distance compared to the restored trajectory from a valid trajectory, through the extended denoising procedure.\n\nWe have conducted a theoretical analysis, outlined in Proposition 1, which supports this approach. The first case error that the reviewer highlights corresponds to the type I error (false positive). According to Proposition 1, the error probability of restoration gap for type I error is bounded at most $\\delta$ through thresholding with a threshold value of $b_I = \\sigma_{\\hat{t}}\\left(C\\sigma_{\\hat{t}} + \\sqrt{d} + \\sqrt{d + 2 \\sqrt{-d \\cdot \\log \\delta} - 2 \\log \\delta)}\\right)$, under specific regularity conditions. To tightly bound the type I error, it is crucial to have a large threshold value $b_I$, requiring a sufficiently large $\\sigma_{\\hat{t}}$, which, in turn, implies employing a large enough $\\hat{t}$. For further details, please refer to Appendix B.\n\nPlease also note that we conducted an ablation study on the sensitivity to perturbation magnitude $\\hat{t}$ in Appendix C.2 which stands for this theoretical analysis. Figure 6 in Appendix C.2 demonstrates that the performance of plans from the restoration gap when setting $\\hat{t}=0.9$ performs the best among $\\hat{t}=0.3, 0.5, 0.7, \\text{and } 0.9$.\n\nWe thank the reviewer for bringing up this issue, and we will explicitly state this aspect in the revised version.\n\n**Q: Lack of experimentation on balancing coefficients.**\n\n**A:** Please refer to the global response provided above."}, {"review_id": "wgIqfJW8iH", "reviewer": "Reviewer_pN8B", "summary": "The paper proposes a novel metric named restoration gap for evaluating trajectory quality generated by diffusion models. It further proposed restoration gap guidance to refine the diffusion generation. The restoration gap predictor also provides explainability of the model. Experiment results indicate improved performance of the proposed model.", "questions": "Can author give some results / explainations on how sensitive the performance of the model is to the additional hyperparameters introduced (beta, lambda in eq. 22) and how hard it is to tune them?", "limitations": "While the restoration gap is intuitive in the given examples (2D map with walls), it is unclear how the gap works in more complex environments or environments with fewer hard constraints.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The contribution of the paper is solid: The restoration gap gives an intuitive metric for measuring trajectory quality: plans with low quality are hard to restore after perturbation; Following this intuition, the author proposed to train a gap predictor and uses its gradient to guide the generation process. Experiment results give strong support for the claim of the paper.\n\nThe presentation of the paper is clear. The paper is well-structured with intuitive visualizations.\n\n ", "weaknesses": "To apply the restoration gap guidance, a hyperparameter is introduced to weight the signal from the gap predictor; and then to mitigate the issue from the gap prediction error, an attribution map regularization is added with another hyperparameter weight. It is not clear how sensitive the performance of the model is to these hyperparameters and how hard it is to tune them.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: While the restoration gap is intuitive in the given examples (2D map with walls), it is unclear how the gap works in more complex environments or environments with fewer hard constraints.**\n\n**A:** In more complex environments with fewer hard constraints, determining infeasibility might not be as straightforward as detecting a plan that involves going through a wall in the Maze2D environment. However, our approach remains valid as the concept of infeasibility can be extended to less constrained environments.\n\nIn the HalfCheetah environment, for instance, a plan could be seen as infeasible if it entails abrupt changes in the direction or speed of the cheetah, which would be physically improbable. Similarly, in the Unconditional Block Stacking environment, plans that involve the robotic arm suddenly teleporting or grasping a block stacked under other blocks could be considered infeasible.\n\nWe reported detailed visualizations and explanations of such instances in Appendix C.4. The plans with high restoration gaps often exhibit such physically improbable transitions in these environments, and our restoration gap measure correctly identifies these as unreliable.\n\n**Q: Can author give some results / explanations on how sensitive the performance of the model is to the additional hyperparameters introduced (beta, lambda in eq. 22) and how hard it is to tune them?**\n\n**A:** Please refer to the global response provided above."}, {"review_id": "GQJULBSmTN", "reviewer": "Reviewer_mq5w", "summary": "This paper investigates the potential of diffusion models in addressing long-horizon sparse reward tasks. While diffusion models are effective generative models, they are not inherently guaranteed to generate feasible plans. To overcome this limitation, the study proposes an approach to refine unreliable paths generated by diffusion models and provides guidance to enhance error-prone plans.\nThe evaluation of individual plans generated by diffusion models is done using a novel metric called the \"restoration gap\" which measures the restorability of a given path. The \"restoration gap\" metric effectively identifies plans with a low error probability, which is considered an indicator of plan feasibility. This metric is estimated using a function approximator called the \"gap predictor.\"\nWith the help of the gap predictor, a Restoration Gap Guidance (RGG) is defined, which refines diffusion planners and improves the low-quality plans generated by diffusion model.\nMoreover, the gap predictor can lead the agent in an undesirable direction due to estimation errors during the denoising process. Therefore, RGG is augmented with a regularizer to prevent plans from heading in the wrong direction, and this modified RGG is referred to as RGG+.\nThe proposed approach is evaluated on three different benchmarks. The experimental results demonstrate that the approach enhances planning performance compared to the benchmarks.\n\nOverall, the paper presents a novel method for refining unreliable plans generated by diffusion models, improving the planning performance in long-horizon tasks. The approach is demonstrated to be effective in various offline control settings and provides explainability through the use of attribution maps.\n", "questions": "What can you say about the trade-off in terms of additional complexity and thus probably increased learning time, induced by the proposed extension, vs. the performance improvement?\n\nI assume the lengths of the original trajectories and those generated by diffusion models are the same, right? If yes, are there additional steps to ensure this, or are they by default equal?\n\nMaze environments might sometimes be solved using different trajectories. For instance, in Figure 2 (last row and last column maze), the generated trajectory takes the shortest path. However, it is possible for the diffusion model to take the longest path if the maze environment has been previously solved using longer trajectory paths. If these observations are stored in the dataset and used to train the diffusion model, it may generate longer trajectories but the original trajectory might be the shortest one. Is this a possible situation? If yes, how does this affect the restoration gap metric and would such a case cause it to inaccurately assess the quality of the generated paths?\n\nWhat could be the reason(s) for the relatively smaller improvement on the Multi2D environment compared to a better improvement for the Maze2D scenario (Table 2)?\n\nHow is the performance score computed for Maze2D, Multi2D environments? E.g., the other score you shared for CQL seems to have been scored between 0 and 100 in the original paper (Fu et al., 2020). Your scores are above 100 though, how is this possible?\n", "limitations": "The authors briefly mentioned the limitations of the proposed approach in the appendix section. I believe the main text should include those limitation discussions. \n\nIn addition, if any of the questions / comments above relates to a limitation, please include them as well. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Novel Approach: The paper proposes a novel approach to refining unreliable plans generated by diffusion models. It introduces the concept of restoration gap as a metric to evaluate the quality of individual plans generated by the diffusion model.\n\nTheoretical Justification: The restoration gap metric is theoretically justified, and the paper provides an analysis of its properties. It demonstrates that the restoration gap can effectively detect artifacts (infeasible plans) with bounded error probabilities.\n\nImprovement in Planning Performance: The proposed approach, called Restoration Gap Guidance (RGG), is shown to enhance the planning performance of the diffusion model. By utilizing the restoration gap metric and a gap predictor, the RGG process guides the reduction of the restoration gap and improves the quality of the generated plans.\n\nExplainability: The paper highlights the explainability aspect of the approach. It presents the attribution maps of the gap predictor, which provide insights into error-prone transitions and enable a deeper understanding of the generated plans. Though, I believe this aspect needs to be evaluated further to prove its usefulness in different settings. \n\nEmpirical Evaluation: The proposed approach is evaluated on three different benchmarks in offline control settings, including Maze2D, Locomotion, and Block Stacking tasks. The experiments demonstrate the effectiveness of the approach in improving planning performance compared to other baseline methods.\n\nRegularization Technique: To mitigate the risk of adversarial artifacts, the paper introduces an attribution map regularization method. This regularization prevents the gap predictor from pushing plans in the wrong direction, resulting in improved refining guidance.\n", "weaknesses": "This could be a weakness or a limitation (or a misunderstanding). If it’s limitation please elaborate on it in the Limitations section. \nBasically, the proposed approach appears to assume that the diffusion model is well-trained and makes almost no mistakes. In the restoration gap part, the effectiveness of the \"restoration gap\" metric relies on the assumption that when a new trajectory is provided to the diffusion model to generate new plans, this trajectory data is assumed to have a similar distribution with the dataset which is used to train the diffusion model. In such cases, the restoration gap, based on L2 distance, might be applicable. However, if a new trajectory data which doesn't have a similar distribution is given to the diffusion model, the diffusion model might generate significantly different trajectories compared to the original trajectory. In this case, the restoration gap based on the L2 distance could potentially fail to provide accurate guidance. Can you comment on that?\n \nNot weaknesses but just some small remarks are provided below:\n\nI’d consider the third point in the contribution list as a result, and thus a verification of the proposed approach, not a contribution. \n\nLine 226-227: I wouldn’t say RGG improves the planning performance of Diffuser in 5 out of 6 tasks confidently. 2-3 out 6 seems like a better judgment of these results. \n\nLine 229: I think the claim “... our approach of refining Diffuser with RGG either matches or surpasses most of the offline RL baselines. Additionally, it significantly enhances the performance of Diffuser …” is too strong. Please be more precise and fair here. It would also help if you color-code the best- and second-best-achieving scores (surely while considering Diffuser, RGG and RGG+ as one algorithm, e.g., if RGG+ has the best score, do not consider RGG as the second-best if it has the highest score amongst others) in Table 1. That helps the reader put in perspective the performance of diffusion-based models to other RL approaches.\n\nLine 235: repetition of a reference - “... and MOPO (Yu et al., 2020)”\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: What if the diffusion model is not well-trained? Can you comment on the accuracy of the restoration gap in such case?**\n\n**A:** We acknowledge that inaccuracies might arise especially in a low-data regime if the diffusion model is not good at estimating score functions for certain trajectories. However, when provided with sufficient data, the temporal compositionality property of Diffuser (Janner et al., 2022) stands out. This ensures that even with novel trajectories differing in distribution, the model can still generate accurate plans, leading to accurate estimation of restoration gap.\n\nThis unique characteristic stems from the model’s ability to generate globally coherent trajectories by iteratively refining local consistency, allowing it to compose out-of-distribution trajectories from feasible subsequences. We will discuss this limitations especially in low-data regime, in our revised paper.\n\nPlease also refer to our response to reviewer N3Wq regarding the potential inaccuracies of the restoration gap and our approach to mitigate them.\n\n**Q: The claim in Line 229 is too strong. It would be also helpful if you color-code the best- and second-best-achieving scores.**\n\n**A:** We agree that our claim is strong, we will tone down in the revised version. Moreover, we acknowledge the reviewer’s suggestion and we will make sure to incorporate the color-coding accordingly in Section 5.2.\n\n**Q: Trade-off in terms of additional complexity vs. the performance improvement?**\n\n**A:** As described in Appendix E.4, refining the diffusion planner requires the gap predictor for restoration gap estimation, using 500,000 synthetic data generated by Diffuser over 30 to 50 hours on a single NVIDIA RTX Quadro 8000 GPU, specifically in Block Stacking experiments. Subsequent gap predictor training on the same GPU takes 5 to 8 hours, which, in turn requires a maximum of 58 hours.\n\nHowever, we would like to highlight that this additional complexity remains tolerable, considering the benefits it yields. Notably, training the gap predictor allows to improve the performance across tasks, while also bringing an additional advantage in terms of injecting explainability into diffusion planners.\n\n**Q: I assume the lengths of the original trajectories and those generated by diffusion models are the same, right? If yes, are there additional steps to ensure this?**\n\n**A:** Yes, your assumption is correct. The lengths of the original trajectories and those restored trajectories should be the same to compute the restoration gap. However, as they are both generated from the same length of Gaussian random noise through the denoising process, there is no necessity for us to manipulate to ensure their equality in length.\n\n**Q: How do sub-optimal trajectories stored in the dataset affect the restoration gap metric?**\n\nA: It is essential to distinguish between two orthogonal aspects of the generated trajectories: **optimality** and **reliability**, wherein the restoration gap metric primarily pertains to reliability rather optimality.\n\nIn general, optimality refers how efficient the trajectory is to solve a given task. For instance, in the context of Maze2D environments, the optimal trajectory refers the shortest path to reaches the goal position. As correctly pointed by the reviewer, the diffusion model may generate sub-optimal trajectories due to its learning from an offline dataset that contains sub-optimal trajectories. To enhance optimality, we can seek to maximize the cumulative reward along the trajectory. Notably, Diffuser employs return guidance during the denoising process to facilitate the maximization of cumulative rewards, represented as $\\nabla\\mathcal{J}$ in Equation 21.\n\nReliability, on the other hand, pertains to the feasibility of the planned trajectory for the agent to execute. This reliability aspect is related to how likely the generated trajectory is aligned with the training data distribution, assuming a sufficient amount of dataset. To evaluate this feasibility, we first perturb the initially generated trajectory through the forward process. Subsequently, we restore the perturbed trajectory through the denoising process by the diffusion model. Given that infeasible trajectories violating physical constraints are absent from the training dataset, it is highly probable for the diffusion model to fail to restore the exact trajectory to the initially generated one. Thus, we facilitate the enhancement of reliability through restoration gap guidance during the denoising process, represented as $\\nabla\\mathcal{G}$ in Equation 21.\n\n**Q: What could be the reason(s) for the relatively smaller improvement on the Multi2D compared to the Maze2D?**\n\n**A:** Maze2D involves tasks that require navigating to a fixed goal located in a corner of the map. This setup, as compared to Multi2D, often leads to longer distances more detours around walls from the start to the goal, presenting more opportunities for generating infeasible plans, thereby accounting for the greater improvement observed in Maze2D.\n\n**Q: How is the score computed for Maze2D and Multi2D?** \n\n**A:** A reward of 0 is given for each timestep, and a reward of 1 is received once the goal is reached. Therefore, the quicker the goal is reached, the higher the sum of rewards will be. Furthermore, in accordance with Janner et al. (2022), we normalize the score as follows:\n\n$$\n\\text{normalized score} = \\frac{(\\text{score} - 6.7)}{(273.99 - 6.7)}*100\n$$\n\n**Q: Main text should include limitation discussions.**\n\n**A:** We inevitably describe limitation of our work in Appendix section due to space constraint. We will include limitation in main text in camera-ready version.\n\n**References**\n\n- Janner et al., Planning with Diffusion for Flexible Behavior Synthesis. In *ICML*, 2022."}, {"review_id": "JSc8o0ZinD", "reviewer": "Reviewer_9DAN", "summary": "In this work, authors propose a new metric called “restoration gap” that is able to measure the reliability of plans emerging from diffusion-based planners. They then suggest using this metric along with guidance, “restoration guidance”, to encourage the diffusion-based planner to generate more reliable plans. In addition, they attempt to improve the performance of restoration guidance by using an attribution map regularizer. Finally, they evaluate their method on three different offline control benchmarks.", "questions": "Overall, the paper is well-written and easy to follow. However, there are some parts that need improvement. For me, these include:\n- In equation 4, the addition of trajectory optimization is vague. Maybe, a reference to section 3.2 of Janner et al. 2022 where trajectory optimality in diffusion models is introduces could be beneficial.\n- In equation 5, g is defined but is not explained what it is supposed to be and how it is supposed to be used. For someone new to the field, this is problematic. \n- In figure 2, the realism gap and rarity score are shown alongside restoration gap but they are not explained until section 5.1. I recommend moving this explanation to the caption of figure 2 or to section 3. \n- In section 3, it is explained why restoration gap is an intuitive measure of plan reliability. It would be better if you could clarify what reliability really means. Are reliable plans ones that are within distribution? Moreover, a discussion on how much noise is required is also essential as too much noise may push the recovered samples too far from the original and too little will not provide any improvements.\n- Attribution map is a good option for showing the explainability of diffusion-planner with the trained gap predictor. However, its significance as a regularizer is not supported by the experimental results. Therefore, I would remove section 4.2 as it doesn’t provided any added value.\n- Overall, the enhancements offered by the new method (RGG and RGG+) in comparison to Diffuser do not appear to be substantial, if any. As a result, the overall significance of the work is diminished in my perspective.\n- Following section 5.1, it would be valuable to show how guidance with restoration gap compares with guidance with realism score and rarity score. This could potentially support the fact that the newly proposed metric is more suitable for the task than other available metrics.\n- Throughtout the paper, it is claimed that the proposed approach for improving unreliable plans emerging from diffusion-based planners is novel but this claim is not supported. Firstly, the provided solution which is to use guidance based on some form of reliability-measuring metric, is not necessarily novel and it seems to be the immediate and instinctive response to addressing the issue. Moreover, the newly proposed metric which is based on the addition of noise for sample recovery and purification has been explored before as well (see SDEdit by Meng et al. and Diffusion Models for Adversarial Purification by Nie et al.). Therefore, the authors’ claim in the approach being novel has to either be removed or clarified.\n", "limitations": "In addition to the recommendations on the specific technical parts (7 bullet points in “Questions”), I highly recommend developing the idea of purifying diffuser-generated plans further using various techniques and comparing what provides the best performance. Moreover, designing an experimental setup that captures the significance of reliability of plans is highly beneficial to support the significance and impact of the work. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "Overall Strengths:\n1. The problem of detecting and/or improving unreliable plans generated by diffusion-based planners is both interesting and important and I appreciate the authors for their efforts towards this direction.\n2. Paper is well-written and well-presented with good quality visualizations, plots and tables.\n3. Experiments are covering a good-enough domain of tasks and compare against a large number of benchmarks.\n", "weaknesses": "Overall Weaknesses:\n1. A critical weakness, in my view, is the lack of substantial improvement demonstrated by the proposed solution compared to the Diffuser benchmark. Across various experiments, RGG exhibits only marginal enhancements (if any) over Diffuser, and similarly, RGG+ shows only minor improvements (if any) over RGG. This observation raises two possibilities: either the solution itself lacks significant impact or the current experimental setup fails to adequately capture the method's importance. While I appreciate the authors' overall research trajectory, it is my belief that this particular submission requires further development and refinement.\n2. Additionally, there are some additional limitations in the work. Firstly, to establish the significance of their approach, it would be valuable to investigate how it compares to rejection-based methods, especially in scenarios where the problem context remains fixed. For instance, in the context of the Maze2D experiments, the primary objective is to find solutions that navigate around obstacles. One potential approach could involve incorporating collision-detection code and subsequently discarding samples that do not satisfy this condition. \n3. Furthermore, to demonstrate the significance of the proposed metric, it would be valuable to show how guidance with restoration gap compares with guidance with realism score and rarity score (or other readily avaible metrics).\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Comparison to rejection-based methods, especially in scenarios where the problem context remains fixed.**\n\n**A:** We respectfully disagree with the reviewer’s comment on comparing our approach to rejection-based methods. It is important to note that rejection-based methods rely on strong assumptions, such as requiring the oracle understanding of the environment, which is uncommon in general machine learning scenarios.\n\nFor example in the context of Maze2D environments, incorporating collision-detection code would necessitate the agent to have an understanding of the location of every wall in the environment where it interacts. However, in Maze2D environments, the agent only observes its 2D position, which lacks information about the location of walls.\n\nWe appreciate the reviewer’s suggestion, but considering the strong assumptions and the context of our approach, comparing to rejection-based methods may not be a suitable avenue of investigation.\n\n**Q: In equation 4, the addition of trajectory optimization is vague.** \n\n**A:** Thank you for pointing out the ambiguity in Equation 4. We’ll add a reference to Section 3.2 of Janner et al. (2022) for clarity on trajectory optimization in our revised version.\n\n**Q: In equation 5, g is defined but is not explained.**\n\nA: $g$ is defined to be the derivative of objective value $\\mathcal{J}$ as represented in Equation 5. However, as there is no explicit description on this, we’ll include it in the revised version.\n\n**Q: In figure 2, the realism gap and rarity score are shown alongside restoration gap but they are not explained until section 5.1.**\n\n**A:** Due to page constraints, we couldn’t initially include the description of them in Section 3. In the camera-ready version that allows an additional page, we’ll ensure the explanations align better with Figure 2 for clarity.\n\n**Q: Are reliable plans ones that are within distribution? Moreover, a discussion on how much noise is required is also essential.**\n\n**A:** In the context of our work, a plan’s **reliability** indicates its feasibility in execution and its alignment with the training distribution. Therefore, yes, reliable plans are ones that are within the distribution, assuming a sufficient amount of offline dataset.\n\nAccording to Proposition 1, minimizing both type I error (false positives) and type II error (false negatives) requires a substantial $\\sigma_{\\hat{t}}$, which, in turn, a large $\\hat{t}$. We empirically observe that the $\\hat{t}=0.9$ works well for all tasks.\n\nFurthermore, in our ablation study, we explored how sensitive the restoration gap is to the value of $\\hat{t}$ as shown in Figure 6 in Appendix C.2. We compared the performance of plans that are chosen from the restoration gap when setting $\\hat{t}$ to 0.3, 0.5, 0.7, and 0.9, in addition from the negative realism score and rarity score. As a result, we observed that the restoration gap is robust to $\\hat{t}$ though $\\hat{t}=0.9$ performs the best which supports Proposition 1. Moreover, the restoration gap demonstrated superior correlation with the planning performance over other metrics.\n\n**Q: Significance of attribution map as a regularizer is not supported by the experimental results.** \n\n**A:** We respectfully disagree with your assessment.\n\nThe beneficial effect of RGG+ as a regularizer is clearly illustrated in Figure 3 in Section 4.2, which shows the performance of RGG+ in the Maze2D-Large task across a range of $\\lambda$ values. In the absence of regularization ($\\lambda=0$), our diffusion planner is prone to generate adversarial plans which lead to a decline in performance as the planning budget increased. On the other hand, RGG+ maintains its effectiveness over a broad range of $\\lambda$ values, where any $\\lambda>0$ consistently outperforms $\\lambda=0$, indicating the advantage of regularization. Moreover, RGG+ performs better than RGG across a wide spectrum of tasks including Maze2D, Locomotion, and Block Stacking. Hence, we believe that the content of Section 4.2 is non-trivial to our work and should not be omitted.\n\n**Q: The authors’ claim in the approach being novel has to either be removed or clarified.**\n\n**A:** We respectfully disagree. Adding a small amount of noise and then performing sample recovery through a reverse generative process has been explored in tasks like image synthesis (Meng et al., 2021) and adversarial purification (Nie et al., 2022). However, these works do not provide a way for evaluating the quality of individually generated samples produced by a diffusion model. Our work uniquely proposes a metric for assessing trajectory quality created by diffusion models, along with a novel refining guidance approach to address the limitations of diffusion-based planning.\n\nFurthermore, other reviewers appear to align with our perspective on novelty. Reviewer iw4y noted that our work appears to be novel, reviewer N3Wq recognized our novel approach as contributing to the field and expanding understanding of plan refinement, and reviewer pN8B stated that our contribution to the paper is solid, recognizing the restoration gap as an intuitive metric for measuring trajectory quality.\n\nWe firmly believe that the novelty of our approach, which focuses on evaluating and refining trajectory quality generated by diffusion models, is non-trivial and has the potential to contribute meaningfully to the field.\n\n**Q: Lack of substantial improvement demonstrated by the proposed solution.**\n\n**A:** Please refer to the global response provided above.\n\n**Q: Show how guidance with restoration gap compares with guidance with realism score and rarity score.**\n\n**A:** Please refer to the global response provided above.\n\n**References**\n\n- Janner et al., Planning with Diffusion for Flexible Behavior Synthesis. In *ICML*, 2022.\n- Meng et al., Sdedit: Guided image synthesis and editing with stochastic differential equations. In *ICLR*, 2021.\n- Nie et al., Diffusion models for adversarial purification. In *ICML*, 2022."}]

Author rebuttal:
We would like to sincerely thank all reviewers for their time and effort in reviewing our paper, and for their constructive feedback and comments. In this thread, we answer commonly asked questions. Please find below our response to your insightful comments and suggestions.

**Q: Lack of substantial improvement demonstrated by the proposed solution compared to the Diffuser benchmark. (to Reviewer iw4y and 9DAN)**

**A:** We would like to clarify that the planning performance improvement by restoration gap guidance (RGG) is not trivial. To the best of our knowledge, concurrent works to our approach that followed Diffuser (Janner et al., 2022) can be summarized into two methods: 1) Decision Diffuser (Ajay et al., 2023) and 2) AdaptDiffuser (Liang et al., 2023), the latter of which we have recently discovered at ICML 2023.

When comparing RGG to Decision Diffuser, we find that RGG performs on par with a slight difference. For instance, in Locomotion tasks (Fu et al., 2020), the average scores achieved by RGG and RGG+ are 81.2 and 81.6, respectively, while Decision Diffuser reaches a score of 81.8. However, Decision Diffuser necessitates training the diffusion model from scratch through classifier-free guidance with low temperature sampling, while our RGG does not require such a process and can be applied to any unconditional diffusion-based planner which offers a chance to further refinement of generated sequence. Moreover, explicitly modeling the restoration gap predictor in our approach enables the diffusion planner to additionally have explainability, facilitating the identification of error-prone transitions within infeasible trajectories. These distinctions make the performance achievement of RGG non-trivial.

Regarding the comparison with AdaptDiffuser, we acknowledge that the planning performance of RGG and RGG+ is slightly lower but still comparable. It is crucial to note that AdaptDiffuser assumes knowledge of the oracle inverse dynamics for collecting additional training data, which is a quite strong assumption and uncommon in general machine learning scenarios. Considering this, we firmly assert that our achievement in performance is indeed significant, as it does not rely on such specific and uncommon assumptions.

**Q: Is it possible to adapt the metrics compared to in 5.1 to downstream control? (to Reviewer iw4y and 9DAN)**

**A:** To directly compare the performance of plans guided by different metrics, we run new experiments in the Maze2D Large and Multi2D Large environments. The results are as follows:

| Environment | Diffuser | Rarity | Negative Realism | RGG | RGG+ |
|:--------------:|:--------:|:------:|:----------------:|:-----:|:-----:|
| Maze2D Large | 123.5 | 126.9 | 128.9 | **135.4** | **143.9** |
| Multi2D Large | 141.2 | 143.4 | 143.3 | **148.3** | **150.9** |

The result clearly shows that the restoration gap is a useful metric for control tasks. Unlike other metrics that need expert data for training, our restoration gap works without such constraints, making it even more practical.

**Q: Can author give some results / explanations on how sensitive the performance of the model is to the additional hyperparameters introduced (beta, lambda in eq. 22) and how hard it is to tune them? (to Reviewer N3Wq and pN8B)**

**A:** The planning performance of RGG and RGG+ is somewhat sensitive to these hyperparameters. In Figure 3, we can observe implicit indications of sensitivity, where the performance varies depending on the choice of the value of $\lambda$.

However, we would like to highlight that there exists an evident tendency in the change of planning performance relative to the selected values of these hyperparameters, making it manageable to tune them effectively through a grid search. For example, Figure 3 illustrates that the planning performance increases as the value of $\lambda$ rises, but then declines for excessively large $\lambda$ values.

In our experiments, we conducted a grid search for the hyperparameters where the search spaces are presented in Table 10, 11, 12, 13, and 14, for Maze2D, HalfCheetah, Hopper, Walker2D, and Block Stacking experiments, respectively. The planning performance exhibited a clear trend based on the choices of $\alpha$, $\beta$, and $\lambda$, thus requiring only a small number of evaluation episodes for the grid search. For instance, we used 10, 15, and 10 evaluation episodes for Maze2D, Locomotion, and Block Stacking experiments, respectively.

We genuinely appreciate the reviewer’s valuable feedback, and we agree that including an ablation study on the sensitivity of the planning performance to the scaling coefficients would enhance the thoroughness of our work. We will incorporate this study in the revised version to provide further clarify on this aspect.

**References**

- Janner et al., Planning with Diffusion for Flexible Behavior Synthesis. In *ICML*, 2022.
- Ajay et al., Is Conditional Generative Modeling All You Need for Decision-Making? In *ICLR*, 2023.
- Liang et al., AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners. In *ICML*, 2023.
- Fu et al., D4rl: Datasets for Deep Data-Driven Reinforcement Learning. *arXiv*, 2020.

Meta-review: Authors propose an interesting restoration gap metric and predictor to help refine diffusion planners towards more feasible path generations (not going through obstacles etc). While some reviewers initially had some concerns, the authors have taken care to address all concerns, and reviewers have larger accepted all these rebuttals. This paper is a clear accept.
Decision: Accept (poster)