SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kyowoon Lee* UNIST
leekwoon@unist.ac.kr
Seongun Kim* KAIST
seongun@kaist.ac.kr
Jaesik Choi  
KAIST, INEEJI  
jaesik.choi@kaist.ac.kr
Diffusion-based planning has shown promising results in long-horizon, sparse-reward tasks by training trajectory diffusion models and conditioning the sampled trajectories using auxiliary guidance functions. However, due to their nature as generative models, diffusion models are not guaranteed to generate feasible plans, resulting in failed execution and precluding planners from being useful in safety-critical applications. In this work, we propose a novel approach to refine unreliable plans generated by diffusion models by providing refining guidance to error-prone plans. To this end, we suggest a new metric named restoration gap for evaluating the quality of individual plans generated by the diffusion model. A restoration gap is estimated by a gap predictor which produces restoration gap guidance to refine a diffusion planner. We additionally present an attribution map regularizer to prevent adversarial refining guidance that could be generated from the sub-optimal gap predictor, which enables further refinement of infeasible plans. We demonstrate the effectiveness of our approach on three different benchmarks in offline control settings that require long-horizon planning. We also illustrate that our approach presents explainability by presenting the attribution maps of the gap predictor and highlighting error-prone transitions, allowing for a deeper understanding of the generated plans.
Planning plays a crucial and efficient role in tackling decision-making problems when the dynamics are known, including board games and simulated robot control (Tassa et al., 2012; Silver et al., 2016, 2017; Lee et al., 2018). To plan for more general tasks with unknown dynamics, the agent needs to learn the dynamics model from experience. This approach is appealing since the dynamics model is independent of rewards, enabling it to adapt to new tasks in the same environment, while also taking advantage of the latest advancements from deep supervised learning to employ high-capacity models.
The most widely used techniques for learning dynamics models include autoregressive forward models (Deisenroth & Rasmussen, 2011; Hafner et al., 2019; Kaiser et al., 2020), which make predictions based on future time progression. Although an ideal forward model would provide significant benefits, there is a key challenge that the accuracy of the model directly affects the quality of the plan. As model inaccuracies accumulate over time (Ross & Bagnell, 2012; Talvitie, 2014; Luo et al., 2019; Janner et al., 2019; Voelcker et al., 2022), long-term planning using imprecise models might yield sub-optimal performances compared to those achievable through model-free techniques. Building upon the latest progress in generative models, recent studies have shown promise in transforming reinforcement learning (RL) problems into conditional sequence modeling, through the modeling of the joint distribution of sequences involving states, actions, and rewards
*Equal Contribution Codes are available at https://github.com/leekmoon/rgg.
Figure 1: Illustration of two plans with low/high restoration gaps with a specified start  $\bullet$  and goal  $\star$ . For each input plan, we first perturb it using Gaussian noise. We then remove the noise from the perturbed plan by simulating the reverse SDE which progressively transforms the perturbed plan into the initial plan by utilizing the score function (Section 2.2). The restoration gap is then computed as the expected  $L_{2}$  distance between the input plan and the plan restored from noise corruption (Section 3). The top example exhibits a smaller restoration gap because of its successful restoration close to the original plan, while the bottom example has a larger restoration gap due to its poor restoration performance. Plans restored from various noise corruptions are differentiated by distinct colors.
(Lambert et al., 2021; Chen et al., 2021; Janner et al., 2021, 2022). For instance, Diffuser (Janner et al., 2022) introduces an effective framework for generating trajectories using a diffusion model with flexible constraints on the resulting trajectories through reward guidance in the sampling phase. Although these approaches have achieved notable performance on long-horizon tasks, they still face challenges in generating outputs with unreliable trajectories, referred to as artifacts, resulting in limited performance and unsuitability for deployment in safety-critical applications.
This paper presents an orthogonal approach aimed at enhancing the plan quality of the diffusion model. We first propose a novel metric called restoration gap that can automatically detect whether generated plans are feasible or not. We theoretically analyze that it could detect artifacts with bounded error probabilities under regularity conditions. The restoration gap directly evaluates the quality of generated plans by measuring their restorability through diffusion models in which plans are exposed to a certain degree of noise, as illustrated in Figure 1. A restoration gap is estimated by a function approximator which we name a gap predictor. The gap predictor provides an additional level of flexibility to the diffusion model, and we demonstrate its ability to efficiently improve low-quality plans by guiding the reduction of the estimated restoration gap through a process, which we call Restoration Gap Guidance (RGG). Furthermore, we propose a regularizer that prevents adversarial restoration gap guidance by utilizing an attribution map of the gap predictor. It effectively mitigates the risk of the plan being directed towards an unreliable plan, enabling further improvement in the planning performance.
The main contributions of this paper are summarized as follows: (1) We provide a novel metric to assess the quality of individual plans generated by the diffusion model with theoretical justification. (2) We propose a new generative process, Restoration Gap Guidance (RGG) which utilizes a gap predictor that estimates the restoration gap. (3) We show the effectiveness of our approach across three different benchmarks in offline control settings.
2.1 Planning with Diffusion Probabilistic Models
We consider the reinforcement learning problem which aims to maximize the expected discounted sum of rewards  $\mathbb{E}_{\pi}[\sum_{t=0}^{T} \gamma^{t} r(\pmb{s}_{t}, \pmb{a}_{t})]$  where  $\pi$  is a policy that defines a distribution over actions
$\pmb{a}_t, \pmb{s}_t$  represents the states that undergo transition according to unknown discrete-time dynamics  $\pmb{s}_{t+1} = f(\pmb{s}_t, \pmb{a}_t), r: S \times \mathcal{A} \to \mathbb{R}$  is a reward function, and  $\gamma \in (0,1]$  is the discount factor. Trajectory optimization solves this problem by finding the sequence of actions  $\pmb{a}_{0:T}^*$  that maximizes the expected discounted sum of rewards over planning horizon  $T$ :
$$
\boldsymbol {a} _ {0: T} ^ {*} = \underset {\boldsymbol {a} _ {0: T}} {\arg \max } \mathcal {J} (\boldsymbol {\tau}) = \underset {\boldsymbol {a} _ {0: T}} {\arg \max } \sum_ {t = 0} ^ {T} \gamma^ {t} r \left(\boldsymbol {s} _ {t}, \boldsymbol {a} _ {t}\right), \tag {1}
$$
where  $\pmb{\tau} = (s_0, a_0, s_1, a_1, \dots, s_t, a_t)$  represents a trajectory and  $\mathcal{J}(\tau)$  denotes an objective value of that trajectory. This trajectory can be viewed as a particular form of two-dimensional sequence data:
$$
\boldsymbol {\tau} = \left[ \begin{array}{l l l l} \boldsymbol {s} _ {0} & \boldsymbol {s} _ {1} & \dots & \boldsymbol {s} _ {T} \\ \boldsymbol {a} _ {0} & \boldsymbol {a} _ {1} & \dots & \boldsymbol {a} _ {T} \end{array} \right]. \tag {2}
$$
Diffuser (Janner et al., 2022) is a trajectory planning model, which models a trajectory distribution by employing diffusion probabilistic models (Sohl-Dickstein et al., 2015; Ho et al., 2020):
$$
p _ {\theta} \left(\boldsymbol {\tau} ^ {0}\right) = \int p \left(\boldsymbol {\tau} ^ {N}\right) \prod_ {i = 1} ^ {N} p _ {\theta} \left(\boldsymbol {\tau} ^ {i - 1} \mid \boldsymbol {\tau} ^ {i}\right) \mathrm {d} \boldsymbol {\tau} ^ {1: N} \tag {3}
$$
where  $p(\boldsymbol{\tau}^N)$  is a standard Gaussian prior,  $\boldsymbol{\tau}^0$  is a noiseless trajectory, and  $p_{\theta}(\boldsymbol{\tau}^{i-1}|\boldsymbol{\tau}^i)$  is a denoising process which is a reverse of a forward process  $q(\boldsymbol{\tau}^i|\boldsymbol{\tau}^{i-1})$  that gradually deteriorates the data structure by introducing noise. The denoising process is often parameterized as Gaussian with fixed timestep-dependent covariances:  $p_{\theta}(\boldsymbol{\tau}^{i-1}|\boldsymbol{\tau}^i) = \mathcal{N}(\boldsymbol{\tau}^{i-1}|\boldsymbol{\mu}_{\theta}(\boldsymbol{\tau}^i,i),\boldsymbol{\Sigma}^i)$ . Diffuser recasts the trajectory optimization problem as a conditional sampling with the conditional diffusion process under smoothness condition on  $p(\mathcal{O}_{1:T} = 1|\boldsymbol{\tau})$  (Sohl-Dickstein et al., 2015):
$$
\tilde {p} _ {\theta} (\boldsymbol {\tau}) = p (\boldsymbol {\tau} \mid \mathcal {O} _ {1: T} = 1) \propto p (\boldsymbol {\tau}) p \left(\mathcal {O} _ {1: T} = 1 \mid \boldsymbol {\tau}\right), \quad p _ {\theta} \left(\boldsymbol {\tau} ^ {i - 1} \mid \boldsymbol {\tau} ^ {i}, \mathcal {O} _ {1: T}\right) \approx \mathcal {N} \left(\boldsymbol {\tau} ^ {i - 1}; \boldsymbol {\mu} + \boldsymbol {\Sigma} g, \boldsymbol {\Sigma}\right) \tag {4}
$$
where  $\pmb{\mu}, \pmb{\Sigma}$  are the parameters of the denoising process  $p_{\theta}(\pmb{\tau}^{i-1}|\pmb{\tau}^i)$ ,  $\mathcal{O}_t$  is the optimality of timestep  $t$  of trajectory with  $p(\mathcal{O}_t = 1) = \exp(\gamma^t r(\pmb{s}_t, \pmb{a}_t))$  and
$$
g = \nabla_ {\boldsymbol {\tau}} \log p \left(\mathcal {O} _ {1: T} | \boldsymbol {\tau}\right) | _ {\boldsymbol {\tau} = \boldsymbol {\mu}} = \sum_ {t = 0} ^ {T} \gamma^ {t} \nabla_ {\boldsymbol {s} _ {t}, \boldsymbol {a} _ {t}} r \left(\boldsymbol {s} _ {t}, \boldsymbol {a} _ {t}\right) | _ {\left(\boldsymbol {s} _ {t}, \boldsymbol {a} _ {t}\right) = \boldsymbol {\mu} _ {t}} = \nabla \mathcal {J} (\boldsymbol {\mu}). \tag {5}
$$
Therefore, a separate model  $\mathcal{I}_{\phi}$  can be trained to predict the cumulative rewards of trajectory samples  $\tau^i$ . By utilizing the gradients of  $\mathcal{I}_{\phi}$ , trajectories with high cumulative rewards can be generated.
As part of the training procedure, Diffuser trains an  $\epsilon$ -model to predict the source noise instead of training  $\mu_{\theta}$  as it turns out that learning  $\epsilon_{\theta}$  enables the use of a simplified objective, where  $\mu_{\theta}$  is easily recovered in a closed form (Ho et al., 2020):
$$
\mathcal {L} (\theta) := \mathbb {E} _ {i, \epsilon , \tau^ {0}} [ \| \epsilon - \epsilon_ {\theta} (\tau^ {i}, i) \| ^ {2} ], \tag {6}
$$
where  $i \in \{0,1,\dots,N\}$  is the diffusion timestep,  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  is the target noise, and  $\tau^i$  is the trajectory corrupted by the noise  $\epsilon$  from the noiseless trajectory  $\tau^0$ .
2.2 Generalizing Diffusion Probabilistic Models as a Stochastic Differential Equation (SDE)
The forward process in diffusion probabilistic models perturbs data structure by gradually adding Gaussian noises. Under an infinite number of noise scales, this forward process over continuous time can be represented as a stochastic differential equation (SDE) (Song et al., 2021):
$$
\mathrm {d} \boldsymbol {\tau} = \mathbf {f} (\boldsymbol {\tau}, t) \mathrm {d} t + g (t) \mathrm {d} \mathbf {w}, \tag {7}
$$
where  $t \in (0,1]$  is a continuous time variable for indexing diffusion timestep,  $\mathbf{f}(\tau ,t)$  is the drift coefficient,  $g(t)$  is the diffusion coefficient, and  $\mathbf{w}$  is the standard Wiener process. Similarly, the denoising process can be defined by the following reverse-time SDE:
$$
\mathrm {d} \boldsymbol {\tau} = \left[ \mathbf {f} (\boldsymbol {\tau}, t) - g (t) ^ {2} \mathbf {s} _ {\theta} (\boldsymbol {\tau}, t) \right] \mathrm {d} t + g (t) \mathrm {d} \bar {\mathbf {w}}, \tag {8}
$$
where  $\bar{\mathbf{w}}$  is the infinitesimal noise in the reverse time direction and  $\mathbf{s}_{\theta}(\pmb {\tau},t)$  is the learned score network which estimates the data score  $\nabla_{\tau}\log p_t(\tau)$ . This score network can be replaced by the  $\epsilon$ -model:
$$
\mathbf {s} _ {\theta} (\boldsymbol {\tau}, t) \approx \nabla_ {\boldsymbol {\tau}} \log q (\boldsymbol {\tau}) = \mathbb {E} _ {\boldsymbol {\tau} ^ {0}} [ \nabla_ {\boldsymbol {\tau}} \log q (\boldsymbol {\tau} | \boldsymbol {\tau} ^ {0}) ] = \mathbb {E} _ {\boldsymbol {\tau} ^ {0}} \left[ - \frac {\epsilon_ {\theta} (\boldsymbol {\tau} , t)}{C _ {t}} \right] = - \frac {\epsilon_ {\theta} (\boldsymbol {\tau} , t)}{C _ {t}}, \tag {9}
$$
where  $C_t$  is a constant determined by the chosen perturbation strategies.
The solution of a forward SDE is a time-varying random variable  $\pmb{\tau}^t$ . Using the reparameterization trick (Kingma & Welling, 2014), it is achieved by sampling a random noise  $\epsilon$  from a standard Gaussian distribution which is scaled by the target standard deviation  $\sigma_t$  and shifted by the target mean:
$$
\boldsymbol {\tau} ^ {t} = \alpha_ {t} \boldsymbol {\tau} ^ {0} + \sigma_ {t} \boldsymbol {\epsilon}, \quad \boldsymbol {\epsilon} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}), \tag {10}
$$
where  $\alpha_{t}:[0,1]\to [0,1]$  denotes a scalar function indicating the magnitude of the noiseless data  $\pmb{\tau}^{0}$ , and  $\sigma_t:[0,1]\rightarrow [0,\infty)$  denotes a scalar function that determines the size of the noise  $\epsilon$ . Depending on perturbation strategies for  $\alpha_{t}$  and  $\sigma_{t}$ , two types of SDEs are commonly considered: the Variance Exploding SDE (VE-SDE) has  $\alpha_{t} = 1$  for all  $t$ ; whereas the Variance Preserving (VP) SDE satisfies  $\alpha_{t}^{2} + \sigma_{t}^{2} = 1$  for all  $t$ . Both VE and VP SDE change the data distribution to random Gaussian noise as  $t$  moves from 0 to 1. In this work, we describe diffusion probabilistic models within the continuous-time framework using VE-SDE to simplify notation, as VE/VP SDEs are mathematically equivalent under scale translations (Song et al., 2021).
For VE SDE, the forward process and denoising process are defined by the following SDEs:
$$
\text {(F o r w a r d S D E)} \mathrm {d} \boldsymbol {\tau} = \sqrt {\frac {\mathrm {d} \left[ \sigma_ {t} ^ {2} \right]}{\mathrm {d} t}} \mathrm {d} \mathbf {w} \tag {11}
$$
$$
\left(\operatorname {R e v e r s e} \mathrm {S D E}\right) \mathrm {d} \boldsymbol {\tau} = \left[ - \frac {\mathrm {d} \left[ \sigma_ {t} ^ {2} \right]}{\mathrm {d} t} \mathbf {s} _ {\theta} (\boldsymbol {\tau}, t) \right] \mathrm {d} t + \sqrt {\frac {\mathrm {d} \left[ \sigma_ {t} ^ {2} \right]}{\mathrm {d} t}} \mathrm {d} \bar {\mathbf {w}}. \tag {12}
$$
3 Restoration Gap
To assess the quality of plans generated by diffusion probabilistic models, we propose a novel metric named restoration gap. It aims to automatically detect infeasible plans that violate system constraints. We hypothesize that for feasible plans, even if a certain amount of noise perturbs them, they can be closely restored to their initial plans by diffusion models. It is attributed to the property of temporal compositionality in diffusion planners (Janner et al., 2022) that encourages them to compose feasible trajectories by stitching together any feasible plan subsequences. However, for infeasible plans that obviously fall out of the training distribution as they violate physical constraints as shown in Figure 5, restoring them to a state near their original conditions is challenging. Based on this intuition, we define the restoration gap of the generated plan  $\tau$  as follows:
$$
\operatorname {p e r t u r b} _ {\hat {t}} (\boldsymbol {\tau}) = \boldsymbol {\tau} + \sigma_ {\hat {t}} \boldsymbol {\epsilon} _ {\hat {t}}, \quad \boldsymbol {\epsilon} _ {\hat {t}} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}) \tag {13}
$$
$$
\operatorname {r e s t o r e} _ {\hat {t}, \theta} (\boldsymbol {\tau}) = \boldsymbol {\tau} + \int_ {\hat {t}} ^ {0} \left[ - \frac {\mathrm {d} [ \sigma_ {t} ^ {2} ]}{\mathrm {d} t} \mathbf {s} _ {\theta} (\boldsymbol {\tau}, t) \right] \mathrm {d} t + \sqrt {\frac {\mathrm {d} [ \sigma_ {t} ^ {2} ]}{\mathrm {d} t}} \mathrm {d} \bar {\mathbf {w}} \tag {14}
$$
$$
\operatorname {r e s t o r a t i o n} \operatorname {g a p} _ {\hat {t}, \theta} (\boldsymbol {\tau}) = \mathbb {E} _ {\boldsymbol {\epsilon} _ {\hat {t}}} \left[ \| \boldsymbol {\tau} - \operatorname {r e s t o r e} _ {\hat {t}, \theta} \left(\operatorname {p e r t u r b} _ {\hat {t}} (\boldsymbol {\tau})\right) \| _ {2} \right], \tag {15}
$$
where  $\hat{t} \in (0,1]$  indicates the magnitude of applied perturbation. The restoration gap measures the expected  $L_{2}$  distance between the generated plan and the plan restored from noise corruption, which is estimated by the Monte Carlo approximation.
Figure 2 provides empirical evidence supporting our hypothesis. In order to analyze the effectiveness of the restoration gap, we define artifact plans generated by Diffuser (Janner et al., 2022) that involve transitions of passing through walls for which it is impossible for the agent to follow. We compare the distribution of the restoration gap for both groups, normal plans and artifact plans<sup>1</sup>. The histogram of the restoration gap for normal and artifact plans demonstrates that infeasible artifact plans have larger
The purpose of defining artifacts in this manner is solely to validate our hypothesis. Artifacts are not explicitly defined beyond the scope of this validation.
Figure 2: The first and second rows show examples of artifact and normal plans, respectively, generated by Diffuser (Janner et al., 2022) in the Maze2D-Large environment, including a predetermined start  $\bullet$  and goal  $\star$ . The third row presents the density of realism score (Kynkänniemi et al., 2019), rarity score (Han et al., 2023), and restoration gap to illustrate the differences in distribution between artifacts and normal plans. Detailed explanation of other metrics is described in Appendix C.2.
restoration gap values compared to normal plans. Therefore, the detection of infeasible artifact plans can be automated by incorporating a statistical test that utilizes the restoration gap and thresholding with a threshold value of  $b > 0$ :
$$
\text {r e s t o r a t i o n} \operatorname {g a p} _ {\hat {t}, \theta} (\tau) > b. \tag {16}
$$
To bound the probability of making errors by choosing the specific threshold  $b$ , we provide Proposition 1. Let  $\mathbb{H}_0$  represent the null hypothesis which assumes that the trajectory  $\pmb{\tau}$  belongs to the normal set  $\mathcal{T}_{\mathrm{normal}}$ , and let  $\mathbb{H}_1$  represent the alternative hypothesis which assumes that the trajectory  $\pmb{\tau}$  belongs to the artifact set  $\mathcal{T}_{\mathrm{artifacts}}$ . The following proposition suggests how to choose the threshold  $b$  in order to bound the error probabilities.
Proposition 1. Given  $t \in [0,1]$  and a positive constant  $C, \Delta$ , assume that  $\| \mathbf{s}_{\theta}(\pmb{\tau},t)\|_2^2 \leq C^2$  for all  $\pmb{\tau} \in \mathcal{T}_{\mathrm{normal}} \subset \mathbb{R}^d$ , and  $\| \mathbf{s}_{\theta}(\pmb{\tau},t)\|_2^2 \geq (C + \Delta)^2$  for all  $\pmb{\tau} \in \mathcal{T}_{\mathrm{artifacts}} \subset \mathbb{R}^d$ . If
$$
\Delta \geq \frac {2 \sqrt {d} + 2 \sqrt {d + 2 \sqrt {- d \cdot \log \delta} - 2 \log \delta}}{\sigma_ {\hat {t}}}, \tag {17}
$$
then setting
$$
b \geq \sigma_ {\hat {t}} \left(C \sigma_ {\hat {t}} + \sqrt {d} + \sqrt {d + 2 \sqrt {- d \cdot \log \delta} - 2 \log \delta}\right) \tag {18}
$$
guarantees both type I and type II errors at most  $2\delta$
Proof Sketch. We begin by deriving thresholds  $b_{I}$  and  $b_{II}$  to control type I and type II errors at most  $\delta$ , respectively. This is done by decomposing the restoration gap into the outcomes of the score and Gaussian noise. To ensure the control of both type I and type II errors, we examine the condition  $b_{I} \leq b_{II}$  and obtain the conclusion. For the complete proof, see Appendix A.
According to Proposition 1, to achieve low error probabilities for both type I (false positives, where normal trajectories are incorrectly classified as artifacts) and type II (false negatives, where artifact trajectories are wrongly identified as normal) errors, it is essential to have a large enough  $\sigma_{\hat{t}}$  to properly satisfy the condition, which implies having a large enough  $\hat{t}$ . In practice, we find that setting  $\hat{t} = 0.9$  works well.
4 Refining Diffusion Planner
4.1 Restoration Gap Guidance
Although Diffuser (Janner et al., 2022) has demonstrated competitive performance against previous non-diffusion-based planning methods by utilizing gradients of return  $\mathcal{I}_{\phi}$  to guide trajectories during the denoising process:
$$
\mathrm {d} \boldsymbol {\tau} = \left[ \mathbf {f} (\boldsymbol {\tau}, t) - g (t) ^ {2} \left(\mathbf {s} _ {\theta} (\boldsymbol {\tau}, t) + \alpha \nabla \mathcal {J} _ {\phi} (\boldsymbol {\tau})\right) \right] \mathrm {d} t + g (t) \mathrm {d} \bar {\mathbf {w}}, \tag {19}
$$
it entirely relies on the ability of a generative model and assumes a perfect data score estimation. For plans with inaccurately estimated scores, the diffusion models could generate unreliable plans that are infeasible to execute and lead to limited performance. To address this, it is essential to construct an adjusted score to refine the generative process of the diffusion planner. Therefore, we estimate the restoration gap by training a gap predictor  $\mathcal{G}_{\psi}$  on synthetic diffused data generated through the diffusion process, taking full advantage of its superior generation ability with conditional guidance from gradients of return. Parameters of the gap predictor  $\psi$  are optimized by minimizing the following objective:
$$
\mathcal {L} (\psi) := \mathbb {E} _ {t, \tau^ {0}} [ \| \text {r e s t o r a t i o n} \operatorname {g a p} _ {\hat {t}, \theta} (\boldsymbol {\tau} ^ {t}) - \mathcal {G} _ {\psi} (\boldsymbol {\tau} ^ {t}, t) \| ^ {2} ], \tag {20}
$$
where  $t \in (0,1]$  denotes a continuous time variable for indexing the diffusion timestep, and  $\tau^t$  is the diffused trajectory resulting from  $\tau^0$  at diffusion timestep  $t$ . With this gap predictor, we define the Restoration Gap Guidance (RGG) as follows:
$$
\mathrm {d} \boldsymbol {\tau} = \left[ \mathbf {f} (\boldsymbol {\tau}, t) - g (t) ^ {2} \left(\mathbf {s} _ {\theta} (\boldsymbol {\tau}, t) + \alpha \left(\nabla \mathcal {J} _ {\phi} (\boldsymbol {\tau}) - \beta \nabla \mathcal {G} _ {\psi} (\boldsymbol {\tau}, t)\right)\right) \right] \mathrm {d} t + g (t) \mathrm {d} \bar {\mathbf {w}}, \tag {21}
$$
where  $\alpha$  is a positive coefficient that scales the overall guidance and  $\beta$  is a positive coefficient that can be adjusted to enforce a small restoration gap for the generated trajectory.
4.2 Attribution Map Regularization
Although guiding the diffusion planner to minimize the restoration gap effectively refines low-quality plans (more details in Section 5), this refining guidance could push the plan in an undesirable direction due to the estimation error of the gap predictor during the denoising process. As a result of this estimation error, guiding plans with the sub-optimal gap predictor may result in model exploitation (Kurutach et al., 2018; Janner et al., 2019; Rajeswaran et al., 2020), yielding sub-optimal results.