Diffusion Self-Weighted Guidance for Offline Reinforcement Learning
Authors: Augusto Tagle
Javier Ruiz-del-Solar
Felipe Tobar
Venue: Accepted by TMLR ()

Abstract: Offline reinforcement learning (RL) recovers the optimal policy $\pi$ given historical observations of an agent. In practice, $\pi$ is modeled as a weighted version of the agent's behavior policy $\mu$, using a weight function $w$ working as a critic of the agent's behavior. Although recent approaches to offline RL based on diffusion models (DM) have exhibited promising results, they require training a separate guidance network to compute the required scores, which is challenging due to their dependence on the unknown $w$. In this work, we construct a diffusion model over both the actions and the weights, to explore a more streamlined DM-based approach to offline RL. With the proposed setting, the required scores are directly obtained from the diffusion model without learning additional networks. Our main conceptual contribution is a novel exact guidance method, where guidance comes from the same diffusion model; therefore, our proposal is termed Self-Weighted Guidance (SWG). Through an experimental proof of concept for SWG, we show that the proposed method i) generates samples from the desired distribution on toy examples, ii) performs competitively against state-of-the-art methods on D4RL when using resampling, and iii) exhibits robustness and scalability via ablation studies.

Published in Transactions on Machine Learning Research (12/2025)
Diffusion Self-Weighted Guidance for Offline Reinforcement Learning
Augusto Tagle
augustotageemontess@aaaaoom
Initiative for Data 8 AI, Universidad de Chile
Javier Ruiz-del-Solar
jruizd@ing. uchiile..
AMTC & Dept. of Electrical Eng., Universidad de Chile
Felipe Tobar
f.tobar@iiperial.coo ac.uk
Deportment of Mathematics, Imperial College London
Reviewed on OpenReview: https: // opwnreviee...ee.eemmm net/ forum?? id- jnXBnpmanv
Abstract
Offline reinforcement learning (RI) recovers the optimal policy и given historical observations of an agent. In practice, π is modeled a weighted version of the agent's behavior policy critic of the agent's behavior. Although recent
μ,, using weight function w working approaches to offline RL based diffusion models (DM) have exhibited promising results, they require training a separate guidance network to compute the required scores, which is challenging due to their dependence on the unknown w. In this work, we construct a diffusion model over both the actions and the weights, to explore a more streamlined DM-based approach to offline RL. With the proposed setting, the required scores are directly obtained from the diffusion model without learning additional networks. Our main uanceptual contribution is novel exact guidance method, where guidance comes from the same diffusion model; therefore, Through an experimental proof of concept for SWG, we show that the proposed method i) generates samples from the desired distribution on toy examples, ii) performs competitively against state-of-tho-art methods on D4RL when using resampling, and iii) exhibits robustness and scalability via ablation studies.
proposal is termed Self- Weighted Guidance (SWG).
1 Introduction
Offline reinforcement learning (RL) learns a policy from dataset of an agent's collected interactions. This setting is commouplace in scenarios such as robotics and healthcare, where real-world exploration with suboptimal agents is costly, dangerous, and thus unfcasible (Levine et al., 2020).
A iralical issue when istaiigg the state-action value function ((s. a) (Q-values) in offline RL. is the unknown effects of the actions absent in the dataset, referred to out-of-sample actions. In practice, methods such as (Kostrikov et al., 2022; Xu et al., 2023; Hanson-Estruch et al., 2023) refrain from qucrying out-of-sample actions during training, thus ensuring that the learned policy remains close to the dataset policy, and avoiding distributional shift (Kumar et al., 2019; Fujimoto et al., 2019; Nair et al., 2020). These methods typically adopt weighted behavior policy form given by
*(ass) x ((als)u((ss))
(1)
where (s, a) the state-action pair, r(ass) is the target policy, μ(aas)) is the dataset or behawior policy, and w(s, a) is the weight function encoding the value of an action given a state.
Previous approaches learn T[ass) in weighted regression procodure (Peters Schaal, 2007: Peng al., 2019; Nair al., 2020; Wang et al., 2020), using, e.g., Gaussian model (Kostrikov et al., 2022); this

---

Published in Transactions on Machine Learning Research (12/2025)
approach is severely limited by the expressiveness of the chosen model. For instance, when the dataset contains multimodal actions, unimodal policies will certainly fail to capture the agent's behavior successfully (Hansen-Estruch et al., 2023; Mao et al., 2024).
Diffusion models (DM) (Sohl-Dicksee Dickstiicsttin et al., 2015; Ho et al., 2020; Song et al., 2021) have been successfully applied to sampling from the unnormalized target policy equation (Janner et al., 2022; Chen et al., 2023; Hansen-Estruch et al., 2023; Lu et al., 2023; Mao al., 2024), while modeling a potentially multi-modal policy (Wang et al., 2022; Chi et al., 2025). This has been achieved by approximating the behavior policy using a DM, ie., μe (a)s) x u(als), learning the weight function using external models (see Secs. 2.1 and 2.2), and then constructing procedure to emulate samples from T(als). The main challenge of this approach is the estimation of the intractable score, where noisy actions and weights are entangled. This is typically addressed by learning an auxiliary model, referred to as the guidance network (Lu et al., 2023).
In this work, we propose a fundamentally different approach diffusion-moddl-asseee offline RL. In addition to sampling actions from the behavior policy μ(a)s) (s), we also sample the weights w(s, a) using the same DM. As shown in Sec. 3, this enables sampling procedure that effectively generates actions from the target policy Tal s). In contrast to previous approaches that rely on the training of two distinct models DM and an external guidance network- our method climinates the need for an external guidance network by embedding the guidance information within the diffusion model itself. We refer to this central contribution as Self- Weighted Guidance (SWG), detailed Sec.3.
The main objective of this work is to introduce SWG, DM over actions and weights for offline RL, and validate its feasibility. Experimentally, we expect competitive implementation of SWG which performs par with the statc-of-the-art. In particular, our contributions summarized as follows.
The derivation of the SWG method for exact guidance (Prop. 3.3 and Thm. 3.5 in Sec. 3).
An implementttton setup of weights models tailored for SWG (Sec. 5).
A proof of concept for SWG's exact guidance capabilities in toy experiments (Sec. 6.1).
An experimental validation of SWG in D4RL's challenging environments, where exhibits competitive performance among recent methods and the ability to leverage a resampling stage (Sec. 6.2).
Ablation studies exploring SWG's robustness and scaling properties (Secs. 6.3, 6.4 and 6.5).
2
Background
2.1 Offline reinforcement learning
Consider an agent that, following an unknown behavior policy p(als), interacted with the environment generating samples (s, a, s') recorded in dataset Dr- The goal of offiine RL is to estimate the optimal policy πt (als)) from Dμ- Modeling the target policy as a weighted behavior policy in equation 1 is commonplace across various offline RT. methods, some of which are presented next.
Constrained policy optimization (Peters et al., 2010; Peng et al., 2019; Nair et al., 2020) regularizos the learned policy to ensure that it remains close to the behavior policy by:
n (als) arg max EAWDD [Eaa-xtala Q(5, D D
(2)
T
where Q(s, a) is the state-action function that indicates the quality of an action given a state, in terms of the expected future discounted rewards under specific policy.
Following (Peters et al., 2010; Peng al., 2019; Nair et al., 2020), it can be shown that the optimal policy T²" (als) in equation 2 has weighted structure as in equation 1, given by:
t *(aas) () ulaa ((aass)e ((G.a)
2

---

Published in Transactions on Machine Learning Research (12/2025)
where Q(s, a) represents the Q-values function of the optimal policy, Note that estimating the target policy Q-values requires querying out-of-sample actions.
Implicit Q-learning (IQL) (Kostrikov et al., 2022) avoids querying out-of-sample actions using an in-sample loss based on expectile regression. In this way, defines an implicit policy that outperforms the behavior policy. Furthermore, IQL employs ad wddaatage-weighted regression (Peters Schaal, 2007; Wang et al., 2018; Peng et al., 2019; Nair et al., 2020) to extract the policy, which can also be interpreted as weighted behavior policy, given by:
(als) p(a)s)e
where A(s,a) - Q(s V(s) is the advantage function learned with expectile regression.
Implicit Diffusion Q-learning (IDQL) (Hansen-Estruch Estruch et al., 2023) generalizes implicit formulations to the use of an arbitrarily convex function f:
(*) arg min Bn-a(a)s) [f(Q(s, -V(s)). V(s)
thus defining a weighted behavior implicit policy given by
Timp (als) ((aas)u((s,a)
-V Furthermore, IQLL explores different convex functions f. such as expectiles where w(s, a) = -(*=a)
(Kostrikov al., 2022), quantiles (Koenker & Hallock, 2001) and exponential functions.
2.2 Diffusion models in offline RL
Diffusion models (DM) approximate an unknown data distribution' (u())) using a dataset D by defining a sequence of & E {O...., K}, E N, noisy distributions defined as follows:
qu(ax ~ N(ay ag of.),
where og and 0% are predefined noise schedules, s.t., ax ~ (O, as K + oo. Therefore, samples from can be obtained by first sampling from (0, and then solving the reverse diffussin process. In practice, the reverse diffusion approximated by model co that predicts the noise be removed from a noisy sample ak ~ 9k, produce target sample ag ~ 40- The training loss for eg is (Ho et al., 2020)
arg min &o cp Sax k))l
(3)
where as qo: co N I), ~~ K) and a = agao + ogEo. The noise is related to the score function (Song et al. 2021) of the intermediate noisy distribution as follows:
E0 Va, log ok (ax)-
σk
Score function
Our motivation for employing diffusion models (DMs) in offline RL stems from two main factors: (1) DMs are highly expressive compared to classical generative models (o.g., Mixture Models), allowing them to model multi-modal action distributions (Chi al., 2025); and (2) their score-based formulation enables flexible sampling from conditioned weighted distributions via guidance. In particular, the weighted behavior in equation 1 can be naturally incorporated using DMs, making them well suited for offline RL.
To condition the DM on the states, the noise prediction network can be made an explicit function of the state, that is, e(ax, kt) = <(a k s). Previous approaches fit a DM (ala)) to the behavior policy μ(a learn the weights w(s a) as described in Sec. 2.1, and then sample from the target policy via resampling or guidance; these are explained as follows.
We denote data with the symbol (instead the standard 2) consistency, since in our setting the data the agent's 's actions.
3

---

Published in Transactions on Machine Learning Research (12/2025)
Resampling involves first sampling batch of actions from the learned behavior policy [o(a)s), and then resampling them using weights as probabilities of a categorical distribution (Hansen- Fattrch et al., 2023; Chen et al., 2023), in a way inspired by importance sampling. Alternatively, one can adopt greedy policy that simply selects the action that maximizes the weight with the aim of improving performance.
Guidance aims obtain samples from the target policy by modifying the diffusion, adding a score term approximated by an external guidance network go (Janner al., 2022; Lu al., 2023; Mao et al., 2024). The k-step score in diffusion sampling procedure is given by:
V ay log Th (as (8) R Vay aup
log Po (ak, k 8) + Va, Io (ak: ki 8),
(ax
(4)
target score
5o (..)
guidance score
where [o are the noisy versions of the actions, with a being the true noise-free action. As shown by (Lu et al., 2023), an unbiased estimate of the guidance score that effectively produces samples from the target distribution should satisfy the following:
a. Do (ak; ki Vns log Eytso () s]],
(5)
where the right-hand term is intractable for most policies and weight functions. Guidance methods that are unbiased towards this objective are referred to as ezact guidance methods.
Diffuser (Janner et al., 2022), trains the guidance network with mean-squared-erroo objective with respect to Q-values, which produces biased target policy, in the sense that it does not match the form in equation 5. Conversely, exact guidance methods such Contrastive Energy Prediction (CEP) (Lu et al., 2023) use contrastive learning objective that requires querying batch of out-of-sample actions for each state in the datasct. Lastly, D-DICE (Mao et al., 2024), proposes an in-sample loss to learn the guidance network and uses & quide-then--eel
approach to oblain samples.
Remark 2.1. While exact guidance methods effectively produce samples from the target distribution, they require learning an external network 9a through intricate loss function (g))). This motivales our proposal Self- Weighted Guidance, which leverages DM's without resorting to special training objective to conduct exact guidance.
3 Self welg-weighted guidance
3.1 Problem statement
Let us consider datasct DA (a0) C Rd of i.i.d. realizations of a random variable A ~ q(a); these will be the agent's 's actions. Recall that we aim to sample from the unnormalized target p(-) that relates to e(-) as
pla) x q(a)w(a).
where w. : Rd + IR is known" weight function. Sampling can be achieved by fitting a DM, denoted cø, to ((a) trained on DA Then, in the k-th denoising step, the score of the target distribution (referred to as the target score) satisfies:
V au log pk (at)
V au log (ax) V a. log E Eytoo (aw) [w(av)],
target
(g(m).))) 2
intractable score
where ag DA is a data sample, aj...., aK are noisy copies of the data on which the DM is trained, and the subindex in the distributions denotes the denoising step. Although Va, log qu (au), referred to as the intermediate score, is provided directly by the DM, estimating the target score in equation 7 is challenging due the intractability of the right-hand term in the general case (Tuu et al., 2023).
Thhough in offline RL the weight function or has to learned, we consider known for now.
4

---

Published in Transactions on Machine Learning Research (12/2025)
3.2 Proposal: diffusing over actions and weights
Let us consider a random variable (RV) Z = (A, W) given by the concatenation of A, describing the actions in the previous scction, and W, which given by passing through the weight function w(-). Since w(-) is a deterministic function of A, the distribution of Z is given by the push-forward measure of the distribution of A through the mapping T' + (a, w(a)). With slight abuse of notation, denote the push-forward measures corresponding to p and in equation 6 with the same symbols, i.e. p(z) Tœp(a) and g(z) ( respectively. Therefore, we have
p(z) p(a, w) p(wla)p(a) p(a))(())))))))) p(a)dw=w(a) ((z) q(a, w) ((wa))g(a) q(a))w=w(aa)
which, combined with equation 6, give p(z) x g(z)w. In this new formulation, w no longer to be considered as function of a, but rather as a variable for which there are available observations. For clarity of presentation, define the following function
Definition 3.1 (Extraction function). The function on.. Rd+1 + I extracts the weight component from the concatenated variable z = =[a, That is, ow [a, w] + ( w|) w.
Remark 3.2. The extraction function allows us to express
p(z) x (())
(8)
thus implying that the target distribution for follows the same structure as the original distribution equation This motivates jointly learning weights and actions with the same DM. Since the extraction function (2) is linear, it is feasible to estimate the (otherwise intractable) score within DM.
We can now implement a DM to sample from the augmented variable Z. To this end, we first extend the dataset D (((() ) to Dz = (( (()) C [Rd+1 where w(()) = w(a(()) is a weight observation. Notice that the dataset Dz provides access to the weight function only on the support of the original data in DA. For consistency, we also denote the realizations of Z as ( = =a(1),())) (()).
Therefore, we define K-step diffusson with noise schedule {ax] [o and train noise- predictioiition DM Eo for Z using the augmented dataset Dz, minimizing the standard diffusion loss in equation 3. In notation, this is
(( = Ex 1l co 13)
(9)
where 2g ~ D2, 2 = 2g + toco 6a ~ NO I), and ~ U(1, (). Algorithm doscríbes our proposed DM training procedure. We emphasize that, in practice, Algorithm 1 is implemented with the expectile weight, function.
Algorithm Training of joint. DM over weights and actions
Input: Dataset DA: weight function w(-), number diffusion sleps K, noise schedule ( for 1K learning rate X, and batch size B.
Initialize: parameters for the diffusion model co Augment. dataset D - Dx as described in Sec. 3.2. for each parameters update do
Draw B samples 2o ~ Dz and B indices & ~ u(1 Add noise to each sampled data Zu for respective & time as 2 ag + OkCo, 6p ~ N(0.I)
Calculate loss I.(0)= B ΣB l|so - c0 k)lB
Update parameters 0 +- (V) end for
Return: Trained diffusion model cœ
5

---

Published in Transactions on Machine Learning Research (12/2025)
3.3 Augmented exact guidance
Proposition 3.3. The required score to sample from the turget distribution in equation is given by
Vz (zz)) Vz log qu (z()
Vux log E (g(zaln)) 1 (zu)]
(10)
splega62 PL
intractable term
The proof for Proposition 3.3 follows as an extension of (Lu et al., 2023) to our setting (see Appendix A.1).
Remark 3.4. Since the extended formulation in equation is defined over z, the gradients required implement guidance in equation 10 are taken with respect to zy. rather than aę (as in equation 7). This is a desired feature, since the actions and weights are highly coupled, and thus guiding them separately can lead to inefficient sampling.
The key insight our proposal is that the intractable term in equation 10 can be recovered directly from the DM described above using the so-called data prediction formula (Efron, 2011; Kingma al., 2021) given by:
Eeian 2.) [o] Rs za(Zz, k) Zi: JKEB OKe (Zk,
(11)
O
Using equation 11, and since the extraction function (-) is linear, the target score Vz, log ps (Zz) in equation 10 can be computed with the proposed DM, as stated in Theorem 3.5 (proof in Appendix A.2). Theorem 3.5. Let (z k) be the DM found the global minimizer of equation The exact target score of equation 10 is given by:
2k C2 C (Zkk k:)
V 22 logp)(Zz) Uzi log (2x) V log Φw
ag
$ k))
self guidonce
where recall that (z) = simply extracts the weight component from = [a,w] w/ (Def. 3.1).
Our method is referred to as Self- Weighted Guidance (SWG), since the DM samples from the target distribution Zo ~ p(z)) by guiding itself, that is, the guidance signal is a function of the same model (see Algorithm 2). SWG provides an clegant way of performing exact guidance, since it computes gradients directly on the DM eg instead of external guidance network with dedicated aadd possibly intricate loss function (see Sec. 4). This translates to not having to train separate guidance network, thereby simplifying the training pipeline. Sec. 4 provides a theoretical comparison between SWG and other DM-based guidance approaches to offline RL, the results of which are summarized in Table 1.
Algorithm Weighted Guidance sampling
Input: Pretrained DM cØ, number of diffusion steps K, and noise schedule (au) { Sample noise 2 ~ ~O( for cach in reversed 0: K do
Forward pass on diffusion model ep (Zk, k) :: (z k) - a V., log (dwr( (". co_ccc(kk Calculate Zk-1 with noise prediction Set 24 = 2
at
end for
Return: Sample zo
4 Relationship to previous guidance methods in offline RL
In this section, we provide an analysis of previous guidance methods alongside our own, highlighting theoretical differences and potential advantages. The two primary criteria for comparing previous methods with SWG

---

Published in Transactions on Machine Learning Research (12/2025)
are: i) whether the method provides exact guidance, and fi) whether it relies external guidance network, as summarized in Table 1.
Table 1: Summary of DM-based guidance approaches to offline RL.
Method
Exact guidance
Requires External
Loss function
Guidance Network
Diffuser (Janner et al., 2022)
No
Yes
Guidance network MSE over Q-values
CEP (Lu et al., 2023)
Yes
Yes
Contrastive loss (see equation 12)
D-DICE (Mao et al., 2024)
Yes
Yes
In-sample loss (see equation 13)
SWG (ours)
Yes
Diffusion loss (see equation 14)
SWG (Ours)
Diffusion Model
Guidance Network
g
Diffusion Model
Diffusion Model
Guidance Network
ZKC
20
40
Figure 1: Comparison between the proposed SWG approach (left) and the standard Diffusion Model with a separate Guidance Network (right). SWG jointly learns actions and weights within a single diffusion model, whereas standard approaches in diffusion for RL trains an additional guidance network to steer the diffusion model. The figure represents one forward pass to the diffusson model during training. Recall that zo |o, wol.
Let us consider the two existing exact-guidance DM methods in the offline RL literature: CEP (Lu al., 2023) and D-DICE (Mao et al., 2024). Assuming a known weight function w(-), CEP learns the guidance network go using the loss
M Σ
(CEP( = EE (1:M)
(s, a
log
Pa.k
(12)
Go c A.A 25=1
Ar
(1:ad) (a s.b.,j E where ls ~ U(11 K), and for simplicity of notation we denote 98.k go a k) and a (0, (1....,M)) M]). Recall that ap the data and a are the noisy copies. Since equation 12 requires M >> action samples from each state in the dataset, its evaluation is likely to require out-of-sample actions, thus possibly leading overestimation errors (Kostrikov et al., 2022; Mao et al., 2024). CEP (Lu et al., 2023) addresses this by pretraining a DM on the behavior policy and generating synthetic action dataset. containing M > 1 samples for cach state in the datasct.
(6)
In contrast, D-DICE proposes a loss that circumvents the need to query oft-of-sample actions through
DD-DICE(O) = Ex m..k w(s, 20)e (k + (s, a k.)
(13)
where & ~ U11 K), ag D is action sample for the state 8, and a is the respective noisy version of ag-

---

Published in Transactions on Machine Learning Research (12/2025)
Although the objectives in equation 12 and equation 13 yield unbiased estimates of the intractable term in equation 5, their formulations introduce potential training instabilities arising from the exponential factor e9t To address this, the authors of these loss functions proposed stabilization techniques: CEP employs self-normalization of the exponential term, whereas D-DICE alleviates the issuc by clipping the exponential term. Furthermore, we contend that these loss functions are either lacking in direct interpretability, as in the case of D-DICE, or overly intricate, as in the case of CEP.
The proposed SWG method, conversely, is trained with the standard diffusion loss
Lswa = E2 1150 eø(2k, k)
(14)
which is equivalent to maximizing a variational lower bound on the likelihood (Kingma et al., 2021). That is, a simple MSE over predicted noise terms free from exponential terms prone to instabilities, thereby easing the method's implementation. Another relevant feature of SWG is that it inherently defines in-sample loss, thus addressing the out-of-sample problem with the CEP objective.
Also, since SWG produces action and weight samples Zy = = wo] simultaneously, it is possible to define a resampling mechanism as described in Sec 2.2 to further improve performance. In Section 6, where our experimental results are presented, we include a variant of SWG called SWG-R, which uses resampling. These results demonstrate that SWG-R exhibits superior performance in comparison to the vanilla (non-resampling) SWG method.
5 Implementing SWG in offline RL
There are two main practical aspects regarding the implementation of SWG in the offline RL sctup. First, recall that, for notational and conceptual simplicity, the SWG was presented in way independent of the states. In practice, however, state-dependent DM's are trained by considering the state as an input to the noise predictor network, that is, <(zz, k) = e(zu, k, s) in equation 9. Second, SWG was introduced in Sec. 3 assuming a known weight function w, however, in offline RL the weights need to be estimated beforehand. We next present how to adapt and implement different weight. formulations within the proposed SWG.
al., 2023) 7-expectile weight is defined as ( a) Jr ((Q(=))<<V())) |: where (s(s) is the value function learned with the expectile loss, and 1 is the indicator function. Since this weight takes only two possible values, (s, a) {7,11-1 c [0, 1]. it may fail
Expectile. The IDQL (Hansen-Esttuch
to distinguish between similar actions by assigning them identical weights. To address this limitation, we propose smooth variant of these weights, given by
w(s.a a) (1
1 (1 (1e-A(ss.)) - ++ (1 )
1+E -(A(aa)) (1
(1-r). r).
(15)
Here, we apply a sigmoid function to the advantage A(s,a) a) to obtain a continuous weight function. This weight function is used in our D4RL experiments (Sec. 6.2), and the full training procedure for the value functions is detailed in Algorithm 3 in Appendix B.2.
Exponential. From the same value functions learned with expectile loss, we can define an exponential weight model derived from the IQL (Kostrikov et al., 2022) policy extraction method w(s a) = @A(s,)) where A(s, a) = Qs,, a) - V(s) is the advantage function. Though this formulation addresses the sparsity issue arising in the expectile weight, its exponential form could lead to arbitrarily large values, and is thus potentially unstable during training of SWG. Using the advantage function can help mitigate instabilities, as the value function serves as a normalizer for the Q-values.
Linex Another exponential formulation can be obtained when learning the value function using the linex function lx u exp(u) As found by (Hansen-Fstruch et al., 2023), this defines the following weight:
Wexp (s, a) al exp (a(((s, - Vexp Vex) -1|
(16)
Q((s, a) - Vex
where Vsxp(s) is the value function learned by using the linex loss, and is temperature coefficient.
8

---

Published in Transactions on Machine Learning Research (12/2025)
We emphasize that, among the various weighted behavior policies in the literature, we focus avoiding querying out-of-sample actions, as they can lead to overestimation errors (Kostrikov et al., 2022). Sec. 6.3 provides an experimental comparison of different weight formulations within SWG on D4RL. More details on the implementation and training on the learning Q(s, a) and V(s) are provided in Appendix B.2.
The standard practice across most offline RL methods is to estimate Q(s, a) and V(s) to then construct the weight function; this adds complexity to the overall training pipeline. We clarify that though SWG does not require an external guidance network during sampling, it still requires learning of the weight function prior to sampling.
Remark 5.1.
In our implementation, employ two techniques to refine the action generation process. First, control the strength of the guidance score using hyperparameter p, termed guidance scale (Dhariwal & Nichol, 2021), follows eg(Zk, k) pa Z2 log (# (*= Second, we introduce SWG-R, resampling variant of SWG, which resamples a candidate batch of M actions, denoted as {am m= and selects the action a" associated with the maximum weight w.
6 Experiments
The proposed SWG was implemented and validated through a set of experiments targeting the following questions:
1. Can SWG sample from the target distribution? (Sec. 6.1)
How does SWG (with and without resampling) perform against the state-of-the-art in the D4RL benchmark? (Sec. 6.2)
3. How do different weight formulations affect SWG's performance? (Sec. 6.3)
4. How does the guidance scale affect SWG's performance? (Sec. 6.4)
How does SWG's inference time scale with increasing model size? (Sec. 6.5)
Additional experimental details and benchmarks are given in Appendix B, with the full computational resources used in Appendix B.4. Our code is available at SWG repository.
6.1 Sampling from the target distribution
We considered the toy experiments from (Lu et al., 2023), which use weight function w(a) exp((33((a) where {(a) is an energy function, and 3 is the inverse temperature coefficient. Figures 2 and show SWG samples from a spiral and an -Gaussian-mixtuus distributions, respectively, and offer a qualitative assessment of its performance. In both cases, samples closely matching the ground truth for different values of B suggest that SWG was able to sample from the target distributions, thus learning the relationship between the data distribution and the weights in both cases. The reader is referred the Appendix B.1 for further details and more examples with different target distributions.

---

Published in Transactions on Machine Learning Research (12/2025)
B 0
8
=
B 10
B 15
SWG (ours)
E(a)
Ground truth
Figure 2: SWG applicd to a spiral distribution for different temperature coofficients B. Top: samples generatod by SWG. Bottom: ground truth samples. When 0, we have w(a) and thus the target distribution matches the data distribution.
==0
B
B=
B 10
B == 15
SWG (ours)
((a)
Ground truth
Figure 3: SWG applied to G-Gaussian mixture distribution for different temperature coefficients B. Top: samples generated by SWG. Bottom: ground truth samples. When B = 0, we have w(a) 1 and thus the target distribution matches the data distribution.
6.2 D4RL experiments
Using the D4RL benchmark (Fu et al., 2021), we implemented two variants of the proposed method: SWG as described above, and SWG-R, which includes action resampling on top of our method. We considered the following methods as benchmarks:
Classical offline RL: CQL (Kumar et al., 2020) and IQL (Kostrikov al., 2022)
Diffusion-based resampling: SIBC (Chen et al., 2023) and IDQL, (Hansen-Fstruch el al., 2023)
Diffusion-guided: Diffuser (Janner et al., 2022), D-DICE (Mao et al., 2024), and QGPO (CEP) (Lu al., 2023)
D-QL (Wang et al., 2022), which trains DM using Q-values regularization term
Furthermore, our experiments are performed across four task groups:
Locomotion: the main ofline RL benchmark, which features moving an agent a plane.
2. Ant Maze: more challenging task, where an ant agent is guided through maze to reach target.
Franka Kitchen: involves goal-conditioned manipulation in a realistic kitchen environment.
4. Adroit Pen: focuses high-dimensional dexterous manipulation reorient a pen.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: SWG against the state-of-art on the D4RL benchmark: Mean and standard deviation of the normalized returns (Fu et al., 2021) under random sceds. Values within 5% of the best performing method in each task (row) are in bold font. SWG and SWG-R correspond to the standard and resampling version of our method, respectively. We used acronyms ME; Medium -ecbbur,E M: Medium, MR: Medium-Replan Avg: Average, HC: HaliCheetah, Hopp: Hopper, W2D: Walker2D, Kitchen: Franka Kitchen, Pen: Adroit Pen. See Appendix B.2 for full experimental details.
Dataset
Env
CQL
IQL
SIBC
IDQL
Diffuser
D-QL
D-DICE
QGPO
SWG
SWG-R
ME
HC
62.4
86.7
92.6
95.9
79.8
96.8
97.3
93.5
92.8
0.6
93.5 + 0.5
ME
Hopp
108.6
108.6
107.2
108.0
ME
W2D
111.0
109.6
109.8
112.7
108.4
110.1
114.1
110.7
110.7 1: 1.7
112.5 0.3
M
HC
44.4
47.4
45.9
51.0
44.2
51.1
60.0
54.1
46.9 0.2
46.9
M
Hopp
58.0
66.3
57.1
65.4
58.5
90.5
100.2
98.0
68.6
74.8 1.61552
M
W2D
77.9
82.5
74.0
MR
HC
46.2
44.2
37.1
45.9
42.2
47.5
49.2
47 6
42.5
42.5 1.5
MR
Hopp
48.6
94.7
86.2
92.1
96.8
101.3
102.3
96.9
73.9 + 2.3
73.9 2.3
MIL
W2D
26.7
73.9
65.1
85.1
61.2
95.5
90.8
84.4
4.1 60.6
79.9 1: 4.5
Avg (Loc)
63.9
76.9
75.6
82.1
75.3
88.0
90.6
86.6
75.5
78.8
Default Diverse
Umaze Umaze
74.0
87.5
92.0
94.0
93.4
98.1
96.4 4
92.1 :2./1:2 2.6
98.3 + 1.7
80.2
66.2
74.4
71.7 8.3
Play
Med Med
61.2
71.2
81.3
84.5
76.6
91.3
83.6
89.6
2.3
90.0
2.9
Diverse
53.7
70.0
82.0
84.8
78.6
85.7
83.8
88.2
5.2
91.7 1.7
Play
Large Large
15.8
59.3
63.5
46
66
55.4
66.7
Diverse
14.9
47.5
45.5
67.9
56.6
72.0
64.8
55.2 1
63.3 1: 8.3
Avg (AutMaze)
50.6
63.0
74.2
79.1
69.6
83
78.3
75.1
80.3
Human
Pen
37.5
71.5
76
72.8
81.4
72.8
78.5 6.3
103 0 5.0
Cloned
Pen
39.2
37.3
64
57.3
83.8
54.2
62.3 6.7
91.7 10.5
Avg (Adroit Pen)
38.4
54.4
70
65.1
84.1
63.5
70.4
97.4
Partial
Kitchen
49.8
46.3
17.9
60.5
78.3
69.0 4.9
73.3 1.4
Mixed
51.0
45.4
62.6
61.0 2.2
65.0 +
Complete
Kitchen
43.8
62.5
77.9
84.0
72.3 2.6
72.3 2.6
Avg (Kitchen)
48.2
53.3
57.1
69.0
67.4
70.2
These experiments were conducted with the expectile weight proposed TDQL. (Hansen-Fstruch et al., 2023) modified detailed in Sec. and using K 15 diffusion steps. See Appendix B.2 for more details.
The results shown in Table 2 reveal that the resampled version of our proposal SWG-R provided competitive performance over state-of-the-art DMs in the challenging environment tasks from Franka Kitchen and Adroit Pen. In Ant Maze, SWG-R also reported very competitive figures, outperforming all methods except D-DICE. However, in locomotion tasks, both SWG and SWG-R achieved average performance compared to previous methods. Also, note that SWG outperformed classical offline RT methods such as CQL and IQL. most. tasks.
Based on the D4RL results, argue that SWG-R offers tradeoff between having simpler training setup with fewer components than current DM guidance- based approaches at the cost of lower performance with respect to the top-performing method, D-DICE (Mao et al., 2024), for the particular case of the Locomotion dataset. Moreover, incorporating resampling was pivotal for the enhanced performance of SWG-R over SWG. This addition strengthened the effectiveness our framework, similarly to previous approaches. However, it is worth noting that the performance gain of SWG-R comes at the cost of an expanded hyperparameter scarch space during inference (see Appendix B.2). Additionally, we remark that the resampling was performed using the separate weight model w (i.e., expectile weights).
11

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Ablation study for different weights models implemented alongside SWG on the locomotion D4RL benchmark.
Expectile
Exp-expectile
Linex
Average (locomolion)
75.5
69.9
63.6
6.3 Ablation study for different weights
We then evaluated SWG's performance under different weight formulations in Sec. 5. This lest was conducted on D4RL locomotion, using the same model architecture and hyperparameter tuning to ensure a fair comparison. Furthermore, we also tuned the guidance scale in each case to find the best possible performance. The weight formulations were implemented as follows:
Expectile weight: Learnt with the same setup adopted in the main results, described in Sec. 5.
Exponential expectile weight: We used the same setup as for IDQL expectile weights, and considered a temperature 3 = 3 to exponentiate the advantage function @A(ss)) To ensure stability during training, we clipped the exponential weights (0, 80).
Linex weight: We followed IDQL (Hanson-Estruch et al., 2023), and used 1.
The results, shown in Table 3, reveal that expectile yielded the best performance, with the exponential expectile showing moderate performance, and lastly Linex performing substantially worse. We conjecture that this is due to the training instability introduced by the exponential weights, in general. Most importantly, these results consistent with the findings reported in IDQL (Hannen-Fsttuuh et al., 2023).
6.4 Ablation over guidance scale sensitivity
We also studied the effect of using fixed guidance scale across all tasks, without tuning p for each task individually. Recall that the choice of the guidance scale P controls the strength of the guidance score, which translates into sharper target distribution concentrated around high-value weights. This effect can be seen in the modified noise prediction é := co(2,, k)) par Z., log ( (*t-ano((x_d))) 0% Ve, log ((== Sec. 6.3, yicld superior performance.
ea (2 k))
We conduct these experiments using expectile weights, which, as shown in
Overall, from Tables and 5, we observe that the best performance was generally achieved with 10 or p == 15. As expected, using low guidance scales (p < 5) resulted in suboptimal performance, as the guidance signal was not sufficiently strong. On the other hand, employing large guidance scales (p 20) led poorer performance, since overly strong guidance signal tends to produce out-of-distribution samples.
Table 4: Guidance scale p sensitivity Locomotion tasks.
P
1
10
15
20
25
Average
63.8
72.0
72.9
73.1
71.2
69.2
68.8
Table 5: Guidance scale p sensitivity in Ant. Maze tasks.
0
1
5
10
15
20
25
30
Average
24.6
37.9
69.6
68.0
67.3
61.8
57.8
12

---

Published in Transactions on Machine Learning Research (12/2025)
6.5 How does SWG's inference time scale with its size?
Since SWG requires computing gradients through DM, investigated its scalability properties through two experiments examining both depth and width scaling of its networks. For depth scaling, we used a simple multilayer perceptron (MLP) with a hidden dimension of 2056 and measured the inference time across varying numbers of layers. For width scaling, we used a fixed 4-layer MLP with varying hidden dimension, measuring the average inference time accordingly (see the Appendix B.3).
As shown in Figures and 5, SWG exhibited linear scaling with respect to model depth and exponential scaling with respect to the width scaling (hidden dimension). Additionally, and expected, sampling with guidance (i.e., computing gradients) translates into larger inference times. However, qualitatively, we observe that both sampling methods scale at a similar rate. Note that scaling issues related to gradient computation can be easily avoided by designing the DM to output the weight. component in its early layers. This allows gradients to be computed with respect to a fixed module, without requiring backpropagation through the entire model. Consequently, the additional computational overhead SWG remains constant, even the nub-component of the model responsible for action generation increases scale.
The training time remains essentially the same that of the related methods (Hansen-Fstruch et al., 2023; Mao et al., 2024) when using similar architecture. The SWG training time measurements are reported in the Appendix B.4.
Effect of Number of layers on Inference Time -3
E Effect of Hidden dimension on Inference Time
x10
SWC
3.0
SWG
Inference time (5)
SWG W/o guldance
SWG W/o guidance
2.5
Inference time
1.5
1.0
3
6
0
1000
2000 3000 4000 5000 6000 7000 8000
Number of layers
Hidden dimension
Figure 4: SWG depth scaling: inference time (s) vs number of layers (mean 2 standard devia- tions over 10,000 runs).
Figure 5: SWG width scaling: inference time (s) hidden dimension (mean + standard devia- tions over 10,000 runs).
7 Conclusion
We have proposed joint DM over actions and weights for optimal policy discovery in offline RL. Unlike current DM-based methods, our approach avoids the computation of guidance score network. Tn our proposal, the computation of the scores is straightforward, as they are readily available from the DM implementation. The proposed method, termed Self- Weighted Guidance (SWG), owes its name to having the weight guidance information embedded into the DM. Compared to previous methods, SWG conducts exact guidance using a strcamlined sctup which requires single DM, and provides novel insights arising from modcling weights as random variables, thus demonstrating one more time the flexible modeling capabilities of DM's.
Via toy examples and the standard D4RL benchmark, have provided experimental validation of SWG's ability to: i) effectively produce samples of the target distribution, and ii) perform on par with -or better than the state-offthe-arl DMs on challenging environments such as Ant Maze, Franka Kitchen and Adroit Pen, while having a simpler training setup. We have also validated the ability SWG to leverage resampling for improved performance. However, we observe that, in general, SWG performs sub optimally on mixed- action datasets (i.e., Mixed, Medium-Replant Diverse), which is likely due to either the choice of weight function and/or the architecture of the DM, both of which may limit how effectively the joint action-wed weight.
13

---

Published in Transactions on Machine Learning Research (12/2025)
distribution is learned. Furthermore, we have provided an ablation study for different weight formulations within SWG, and also studied how SWG's s inference time scales with its network's s width and depth. We stress that the main advantage of SWG comes from its simplicity and reduced training setup compared to previous guidance methods.
Lastly, we claim that the proposed DM-based sampling methodology, which guided by the same DM, has the potential for applications beyond the offline RL setting. Tasks involving weighted distributions, such controllable image and text generation (Dhariwal & Nichol, 2021; Li et al., 2022), can be addressed by the proposed SWG approach.
Limitations and future work. We have provided a proof of concept for SWG, which performed among the top 2 methods across D4RL's datasets except in Locomotion, where SWG was outperformed by QGPO, D-QL and D-DICE. To address intrinsic limitations, we note that jointly learning actions and weights reduces flexibility (changing the guidance requires retraining the entire diffusion model) and makes training more challenging compared to learning actions alone, potentially needing higher-capacity networks. Both issues could be mitigated by dedicating a portion of the network exclusively learning weights. Future work could further improve SWG with tailored weight models and more sophisticated ODE solvers.
Broader impacts. This work presents promising methodology for offline RL, where policics can be learned from controlled offline dataset. However, with most diffusson models, potential misuse includes the generation of harmful data, such as fake images, videos, or audio.
Acknowledgements JRdS acknowledges financial support from ANID-Chile grants Basal CIA250010 and Fondecyt-Regular 1251823.
References
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. wrXiu:107.06444 2016.
arXiv preprint
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao Zhang, JAX: composable transformations of Python+NumPy programs, 2018. URL http://gi thub. jaxmm//axx
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In International Conference on Learning Representations, 2023.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684 1704, 2025.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Aduances in Neural Information Processing Systems, 34:8780 8794, 2021.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106 (496):1602 1614, 2011.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datascts for deep data-driven reinforcement learning. ai preprint arXiv:2004.0221 2021. URL https://arxiv.orr abs/2004. 07219.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference on Machine Learning, volume 80, pp. 1587-1596, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. Tn Proceedings of the International Conference Machine Learning, volume 97, pp. 2052 2052-2062, 2019.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Jamner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as actor-critic method with diffusion policies. orXiv preprint arXiv:2304. 10573, 2023. URL https://arxiv. http:/iww..rr//b////44444// 2057/ 10573.
Jonathan Hock, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2024. URL http://github.com google/flax.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 93:6840-6851, 2020.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the International Conference Machine Learning, volume 162, pp. 9902-9915, 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, volume 34, pp. 21696 (1696-21707, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Roger Koenker and Kevin F. Hallock. Quantile regression. Journal of Economic Perspectives, 15(4):143-156, December 2001. doi: 10.1257/jep.15.4144.
Nya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 2021. URL https://github.. com//Kostrikkk//jajrr
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline rein- forcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 1179 1191, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXtv:2005.01643, 2020, URL https://arxiv.ot org/ abs/2005. 01643.
Wenhao Li. Efficient planning with latent diffusion. In International Conference Learning Representations, 2024.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-LL improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (ods.), Advances in Neural Information Processing Systems, 2022.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wicrstra. Continuous control with deep reinforccment learning. In International Conference on Learning Representations, 2016.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffussin sampling in offline reinforcement learning. Tn International Conference on Machine Learning, pp. 22825-22855, 2023.
Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-DICE: In-sample diffusion guidance for offline reinforcement learning. In The Tiirth-eighth Annual Conference on Neural Information Processing Systems, 2024.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprinl arXiv:2066.0800 09302, 2020.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8177 8171, 2021.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In Porty-second International Conference on Machine Learning, 2025.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high perforrrformece deep learning library. In Advonces in Neural Information Processing Systems, volume 32, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weigltted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.0011 2019. URL https: // //aaxii.org/ab////0///.
00177.
Jan Peters and Stefan Schaal. Reinforcement learning by rewihe- regression for operational space control. In Proceedings of the International Conference on Mochine Learning, volume 227, pp. 745-750, 2007.
Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1607 1612, 2010.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisteee Intervention, pp. 234-241. Springer, 2015.
Jascha Sohl- Dick-Dick,,,in, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibriu thermodynamics. In International Conference on Machine Learning, pp. 2256-2266 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Nitish Srivastava, GeolTrey Hinton, Alex Krizhevsky, Ilya Sulskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56): 1929-1958, 2014.
Qing Wang, Jiechao Xiong, Lei Han, peng sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In Advances in Newral Information Processing Systems, volune 31, 2018.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffussion policies as an expressive policy class for offline reinforcement learning. Tn International Conference Learning Represenlations, 2022.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Advances in Neural Information Processing Systems, volume 33, pp. 7768 7778, 2020.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations, 2023.
16

---

Published in Transactions on Machine Learning Research (12/2025)
A Proofs and derivations
A.1 Proof of Proposition 3.3: computation of the target score
We follow the derivation utilized in (Lu et al., 2023).
Let us express the normalizing constant of (zo) as
A= A= Recall that (.)+)) extracts the w component of Then, we have:
qu (z( (Zu) = (oo)) 1o (2o))
Po(2)) qu (u(zo) Du
According to the definition of diffusion models, we have:
(=(2 = p (2002 )ozo = / p20(22)20)90(20) Pen (zo) d2o Ā
/
9&o (20 (20 (20)
<x((z)) 1
E (zo mxw)
1 (2o)]
A (0(20)2))@x(2))deee 9x(Zk))
A
and finally
22, (z Vza (zt))(zz) + Vza log Exu(zo ac) (o
A.2 Proof of Theorem 3.5
Recall that we want to sample from p(z) equation 10.
Lot us assume that the weight function w(-) is known, so we have access a labeled datasct consisting of samples %o (ao, w((a)). We suppose K step diffusson process over Zo, Zд Og agen, é (0,,
To address the problem in Equation equation 4, let assume that train uoise-predicting diffusion model on the data Zo, (2 k), & E [0, K1, parametrized by 6. Then, the optimal diffusion model c* is defined by the following optimization problem.
argmin I|co - co(2,,
(17)
Let us also recall the data prediction formula (Efron, 2011; Kingma et al., 2021) for diffusion models, which states that the optimal diffuusio model satisfies:
E you (2n) (o) xs =ē =Z k) = 2s one (Zk, k))
(18)
Ork
We can use this formula to compute the intractable score in equation 10. Since the extraction function ew is linear (see Def. 3.1 in Sec. 3), we can swap the expectation and @ to obtain
Zzk T (Zk:
Egos (zo [Zo)] = (w (2o vn.) [2o]] = ow
(19)
Finally, plugging this term in equation 10
Zy: C (Zk, k)
Vz. (Zx) = Vzi log qu (Zx)
Vz log @w
(20)
Olk
(k)
self guidance
17

---

Published in Transactions on Machine Learning Research (12/2025)
Note that if we utilize data prediction diffusion model gos ( [z0] expressed as: V 22 (zk))(Zz) = Vz log (zx))
2Φ (2k, 1))
the target score can be
+ V Be log (ow (=g (zk,
(21)
(32. 5k
self guidance
Notice we must make the assumption that @w ("t (ze-o1e > O, which the optimal diffusion model satisfies if the weight function is positive w(a) > 0 Va E supp(A).
B Experimental details
B.1 2-D Toy examples
We followed the toy examples proposed in CEP (Lu et al., 2023), predefining a weight function w((a) and a dataset distribution ((a). We built. the dataset D with 1M samples from g(a) and extended it to Dz = (z = (a, w(a))]== using the predefined weight function.
We then trained a diffusion model E9 on the extended dataset Dz. For all toy examples, the diffusion model trained for K 100 diffusion steps, using Cosine noise schedule (Nichol & Dhariwal, 2021). The architecture consisted of 6-layer activations. We trained over 200k gradient steps and batch size 1024 using the Adam optimizer (Kingma. & Ba, 2015) with learning rate 10-4.
(MLP) with hidden dimension 512 and GeLU
To avoid training separate diffusion model for cach B, we conditioned the diffusion model on 8, co(zu, k. B), allowing us to train a single model per distribution q(a). We trained over B E [0, 20). For the sampling process, we set the guidance scale p to as p recovers the unbiased estimate of the target distribution score.
Figure 6 presents four additional experiments on the swiss roll, rings, moons, and checkerboard distributions.
R-0
p-1
p-s
1-10
#-15
p-5
e-10
p-15
SWD onrd)
SWO (our))
Dround truth
Stay
buth
Stat
8-0
A-1
d-5
S-10
2-15
F-5
6-10
P-15
SWC Coun)
bruth (ours)
Grouzd trith
Slal
Figure Additional examples for the experiment in Sec. 6.1. Target distributions from top-left, clockwise: swiss roll, rings, moons, and checkerboard distributions
B.2 D4RL experiments
For the D4RL Locomotion tasks, we averaged mean returns over random seeds and 20 independent evaluation trajectories per seed. For Ant. Maze tasks, averaged over 50 evaluation trajectories and five random seeds.
18

---

Published in Transactions on Machine Learning Research (12/2025)
The results found in Table 2 correspond to the normalized score in each domain, following the suggestion of the authors of the dataset (Fu et al., 2021). The results of our method correspond to the evaluation in the last training step. The results for other methods are taken from their respective original articles, except for Franka Kitchen, QGPO and IDQL results were obtained from (Li, 2024) and (Park et al., 2025). respectively.
Before training the models, we followed the suggestions found in (Fu et al., 2021) and standardized locomotion task rewards in the datasets by dividing them by the difference of returns of the best and worst trajectories in each datasct. For the Ant Maze dataset, subtracted 1 from the rewards.
To perform the offline RL experiments, it was first necessary to estimate the weight function. We used the expectile weights presented in IDQL (Hansen-Estruch et al., 2023) for the main D4RL results, as we found that they produce the best performance compared to the weights tested.
For the expectile weights, we needed to estimate the value functions Q(s, a) and V(s). We followed the same setup indicated in IQL (Kostrikov et al., 2022) and IDQL (Hanscn-Estree Estruch et al., 2023), which consists of estimating Q-values Qy and the value function Va. both with 2-layer MLP with ReLU activations and 256 hidden units. We trained the networks with Adam optimizer (Kingma & Ba, 2015) for 1M gradient steps, using learning rate 3e - 4 and batch size 256. We used double Q-lcarning (Fujimoto et al., 2018) and soft updates (Lillicrap et al., 2016) to stabilize the training. For the soft update, we used 0.005 in all experiments to update the Q-network. We also set y = 0.99 for all tasks, and T = 0.7 for the expectile loss, except for Ant. Maze where we use T = 0.9. The training procedure is described in Algorithm 3.
Algorithm Expectile Loss critic learning
Input: Datasct D {(s;, a,, s/) discount factor % expectile T, batch size B, learning rate a, target update parameter X.
Initialize: Q-network Qv, target Q-network Qgi, value network Vo. for cach gradient stop do
Get B samples from D
Compute expectile loss Lv(Ø) Blea)=B L3 (Qu(0,))- Update value network Φ + Ø- aVoLv(d)
Compute Q-value loss (V()) = )
(1+
Update Q-network vi + v
a a (1((
Update target Q-network & + (1 A)es + Xx
end for
Return: Optimized Q-network Qú and value network V
Once the value functions are learned, we were able to calculate the weights as detailed in Sec. 5. Then, we extended our dataset to Dz following the methodology described in Sec. 3. We trained a diffusion model eq this extended dataset for K = 15 diffusion steps in all experiments, using Adam optimizer (Kingma & Ba, 2015) with learning rate 3e- 4 for all tasks, except for Adroit Pen where use 3e - 3. We used batch size of 1024, a parseiig noise schedule (Song et al., 2021), and trained for 1M gradient steps in all tasks, except for Ant Maze, where we used 3M gradient steps.(Jannee et al., 2022; Hansen-Estrudd et al., 2023). The diffusion model consisted of a ResNet of 3 Residual blocks, with hidden dim 256 and Layer Normalization (Ba et al., 2016). We used a dropout (Srivastava et al., 2014) of 0.1 in all tasks. For the weight component. output, we used the features from the middle layer of the RosNet and passed them through a simple MLP with a hidden dimension of 256 to obtain more expressive representation of the weights. This architecture closely follows the one used in IDQL (laanen-EEstrucd et al., 2023), with the exception of the additional MLP dedicated to the weight component. However, this MLP did not introduce a significant number of additional parameters. The complete ResNet architecture contained approximately 1.8M parameters, ensuring a fair comparison in model capacity, as other guidance methods (Lu et al., 2023; Mao et al., 2024) use U-Net. architectures (Ronneberger et al., 2015) with a comparable number of total parameters.
During inference, we tuned the guidance scale for each task. We swept over p E {1,5, 10, 15, 20, 25. 30). such relatively large guidance scales were considered due to the expectile weights used, which were bounded in
19

---

Published in Transactions on Machine Learning Research (12/2025)
range. The guidance scale used for each experiment to obtain SWG results can be found in Tables 6, 7, 8 and 9. The results found in Table 2 correspond to the evaluation of the model at the last training step. For SWG-R (with the added resampling step), we conducted an additional hyperparameter search over the batch size, sweeping M& E (1,2,4,8,66)
Table 6: Utilized guidance scales for locomotion results in Table 2
Half Cheetah
Walker2d Hopper
Medium- Expert
15
Medium
20
20
Medium Meeiumy-Replay
30
10
10
Table 7:
Utilized guidance scales for Ant Maze results in Table
U-Maze
Medium
Large
Play Default
15
Diverse
15
Table 8: Utilized guidance scales for Franka Kitchen results in Table
Partial
Mixed
Complete
10
15
Table 9: Utilized guidance scales for Adroit Pen results in Table 2
Human Cloned
25
B.3 Scaling of inference time in SWG
Each data point in Figures and 5 represents the average inference time with our method for K = 15 diffusion steps, obtained over 10,000 runs of the algorithm, along with the corresponding error region. For completeness, we report the total number of parameters in each network used our experiments. The table below shows the parameter counts corresponding different width and depth configurations:
Table 10: Number of parameters for each network configuration used in our experiments. Left: scaling with width (hidden size). Right: scaling with depth (number of layers).
Width (hidden size)
Parameters
Depth (layers)
Parameters
128
180K
660K
256
370K
4.5M
512
1M
9.1M
1024
3.5M
4
13.3M
2056
13.3M
17.5M
4096
51.5M
6
21.8M
8192
203.7M
7
26M
20

---

Published in Transactions on Machine Learning Research (12/2025)
B.4 Computational resources
Our implementation is based on the jaxrl repository (Kostrikov, 2021) using the JAX (Bradbury et al., 2018) and Flax (Heek et al., 2024) libraries. We also provide an implementation of our method in Py'orrh (Paszke et al., 2019). All experiments were conducted on a 12GB NVIDIA RTX 3080 Ti GPU. Training takes approximately 0.5 hours for value function learning (i.e., weight training, 1M steps), and an additional 0.5 hours for 1M steps of the diffusion model. However, since experiments can be run parallel, the overall training time can be significantly reduced. The code used for training and experimentation be found in the SWG repository
21


Reviews:
[{"review_id": "24ScaIvUAK", "reviewer": "Reviewer_rDyL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Overview\nThis paper presents a novel exact guidance approach for diffusion policies called Self-Weighted Guidance (SWG).  \nUnlike previous methods that require a separate guidance function with often complex loss formulations, SWG eliminates the need for such explicit guidance by jointly learning actions and weights within a single diffusion model.  \nThis enables the guidance score for sampling target policies to be computed directly from the model itself, resulting in a streamlined training and inference process.\n\n## Experimental Validation\n- SWG is validated on toy distributions, showing accurate sampling capabilities.  \n- The authors benchmark SWG and its variant SWG-R on standard D4RL datasets.  \n- Ablation studies are conducted to evaluate different weight formulations and model scalability.\n\n## Strengths\n- Delivers exact guidance directly from the model  \n- Simplifies the training procedure  \n- Broad applicability to weighted sampling tasks  \n\n## Weaknesses\n- Relies on the choice of weight formulation  \n- Computing gradient through the diffusion model can be slow  \n- Requires retraining for changes in guidance  \n- Shows mixed performance on the D4RL benchmark tasks", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The paper provides a clear derivation and theoretical proof for the SWG method, demonstrating the mechanism for exact guidance in diffusion models.  \n\n- It includes validation on toy distributions with known target energy functions, showing that SWG can sample accurately and consistently from these distributions.  \n\n- The method is tested on the D4RL benchmark suite, where results exhibit competitive performance in several challenging environments (Ant Maze, Franka Kitchen, Adroit Pen), although SWG shows only average results on locomotion tasks.  \n\n- The ablation studies on different weight formulations and model scaling further support the paper.", "audience_yn": "Yes", "audience_explanation": "The work addresses active research areas by introducing a novel self-weighted guidance mechanism for diffusion models in offline reinforcement learning, a topic relevant to both the generative modeling and RL communities. The paper demonstrates theoretical understanding through exact guidance derivations and provides rigorous empirical validation on standard benchmarks (D4RL, toy distributions, and ablation studies). These contributions are important for researchers interested in both offline RL and generative modeling using diffusion models. The paper aligns well with the interests and scope of TMLR’s readership.", "requested_changes": "## Requested Changes\n\n# Critical\n\n- **Introduction:** Please moderate or clarify claims about SWG’s “top performance” in D4RL, as the results indicate it is not consistently superior across all datasets.  \n\n- **Section 2.2:** Rephrase the statement that diffusion models (DMs) are “highly expressive compared to classical neural architectures.” While DMs employ a different training paradigm, their underlying architectures are often similar to standard neural networks.  \n\n- **Algorithm 1:** Explicitly state which w function is used for dataset augmentation to improve clarity and reproducibility.  \n\n- **SWG-R Details:** Clarify whether the weights used in SWG-R for resampling are the diffused weights from the joint model or from athe separate weights model (w) .  \n\n- **Results:** Please define all dataset acronyms used in Table 2 (e.g., ME, M, MR, partial, mixed, complete, Avg) within the table caption or in a dedicated glossary/footnote for clarity. This will help readers unfamiliar with the terminology to interpret the results correctly.\n\n- **Locomotion Results:** The paper downplays the importance of the saturated locomotion benchmarks, but the average or below–state-of-the-art performance of SWG-R on these tasks is notable.\n\n# Strengthening contribution\n I consistently observe that SWG-R underperforms in the partial, mixed, mixed-replay, and diverse settings. Please discuss whether this underperformance could be attributed to the choice or quality of the pre-trained weight function, and elaborate on how dataset composition or weight function design might impact SWG-R’s effectiveness.", "broader_impact_concerns": "Broader impacts of this work include the possibility of amplifying biases present in the underlying datasets, since policies learned using weighted behavior can perpetuate those. Additionally, training and deploying diffusion models with increased computation—as done in this paper—require substantial computational resources, leading to increased energy consumption and environmental impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ODiGv5LwQp", "reviewer": "Reviewer_Upva", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies diffusion models (DMs) for offline reinforcement learning (RL), where the target policy weighs between the behavior policy and a critic. DM-based methods approximate the behavior policy with a DM, then learn the weight function/critic via resampling or using a guidance network. This latter step is challenging because of intractable score functions and entangled weights and actions that yield numerical instability. The proposed method, SWG (self-weighted guidance), avoids those issues by embedding the guidance network within the same DM used for sampling from the behavior policy. Specifically, SWG leverages the fact that weights are deterministic functions of actions to adapt the guidance loss accordingly.", "claims_and_evidence_yn": "No", "claims_explanation": "As a disclaimer, I am unfamiliar with diffusion models and their training methods. \nHowever, the problem description and solutions *lack clarity* and self-contained explanations:\n- How does IDQL generalize IQL? How does Eq. (5) relate to Eq. (4) or the loss used for expectile regression? \n- How does one solve the reverse diffusion process after Eq. (7)? What does the score function in (9) model, and where does its relation with the noise come from? \n\nFrom a broader perspective, the *motivation* for using DMs in offline RL remains unaddressed in this work. Why privilege DM over other generative models, e.g., ensembles? The approach relies on weighted behavior policies: how limiting is this assumption? Does SWG specifically cope with out-of-data actions compared to other similar methods? \n\nThe *performance of SWG* compared with other baselines is not convincing. The authors claim that SWG competes with other DM-based methods, but D-DICE typically outperforms in every benchmark. The potential instabilities mentioned after Eq. (22) are not very convincing to me, and a more thorough comparison with those SOTA DM-based methods would have been much appreciated in addition to the ablation studies proposed in this work. \n\nThe *limitations* of SWG are not properly addressed. Those mentioned in Sec. 7 do not refer to shortcomings that are intrinsic to SWG but rather to incomplete parameter and computational cost optimizations. I would expect the authors to explore the limitations of removing the guidance network instead: there should be a trade-off. We arguably win in stability and cost but lose in performance/sth else?", "audience_yn": "Yes", "audience_explanation": "Avoiding additional training for the weights could be beneficial and make DMs for offline RL more accessible, but the authors should develop on the loss incurred by such avoidance.", "requested_changes": "Besides my substantive comments above, the following minor changes should be made: \n- Remove equation numbering when not referred to in the text\n- Equation referring should be in parentheses + capital letter (Equation (3) instead of equation 3)\n- Table 1 is superfluous to me -- these characteristic comparisons are already given in the text\n- Changes in notations are confusing -- from Sec. 2 to 3, $\\pi$ becomes $p$, $\\mu$ becomes $q$? Please double-check consistency to avoid misunderstanding. \n- The first sentence after Eq. (9) does not mean anything. Please review phrasing here, as well as in other places (repetitive words, starting sentences with Though, etc.)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fPHNw0JeKb", "reviewer": "Reviewer_Su2U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Problem Setup:** Offline RL addresses scenarios where agents must learn effective policies from static datasets without environment interaction. The main challenge is avoiding dependency on out-of-distribution action evaluation when learning the target policy, as inferring dynamics or rewards for unseen state-action pairs typically leads to overestimation errors. Beyond model-based methods or trajectory modeling approaches, most methods model the target policy as π(a|s) ∝ μ(a|s)w(s,a), where μ is the behavior policy and w represents action quality. Methods like IQL learn these weights using advantage functions A(s,a) = Q(s,a) - V(s). Here the value are calculated by leveraging expectile regression, which avoids querying good actions by learning expectiles of behavioral Q-functions rather than requiring max/min operations over the action space. \n\nIn the diffusion paradigm, models can effectively capture behavioral distributions, but extracting improved policies requires additional mechanisms. Resampling approaches like IDQL sample multiple candidates from the behavioral diffusion model and select actions using learned critics. Guidance-based methods like QGPO learn explicit guidance networks to steer the diffusion process toward higher-value actions during sampling. D-DICE combines both paradigms—using in-sample guidance learning to generate high-quality candidates, then selecting among them—achieving state-of-the-art performance across D4RL benchmarks. \n\n\n**Proposed Solution:** The proposed SWG approach attempts to simplify this pipeline by jointly modeling actions and weights within a single diffusion model. Instead of learning separate models for behavioral policy and guidance networks, SWG trains on augmented data tuples (state, action, weight) where weights come from learned advantage functions. During sampling, at each denoising step, the guidance mechanism uses the data prediction formula to estimate what the clean (action, weight) pair should be, then extracts the weight component from this prediction and computes its gradient ∇_{z_k} log(ϕ_w(z₀_predicted)). This gradient is used to modify the noise prediction, steering the denoising process toward configurations that would yield higher weights. This eliminates the need for separate guidance networks since the weight predictions come directly from the same diffusion model, making the guidance signal a natural byproduct of the joint modeling approach.\n\n\n**Strengths:**\n- **Direct Score Computation:** The joint modeling of actions and weights enables direct computation of guidance scores from the diffusion model's own prediction head, eliminating the need to approximate intractable expectations E[w(a_0)|a_k] through auxiliary networks with complex loss functions.\n- **Simplified Training Pipeline:** Uses only standard diffusion loss minimization (MSE on predicted noise), avoiding slightly complex exponential terms, contrastive objectives, and potential training instabilities found in D-DICE and CEP guidance networks. This consolidates behavioral modeling and guidance into a single network, reducing the number of components that need to be trained, tuned, and deployed.\n- **Good set of experiments and Ablations:**  Includes comprehensive ablations on guidance scaling, inference time scaling with model size (with/without guidance), and different weight formulations. Demonstrates competitive performance with established baselines including IQL and D-QL across D4RL benchmarks.\n\n**Weakness:**\n* **Fundamental Performance Gap Without Justification:** SWG consistently across all tasks underperforms state-of-the-art methods, with D-DICE (case in point D-DICE achieves 90.6 vs SWG's 75.5 on locomotion tasks). This substantial gap undermines the paper's central premise, especially since D-DICE already effectively solves the \"intractable\" guidance problem that SWG claims to address. So some questions still arise.\n    * Why joint modeling should outperform specialized guidance networks ? Should they ?\n    * How do we quantify practical benefits the claimed simplicity actually provides ?  Why should one accept the performance degradation when there is no evidence that this framework can be better tuned for a specific problem we care about ?\n    * Why did D-DICE actually perform better ? Is it that specialized guidance is just better and modeling guidance signals ? \n    * How much of performance of D-DICE is attributed to better guidance modeling as compared to the resampling of actions ?  \n    * Can the performance gap be mitigated by adding resampling on top of SWG? How does resampling budget affect performance?", "claims_and_evidence_yn": "No", "claims_explanation": "**Supported Claim:** The paper's mathematical derivations are technically sound, and the toy experiments provide clear evidence that SWG can sample from target distributions exactly when the weight function is known.\n\n**Unsupported Claims:** \n\n- While the D4RL experimental evaluation is comprehensive, it consistently shows SWG underperforming state-of-the-art methods across most benchmarks. The authors present this as a reasonable trade-off for simplicity, but provide insufficient evidence that this trade-off is worthwhile.\n\n- The claim that SWG offers a \"streamlined training pipeline\" is supported by the elimination of separate guidance networks, but the practical benefits remain unclear given that weight function learning is still required. The paper subtly assumes architectural elegance translates to practical value, which may be true but needs a more thorough investigation.", "audience_yn": "Yes", "audience_explanation": "The paper addresses an important problem in offline RL and proposes a theoretically interesting approach. \nHowever, the practical impact is not fully validated given the performance gaps with existing methods. Still The joint modeling perspective offers insights that could inspire future work.", "requested_changes": "Proposed Critical Changes for Acceptance:\n- **Strengthen the motivational argument:** Either empirically demonstrate clear practical advantages (training stability, computational efficiency, memory usage) that justify the performance trade-off, or reframe the contribution as a theoretical exploration rather than a practical improvement.\n- **Decompose D-DICE Contributions:** Isolate how much of D-DICE's performance stems from guidance quality vs. action resampling through ablations with D-DICE variants (guidance-only, selection-only).\n- **Hybrid Approach Evaluation:** Test whether adding resampling on top of SWG can close the performance gap, and analyze how resampling budget affects results.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)