D2 Actor Critic: Diffusion Actor Meets Distributional Critic
Authors: Lunjun Zhang
Shuo Han
Hanrui Lyu
Bradly C. Stadie
Venue: Accepted by TMLR ()

Abstract: We introduce D2AC, a new model-free reinforcement learning (RL) algorithm designed to train expressive diffusion policies online effectively. At its core is a policy improvement objective that avoids the high variance of typical policy gradients and the complexity of backpropagation through time. This stable learning process is critically enabled by our second contribution: a robust distributional critic, which we design through a fusion of distributional RL and clipped double Q-learning. The resulting algorithm is highly effective, achieving state-of-the-art performance on a benchmark of eighteen hard RL tasks, including Humanoid, Dog, and Shadow Hand domains, spanning both dense-reward and goal-conditioned RL scenarios. Beyond standard benchmarks, we also evaluate a biologically motivated predator-prey task to examine the behavioral robustness and generalization capacity of our approach.

Published in Transactions on Machine Learning Research (10/2025)
D2 Actor Critic: Diffusion Actor Meets Distributional Critic
Lunjun Zhang
lunjumn@@s.toronto.ede
Deportment of Computer Science, University of Toronto
Shuo Han
Department of Statistics, Northweeserrn University
Hanrui Lyu
Deportment of Statistics, Northwestern University
Bradly C Stadie
bradly. bradly.saatie@hhottuussst.e
Department of Statistics, Northwestern University
Reviewed on OpenReview: https: openrevveew net/ forum? id- 8kbstCUXMH
Abstract
We introduce D2AC, a new model-free reinforcement learning (RL) algorithm designed to train expressive diffusson policies online effectively. At its core policy improvement objective that avoids the high variance of typical policy gradients and the complexity of backpropagation through time. This stable learning process is critically enabled by our second contribution: robust distributional critic, which we design through fusion of distributional RL and clipped double Q-learning. The resulting algorithm is highly effective, achieving state- of-the-art performance on benchmark of eighteen hard RL tasks, including Humanoid, Dog, and Shadow Hand domains, spanning both dense-reward and goal-conditioned RL scenarios. Beyond standard benchmarks, we also evaluate biologically motivated predator-pres task to examine the behavioral robustness and generalization capacity of approach.
1 Introduction
Actor-Critic methods have been at the center of many recent advances in Reinforcement Learning, In continuous control and robotics, Soft Actor- Critic (Haarnoja et al. 2018) outperformed strong prior model- based methods (Wang et al.) 2019). On the ALE environment, Atari agents such as Rainbow (Hessel et al. 2018), BBF (Schwarzer et al.] 2023). and ULTHO (Yuan et al. 2025 have continued to push forward stale-oftthe-ar reinforcement learning performance.
Each instantiation of actor-critic methods crucially relies on two components: an actor and critic. The critic tells us how valuable it is to be in given state, and the actor tells us what should do when we find ourselves in that state. Generally,
actor-critic algorithın will only be as strong as the weakest, link; a powerful critic is wasted on an actor that can't optimize for the correct actions, and vice versa. This naturally leads us to ask the question of how we can use modern advancements in machine learning and function estimation to learn a critic and an actor that are a perfect match, equally powerful, and capable of complementing one another.
When estimating the critic, classical RL favored point estimates for the value function, often resulting in unstable estimation. While modeling the full distribution of returns (Bellemare et al. 2017) significantly improves robustness-a principle reinforced by recent trend advocating for classification-stlle objectives
"Equal contribution.

---

Published in Transactions on Machine Learning Research (10/2025)
over standard regression for value learning (Imani et al. 2024 Farebrother et al. 2024 we observe that this approach is not immune to the overestimation bias inherent in Q-learning. Our first contribution is to create a more stable teacher for the actor: we find that applying the clipped double Q-learning technique from TD3 Fujimoto et al. 2018) to approximate the return distribution reduces noise and provides a more reliable learning signal for policy improvement.
However, even a stable teacher is wasted on an actor that cannot effectively utilize its guidance. While recent work has explored powerful denoising diffusion models for the policy (Yang al.] 2023 core challenge remains: how to effectively bridge the critic's guidance with the actor's learning process? Policy gradient estimation for diffusion policies often suffers from high variance when applicd to diffusion models (Jeha et al. 2024). while other methods can be complex and unstable (Ding et al. 2024 Ma et al. 2025
This brings us to the core technical contribution of our work. We derive a new policy improvement objective for diffusion actors that both thcorctically grounded and practically efficient. Taking inspiration from monotonic policy improvement theory (Schulman et al., 2015), our derivation simplifies the complex, multi- step policy optimization into stable, single-step supervised objective guided directly by the critic's value gradients. This simplified objective provides the crucial link, allowing expressive diffusion actor to effectively learn from our stable distributional critic (i.e., critic that models the full distribution of returns), forming a more synergistic actor-critic system.
In this paper, we show how to marry a distributional critic to a diffusion actor, leading to the development of D2AC, a new actor-critic algorithm. D2AC achieves excellent performance on a variety of hard robotics environments, including locomotion tasks such Humanoid and Dog domains in DeepMindd Control Suite Tassa ct al. 2018), and manipulation tasks such as Pick-and-Placc and Shadow-Hand Manipulate in multi- goal RL environments with sparse rewards (Plappert et al. 2018). In addition, we evaluate D2AC on a biology-inspicee predator-pan prey benchmark (Lai et al. 2024), where it demonstrates higher exploration coverage, richer path diversity, and superior zero-shot transfer compared to SAC and TD-MPC2. Importantly, D2AC consistently performs well across difficult domains where prior model-free RL methods struggle or completely fail. Moreover, it significantly closes the performance gap between model-free RI, and SOTA model-based methods such as TD-MPC2 (Hansen al.) 2023), despite using an order-of-magnitus less compute compared to TD-MPC2 and being considerably faster to run. Our results show that D2AC can serve a strong base for policy optimization in continuous control. Videos of our policy in action can be found here
Distributional Critic
Diffusion Step k-K
Diffusion Actor
Diffusion k-0 k-0
Distributional Critic
Pick
alk) ~ p(.) a 1) ~ mTl.) (x))
and Place
Denoising
(ss.a)
(()
((-11)
Fetch Slide
g(a (A)
a (11)
Q(.aa
Q(saa
Adding Noise
Figure 1: D2 Actor Critic uses a critic that models a distribution over possible returns. A diffusion actor uses the expected value of this distribution (the Qufuctioo) to help align the denoising process with policy improvement. Above are visualizations of the Bick-and-Placc and Fetch Slide environments.
http:://eacc-attorrrritt... io/
2

---

Published in Transactions on Machine Learning Research (10/2025)
2
Related Works
Model-free R.L has two families of approaches: policy gradient methods and value-based methods. Policy gradient methods typically require on-policy data and trust-region regularization (Schulman et al.] 2015
2017b). Valuc-based methods, on the other hand, enable off-policy training, which can greatly improve sample efficiency. Since DQN (Mnih et al., 2015 actor-critic extensions like DDPG (Lillicrap et al. 2015 brought these methods to continuous control, and subsequent improvements include tackling Q-value overestimation in TD3 (Fujimoto et al. 2018) and the introduction of soft Qllearning (Haarnoja et al. 2018). A particularly relevant line of work is distributional RL (Bellemare et al., 2017), which moves beyond point-cstimates to model the entire distribution of returns. This concept has been successfully integrated into actor-critic frameworks to address value estimation errors or enable risk-sensitive learning, as demonstrated by some concurrent works like DSAC (Ma et al., 2025). D2AC builds directly on this progross, unifying a robust distributional critic with clipped Q-learning and, crucially, introducing novel actor update mechanism designed to leverage these richer value representations.
The paradigm of learning a world model and planning with it has achieved significant breakthroughs (Schrittwieser al. 2020 Hansen et al., 2022). We are particularly inspired by core mechanisms of modern planners like Model Predictive Path Integral Control (MPPI) (Wiliims et al., 2015), where action proposals are iteratively refined based insights from full distribution of simulated future returns. D2AC is attempt to distill these principles into cohesive, purely model-free framework, making our approach orthogonal to and potentially a stronger base for future hybrid methods.
Model-based RL and Planning:
Diffusion Policy for RL has attracted interest following the success of denoising diffusion models (Sohl Dickstein et al.) 2015 Ho al. 2020 Song et al. 2020b). The most straightforward application is conditional
behavior cloning (BC) (Janner al.
2022 Chi al.
2023 Römer al. 2024). Extending diffusson policies
beyond BC, prior works have explored both offline RL
Hansen-Estruch et al. 2023 Gao et al. 2025 and
online RL settings (Chen et al., 2023 Ma et al. 2025
Dong et al. 2025). A key challenge, however, is
that policy improvement in online RL necessarily induces a non-stationary target distribution, which makes diffusion-based training unstable. Our work takes a different perspective: drawing on monotonic improvement theory from policy optimization literature (Schulman et al., 2015), we derive a new method for training diffusion policies the online setting. This different from some concurrent approaches, such as Liu et al. (2025), which also combine a distributional critic with a diffusion policy but rely on policy gradient updates. Our core technical contribution thus offers distinct and efficient alternative for actor training in this area. We note that Liu et al. (2225) was developed concurrently and view our work as complementary in exploring different optimization perspectives.
3 Background
3.1 Reinforcement Learning
A Markov Decision Process (MDP) is defined by the state space S, action space A, the initial state distribution Po(s), the transition probability p(s' s, a,),, and the reward function r(s, a).
The policy =(a s) generates a trajectory I with horizon T within the environment:
T -So, ag. .h....... ST}
(1)
where So ~ Po(-), a, n|- Bx), and s1+1 p(- st a,) for t e (..... T-11. 1}.
We denote the trajectory distribution starting from state under policy as p With a discount factor 7 E (0. 1), the goal is for the policy æ to maximize the discounted cumulative rewards:
T Σ
[(T) = E poo(so)p (r|s0)
2r(rss, at)
(2)
3

---

Published in Transactions on Machine Learning Research (10/2025)
Value-based RL methods optimize this objective by estimating Q function for the current policy
(sr. au) Bp(s (r
Σ Ar(ssss
(3)
and then training the policy to maximize Q.
3.2
Denoising Diffusion Models
Denoising Diffusson Models are generative models that estimate the gradients of data distribution (Vincent
2011 Song Ermon 2019 Song et al. 2020a). Diffusion models train a denoiser D& to reconstruct data
X Pdata from noised data [J) N(x, If under various noise levels max = OK > > dg Omin
Let the distribution of noised data be defined as:
r(x(k)) = panta (x)((x(1)))))))))))))))))))))() of
(4)
The min min and max max of noise levels are selected such that p(x(01) pdata(x) and p(x( NO I I).
The diffusion loss is typically:
Eo-N(())), kk) xX cσRiON) -x12]]
(5)
where X(k) is the loss weighting based noise level.
Then the score for arbitrary input x is:
Vx logp(x;o) Do(x;o) -
(6)
x²
which can be used to sample data by starting from noise and solving either an SDE or ODE (Song et al. 2020bb.
We follow EDM Karras et al. 2022 to define the denoiser as:
D Dx 0) = Calāp(o) + Cout (a) Fo (Cin((()) (o)x, Cuoise (a))
where Fg, is a neural network, and (o), Cout (o), Cm (a), Coniso (a) are o-dependent scaling coefficients defined in (Karra et al. 2022). These coefficients control the relative weighting the input signal, the output the neural network, and the noise embedding across different noise levels.
3.3 Clipped Double Q-learning for Categorical Return Distributions
While distributional reinforcement lcarning (RL) provides richer return modeling by capturing full return distributions instead of scalar estimates (Bellemare et al. 2017), inherits a well-known flaw of traditional Q-lerrning: overestimation bias (Van Hasselt [0100] Van Hasselt al. 2016). In distributional RL, this bias manifests not just in the expected value, but across entire regions of the predicted return distribution, leading to pathological optimism.
To address this, we apply clipped double Q-learning approach (Fujimoto et al. 2018 within the distributional setting. Though this combination is conceptually simple, we demonstrate in Figure that it plays a crucial role in stabilizing training and improving empirical performance.
We represent return distributions using the categorical formulation introduced Bellemare al. (2017). Specifically, our distributional model learns to predict probability distribution, a), fixed, discrete support. of possible returns 2g
Categorical Representation of Return Distributions:
4

---

Published in Transactions on Machine Learning Research (10/2025)
Wmiin..... ... Vmnal- This distribution is parameterized by softmax output over N bins, where the probability of each bin corresponds to an atom in the support z. The expected Q-value is computed as:
Qo a) (s.
Any continuous value within the support range can be represented via linear combination of two adjacent bins using a two-hot encoding function hz, (-).
Categorical Projection: To perform bootstrapped learning, we shift the support according the observed reward and discount factor: 2p r2g The predicted distribution p = 47(s', a') under this shifted support must be projected back to the original support Zg compute training target. This is done via:
N
dist (Zp. 2.
h22 (zpli))( pbl
i=0
This yields valid probability distribution over the original support, which serves supervised signal.
Clipped Double Distributional RL: To reduce overestimation, maintain two independent critics, qa, and q0 each producing distribution over returns. For each training step, we compute expected Q-values:
Qo. (s. a) (s, 2n
We then select the entire distribution from the critic with the lower expected value:
qo, a) qo, as otherwise
if Qa Q0,
(s,a)
This clipped distribution is projected and used to train both critics via cross-entropy loss. The full critic loss becomes:
Leritic (0) l (log qo, (s, + log qo (s, a))
where l = Paist + YZg, 9
clip (s', a²), 2,)) and a' ~ TT(- ss)
4 Policy Improvement for Diffusion Actors
This section introduces the core technical contribution of our work: an efficient policy improvement objective for diffusion actors. To derive this objective, we first reframe the multi-step denoising process as a special type of Markov Decision Process. We then leverage the principles of monotonic policy improvement (Schulman let al. 2015) and, through a key theoretical simplification, arrive at a tractable objective that aligns the single-step denoising function with policy improvement.
When the policy in actor-critic algorithms is a diffusion model (Sohl-Dicksteinn et al. 2015 it starts from the noise distribution (N o2 max I) and goes through K diffusion steps a (k)
Ho al. 2020
using sequence of noise levels Fmax ок > sampler based on the Kuler-Maruyama method (Song et al. 2020b), then one step of the diffusion process
01
J'ia to sample action. If we assume
a (k) - (k-1) for policy Ts can be written as:
(a (k-1) s, (k)) = (N(ak-11) (a(((()) k; s), (of I)
Where us a learned denoising process that takes actions away from the noise and towards the desired action distribution. is parameterized as:
(s (k) kk;) = a (k) + (of - of Va (k) lok.s)
V. loogoopa a,s) = (Da(a, a)/o²
5

---

Published in Transactions on Machine Learning Research (10/2025)
where Ds(a,o;s) Cakip Cout (ou Fa, Ca( Cu)iso The functions Cakip(o) Cout and (n)ise are flexible design choices (Karras et al., 2022). Thus, taking action (0) from the denoising process a (k) - ....a"" (0) can be written as an integral over the denoising steps:
K
Ta s) = / p(a)k)) II The (a
s, d
(8)
We would like a way to guarantee the diffusion process actually provides with better action. Define the optimal policy to be pa(a s) (exp(Q(s, (exp(Q(s,a//a)) a)/a)) where is the temperature. The goal of
entropy-rogularizee policy improvement is to minimize the following (Schulman et al. 2017a Haarnoja et al. Black et al. 2023):
2018
arg max (KL(Ts(a|s) pa s)) = Eawa, (1 (Q((s))) + (H s))
Diffusion Policy Gradients: In the case of diffusion policies, our main challenge with optimizing for policy improvement is that accessing Th requires K intermediate sampling steps. To optimize lower bound the policy gradient objective, the variational lower bound in Kingma al. (2021) can be applied with positive transformation of the Q-valucs (Peters & Schaal (k) = (1/0k-1 1/of) (K/2):
2007
Abdolmaleki ct al. 2018), with
1 (o.. (k)]la Dø(a + oxe, oz; ox: exp (s,
(10)
However, policy gradients tend to suffer from high variance (Jeha et al. 2024), while value gradients (Heess et al. 2015) for diffusion require backpropagation through time (BPTT), which is notoriously difficult.
Pascanu et al. 2013 Wallace et al. 2023 Black et al.) 2023). Next, we will show that by re-examining the diffusion process through the lens an MDP, we can circumvent these challenges.
To ground our approach in established RL theory, we first define a Diffusion MDP taking K steps of actions from p(a(k) to ((a(111) ss ak)) reaching the final state ( After this diffusion process (under discount factor 7), the policy receives reward Q(s, / K at the final timestep of diffusion (reward is 0 elsewhere). The objective is to maximize the expected terminal reward:
Diffusion MDP:
n(s. π) (K). [(ss [0)]
(11)
This objective makes the dependence on the intermediate diffusson steps aK) . in the diffusion MDP, the Q-function, value function, and advantage function are given by:
... (1) explicit. Correspondingly,
Vdlffiusion ((s, a ())) A B (x k-1 1//4//1///// aQ(ss (()
x diffussion ((s, a (k+1)), ()))) & E E [y Vidifuusion ((s, ((s,a (*
A diffiusion
((s, a (1+1)) (k))) A Q diffusiio ((s, a (1+11), a (1)) ddifiuusonn ((s, a (++1)))
Monotonic policy improvement theory, specifically the policy improvement objective from Trust Region Policy Optimization (Schulman et al. 2015 can now be applied to this Diffusion MDP. This gives us the following proximal objective for policy improvement:
4e7
1/(8, T) 2 Ja (8, T)
max k((l))
ss (k)), T(. 8s a aa))))
(1 7)2
J Jane (s, T) n/s, Tret) + En-Unif(1,, .aa ( warer
h A Adifiusion (astiisiiossssss ((s,a (k)), (k--1))] a
A
((t-1) (--aa()) (a)))
This result provides a lower bound on our policy objective equation 11 expressed through loss that depends the Diffusion MDP advantage of the reference policy A diffussio and the KT divergence between the reference
Reaf

---

Published in Transactions on Machine Learning Research (10/2025)
policy Whet and our learning policy When our policy remains sufficiently close to the reference, maximizing this lower bound guarantees that performance will either improve or at least not deteriorate. We can therefore make progress by following the gradient of the advantage term in this lower bound:
(8, = V UeWUnir(L " ( Tσ)
KK (210-11, Din 1 Q ((s, a ( (
(12)
jprentmaal (8,T4)
sot
Simplification via a One-Step Lower Bound: Learning the multi-step value function Quiit still difficult. To make our objective practical, we introduce key simplification based on core property of diffusion models. We utilize the fact that the denoising function D& (a(a),, o; aims to predict strictly cleaner signal with each step (Song et al.
2020a). For well-trained model, more sampling steps generally lead to higher quality actions à (Ho et al. unnormalized log probability (Levine) 2018), we build on the intuition that actions generated with more refinement steps should, on average, yield higher Q-values. This insight allows us to posit that the value of a single-step generation can serve as a tractable lower bound for the full, multi-step generation. We formalize this crucial simplification the following proposition:
2020 Song et al. 2020b). Since the Q-value is analogous to an
Proposition 1. Define the data distribution under diffusion step & 2 0 be (with â 2 aminn:
(a(aas) = / ((a (a)aa (0), (f (0) Pu (als)) - Pa+ (uls) Do () 6
Under the assumptions that (i) entropy of Ph. and Pu strictly increases with k; (ti) the KL distance from Pe. and Pk to optimal policy p' is non-decreasing with k; (ttt) â is sufficiently large such that DKL pr) D p"1 Then for any state and diffusion step k we have
Epo [Q1 2 1px
(13)
Proposition formalizes simple intuition: a single denoising step always improves upon a noisier action distribution, even if it does not reach the final, clean action distribution. This insight powerful because it justifies replacing the multi-step expectation in equation
12 with more tractable one-step lower bound. Specifically, the expectation over the remaining k - 1 steps can be bownrebounded by the expectation over single denoising step:
* (a (ddiffusion ((s, a (k)) a a (x-1)
-726- 2k- K "Egtt-
Eg (x(t-2) ... nosten
7 _k EN(aaDa (Q a)]
Additional
SDE steps
One step generation
This one-step lower bound is much easier to optimize since it avoids learning separate value functions Qdiffuusion in the diffusion MDP. Substituting it into equation 12 yields:
Diffusion Value Gradients:
jproximal (s, TT) 2 E Unif((11 " /) Q(s, Ds(a ox:ss) - GE)]
(14)
This simplified objective can be expressed in a form similar to the L2 loss commonly used in diffusion models Kingma et al. 2021 Setting nhe = Taz we arrive at the final D2AC policy loss:
V L simple (0)
V + Do(a =
(15)
While these formal assumptions, they are grounded in the empirical behavior of well-trained diffusion models
7

---

Published in Transactions on Machine Learning Research (10/2025)
The function A(k) is the weighting according to the noise schedule. X(k) > // in our derivation; in practice we find that simply setting X(k) works well. Conceptually, this loss teaches the denoising network D& to predict a target that has been pushed slightly up the gradient of the Q-function. In essence, each denoising step learns to do so in a direction that groedily improves the action according to the critic. Compared to diffusson policy gradients equation 10, this value-gradient approach avoids BPTT and provides a more stable training signal, whose quality critically dependent on the rich information provided by our distributional critic.
5
Experiments
Humanold: 21 DoF
Dog: 38 DoF
Quadruped: 12 D2
Pick-and-Place 4-DoF
Manipulate Block. 20 DoF
Manipulate Egg: DoF
Manipulate
Figure 2: D2AC works out of the box across a wide range of environments, including locomotion and manipulation with sparse and dense rewards.
In these experiments, we evaluate the performance of D2AC across a range of domains to assess its generality, efficiency, and behavioral characteristics. We also investigate how D2AC compares to strong model-based baselines selected tasks, aiming understand how effectively bridges the gap between model-free and model-based approaches.
Evaluation environments. We evaluate D2AC across three primary domains: (i) the DecpMind Control Suite (Tassa el al. 2018), which consists mainly of locomotion tasks with dense rewards, (ii) the Multi-Goal RL environments from Plappert et al. (2018), which include robotic manipulation tasks with sparse rewards, and (iii) a predator prey environment inspired by biological survival dynamics (Lai et al. 2024), which emphasizes adaptive behavior in dynamic and high-stakes scenarios. A list of the environments shown in Figure 2 Those environments include 21-DoF Humanoid, 38-DoF Dog, 12-DoF Quadruped, a 4-DoF Pick-and- task, and series of 20-DoF Shadow Hand tasks aiming to manipulate Block, Eggg and Pen to larget orientations.
use is
5.1 Benchmarking on dense-reward environments
In Figure we showcase our benchmarking results on 12 tasks the DeepMind control suite. We compare our mothod to a variety basclines, including: Soft Actor Critic (Haarnoja al. 2018). D4PG (Barth-Maron) let al. 2018), and the model-free component of TD-MPC2 (Hansen CDQ, because it essentially combines SAC, two-hot critic representation, and clipped double Q-learning. We also compare against DIPO (Yang et al.
2023). which we denote as Two-hot
2023) which is model-free RL method based on diffusion policy. D2AC achieves higher asymptotic performance across all 12 environments, and generally learns faster.
When compared to TD-MPC2 Hansen et al. 2023 trained with the same number environment interactions, we see that D2AC is very competitive despite being model-free. In Figure 4 we see that D2AC achieves performance within five percent of TD-MPC2 on 8 out 12 benchmark environments. In 4 of the environments, the performance D2AC exceeds TD-MPC2. The two standout environments were Dog Walk and Humanoid Stand, where D2AC significantly outpaced TD-MPC2 performance. This comparison is particularly revealing when considering the performance of the Two-hol CDQ baseline, which represents the model-free component of TD-MPC2. Its relatively poor performance suggests that the strength of TD-MPC2 does not lie in its model-free foundation alone, but heavily relies on its sophisticated planning module. In contrast, D2AC closes the performance gap to this state-of-tho-art model-based method without any planning. Our proposed model- free learning algorithm is orthogonal to any planning algorithm and can be used as drop-in replacement any model- based method, such TD-MPC2
8

---

Published in Transactions on Machine Learning Research (10/2025)
Walker Run
Hopper Hop
Finger tur_eesy
Finger turn, hard
300
Rumber
10
50
680
125
000
6.7
140
Quadruped Walk
Humanold Stand
Humanoid Walk
Humanold Run
200
Dog Stand
Dog Walk
Dog Run
Dog Trat
40
200
02
Nanonofraae
16
10
0.5
10
D4PG
SAC
DIPO
TwoHot
D2AC
Figure 3: Experiments on DeepMind Control Suite. Results over 5 seeds. In model-free RL, D2AC achieves much better sample efficiency and asymptotic performance compared to all other basclines.
D2AC is highly performant in a wide variety of settings: D2AC achieves significantly better sample efficiency and asymptotic performance compared to all other model-free baselines we tested. This is true across multiple control modalities (grasping, locomotion, manipulation). Thanks being model-free, our Figure 10 for clear comparison and see Appendix A for detailed discussion. In particular, D2AC can make significant progress the Dog Trot task within 24 hours, where TD-MPC2 has barely started increasing its episodic return; SAC completely fails on this task.
method is considerably faster to run compared to TD-MPC2:
Comparison Against Model-Based TD-MPC2
TDMPC2 (model-besedd)
SAC (model-free) (model-free)
Performance
tan, lnoal
Figure 4: Comparison between model-based TD-MPC2 Hansen et al. and our method D2AC on DeepMind Control Suite. D2AC without planning can achieve results on-par with TD-M
2023
SAC (Haarnjja et al. 2018
Overall, these results challenge the notion that the recent success of model-based RL stems entirely from explicit planning with rollouts. The strong performance of D2AC suggests that the core principles often associated with model-based planning-spedde and to iteratively refine action proposals-can be effectively distilled into a purely model-free framework.
the ability to model a distribution of possible returns
5.2 Benchmarking on goal-conditioned RL environments
In Figure we present results on robotic manipulation tasks with sparse rewards. All methods are trained with Hindsight Experience Replay (HER) (Andrychowicz et al. 2017 The baselines include: DDPG [Lillierap)

---

Published in Transactions on Machine Learning Research (10/2025)
Fetch Push
Fetch pick and place
Fetch Slide
1.0
06
Rate
0.0
0.8
0.4
0.4
0.
0.0
0.3
3000
10003
70000
200200 200055
30
233
20
030
102
Hand manipulate block -lock
Hand manipulate egg
Hand manipulate pen
1.0
0.80
0.4
0.
0.2
1.0
Numberdd Trarrss
Namber
Number of Fiames
BVN
D4PG
DDPG
D2AC
Figure 5: Experiments on Multi-Goal R.L environments with sparse rewards. Results over 5 seeds.
2015 (the original algorithm used in the HER paper). distributional version of HER |Eysenbach 2019), and recently proposed Valinca-VValue Network (BVN) (Hong et al.) 2022 found to be effective in goal conditioned RL. We use a centralized learner and 20 sampling workers for all methods. We find that on Fetch tasks, D2AC performs better or on par with the previous SOTA method, BVN, On 20-DoF Hand Manipulate tasks, the sccond-best method still DDPG, and our method can outperform by a large margin, often both in terms of learning speed and policy performance at convergence. This performance shows that D2AC is strong base algorithm that can work out-of-the-box in variety of settings.
et al.
ct al.
Interestingly, the previously proposed distributional critic function in Eysenbach et al. 2019 achieves worse performance than non-distributional critics. We hypothesize that this is because its specialized type of goal-conditioned distributional Bellman backup uses a one-hot target the moment a goal is reached, making it more difficult for the agent to learn how to consistently stay at the goal. Our distributional critics do not make any particular assumptions about the problem structure of goal-conditioned RI, but still achieve very strong results in this setting.
5.3 Bnspiey Benchmark: Behavioral Adaptation in Predator-Prey Scenarios
Retum Chube
Figure 6: Left: Predator prey arena arena(hexagon (hexagonal maze, start to goal, +1 at goal, when predator is within 0.1 units, matching the moule water /airr //aaa-riuutt setup). Middle: Prey's egocentric view on Map Level 5 Predator's egocentric view on Map Level (RI. sim).
(RL sim). Right:
While standard reinforcement learning benchmarks primarily focus on task completion or reward maximization, they often overlook structural differences in the learned policies. To address this limitation, we introduce predator prey environment inspired by biological survival dynamics (Lai et al. (2024). Such scenarios serve foundational models in both natural and artificial systems for analyzing adaptive decision-making
10

---

Published in Transactions on Machine Learning Research (10/2025)
under pressure (Marrow al. 1996 Han al. 2025). This domain provides natural testbed for analyzing the internal structure of learned policies, including diversity and strategic flexibility to adversarial changes, qualities often hidden behind reward curves.
TDMPC2
D2AC
SAC
Figure 7: Illustrative example of state density for TD-MPC2, D2AC, and SAC in the prodator-prcy environ- ment. Each visualization corresponds to 2000 sampled episodes from one representative run, where purple indicates the agent's state density as the prey reaches the goal while avoiding predators. Quantitative results in the main text are averaged over 5 independent runs.
Table 1: Exploration Coverage Analysis over five seeds (values reported coverage % + standard deviation)
Algorithm Visit Coverage (r 2 1)a
Main Coverage (r 2 10)
SAC
35.07 + 6.07
10.18 + 1.18
TDMPC
31.32 + 11.64
10.09 + 1.76
D2AC
44.58 + 3.16
20.04 + 0.49
Notes. "tates visited least
"States visited least times.
Metrics: We train three agents (TD-MPC2, SAC, and our proposed D2AC) for 50,000 steps on Map Level and evaluate: (i) Exploration Coverage (Table through visit coverage and core-arca coverage; (fi) Survival Rale over 2,000 episodes; (iii) Zero-Shot Transfer to unseen Map Level 9 (Table 2
Table 2: Zero-shot transfer performance from Map Level 5 (trained) to Map Level 9 (unseen). Results are reported as survival rate % standard deviation over five seeds.
Algorithm
Level 5 (Trained)
Level 9 (Zero-Shot)
Change (%)
TD-MPC2
72.33 + 1.95
62.35 1.36
-9.98
SAC
86.37 + 3.23
75.95 + 7.41
-10.42
D2AC
87.05 + 4.24
90.69 + 3.56
+3.64
Results and Analysis: As shown in Table 1 D2AC achieves the highest visit coverage (44.6%, 9.5% over SAC and 13.2% over TD-MPC2) and doubles TD-MPC2's core coverage, suggesting more comprehensive exploration. Interestingly,
observe potential connection between exploration patterns and generalization: In zero-shot transfer (Table 2) both TD-MPC2 and SAC degrade by approximately 10%, while D2AC improves by 3.6%. Figure 7 further reveals that TD-MPC2 and SAC exhibit relatively consistent trajectories, while D2AC demonstrates more varied paths and appears to adapt more dynamically to predator movements. These findings suggest our framework not only achieves competitive survival rates but may also develop more flcxible behavioral strategics.
Insights on D2AC's Performance: The results suggest D2AC's approach offers advantages in this envi- ronment. We hypothesize that the exploratory benefit emerges from our architecture the distributional critic provides richer uncertainty information while the diffusion actor generates diverse action proposals through
11

---

Published in Transactions on Machine Learning Research (10/2025)
iterative sampling. However, we acknowledge that higher exploration doesn't universally guarantee better performance-in in some domains, focused exploitation might be preferable. The predator-prey environment specifically rewards strategic diversity, making it suitable for showcasing D2AC's exploration characteristics.
5.4
Distributional Critic vs. Diffusion Actor
Quadrupeci Walk: P(DZAC
Finger-Tur_baer hard: PJD2AG Y]
Humanoid-Walk: PID2AC
Dog-Trot: PIDZAC Y)
DMC
Agxtert
D4"0
PIDRAC Yyd.8
PIDDAC V1+
PIDSAC + Yyd
PODMA + Vir03
Telto
080
Frobobilty
Probabiity
Probabilty
Figure 8: Ablation Studies based Probability of improvement (Agarwal et al. 2021 using the recommended protocol from rliable. The X-axis gives the probability that D2AC improves upon the performance of the baseline algorithm on the Y-axis, The boxes represent 95% confidence intervals for each baseline.
First, we ask whether our diffusion actor is indoed critical. We test this by introducing DIAC, which uses our clipped double distributional critic but standard Tanh-Gaussian actor. As shown in Figure g P(D2AC Y) is consistently above 0.5 for all choices of algorithm Y. In particular, the high probability of D2AC improving over DIAC (purple bars) directly demonstrates the benefit of the diffusion actor.
Beyond the actor, also observe that D1AC remains competitive against strong distributional baselines such as D4PG and the model-free component of TD-MPC2 (Two-hot+CDD), This seems to suggest. that our proposed critic design provides substantial gains.
Hence, to directly isolate the contribution of our critic design, we conduct an ablation that disentangles two key components: the distributional return modeling and the clipped double Q-learning mechanism. We evaluate four variants our full D2AC, version without clipped double Q-learning, scalar critic with clipped double Q-learning, and scalar critic with single Q function across three continuous control tasks.
Quadruped
Dog
1000
1975
3000
632
632
Roum
612
202
Number Tremmss (x129)
Number Tramess
Number Framme 1x109
gina D'AC
Distiiutioni: Singl
Scalar
Clippdd Double
Scatar Singls
Figure 9: Ablation of critic components three locomotion tasks. Both the distributional critic and clipped double Q-learning are essential, and the full D2AC achieves the best performance.
As shown in Figure g the full D2AC consistently outperforms all ablated versions. Removing either the distributional critic or the clipped double Q-lcarning leads to clear degradation, while eliminating both components results in the weakest performance. These results provide direct evidence that both mechanisms make additive and indispensable contributions to stability and overall performance.
12

---

Published in Transactions on Machine Learning Research (10/2025)
5.5 Limitation of Our Method
Fundamentally, method is still model-free RL method; denoising score matching cannot fully substitute for the role of planning. Although the denoising diffusion policy improvement we propose already has a lot of similarity to MPPI (Willaas et al. 2015 2016) in terms of optimization objective, MPPI is explicitly guided by the critic function during the sampling process, and simultaneously considers multiple possible trajectories. We expect that explicit value- based planning should be able further enhance diffusion performance, but leave it for future work.
more expressive policy class based
6 Conclusion
We have presented D2AC, novel model-free RL algorithm that successfully integrates a powerful diffusion actor with a stable distributional critic. Our core technical contribution is a new, theoretically-grouddee policy objective that makes training diffusion actors in an online sctting both stable and efficient. This was enabled by our second contribution: a carefully designed distributional critie, stabilized with clipped double Q-learning, that provides the robust learning signal necessary for our actor thrive. This synergistic design leads to statc-of-the-art method that excels on both sparse and dense reward tasks, runs significantly faster than model-based competitors, and demonstrates superior exploration and generalization challenging biology-inspired benchmark.
There exist several exciting avenues for future work. D2AC can be plugged into any model-based methods as the model-free RL component; we aim to explore how simple planning modules can boost its performance. While we built upon the C51 framework for our distributional critic, recent approaches like classification-basell value learning [Farebrother al., 2024) and histogram loss (Imani et al.] 2024) offer promising alternatives that could potentially enhance D2AC's performance further. Finally, we interested in improving the
computational efficiency of D2AC by investigating faster diffusion models (Song et al. 2023 Song & Dhhriwal
2023
Chadebec ct al. 2025 and beyond.
Acknowledgments
We are grateful to Duruo Li (Department of Statistics, Northwestern University) for her valuable assistance with carefully verifying the correction of Proposition 1.
References
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXio preprint arXiv:1806.0688 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 14:29344-29300 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Jimmy Lei Ba, Jamic Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607. 1607.66450 2016.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arxiv: (oo:1804.08611, 2018
Mare Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pp. PMLR, 2017.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprinl or Xin:2330. 13300, 2023.
Clement Chadebec, Onur Tasar, Eyal Benaroche, and Benjamin Aubin. Flash diffusion: Accelerating any conditional diffusion model for few steps image generation. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 39, pp. 15686-15695, 2025.
Yuhui Chen, Haoran Li, and Dongbin Zhao. Boosting continuous control with consistency policy. arXiv preprint arXiv:2310.0623 2023.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint orxiv:2003. 0.0000....00 04137, 2023.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. Advances in Neural Information Processing Systems, 37:53945 53968, 2024.
Xiaoyi Dong, Jian Cheng, and Xi Sheryl Zhang. Maximum entropy reinforcement learning with diffusion policy. orXiv preprint arXiv:2502.1 1 11612, 2025.
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga, Yovgen Chebotar, Ted Xiao, Alex Trpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. orXiv preprint ar *Xiv:2403.03956 2024.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Chen-Xiao Gao, Chenyang Wu, Mingjum Cao, Chenjun Xiao, Yang Yu, and Zongzhang Zhang. Behavior- regularized diffusion policy optimization for offline reinforcement learning, arXiv preprint ar arXiv22502.04776 2025.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861 1870. PMLR, 2018.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04004 2023.
Shuo Han, German Espinosa, Junda Huang, Daniel Dombeck, Malcolm A MacIver, and Bradly C Stadie. Of mice and machines: A comparison of learning between real world mice and rl agents. orXic preprint 1xXi::2505.1224, 2025.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprinl arxiv:2003. 04955, 2022.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint 2rXii:2310.16895 2023.
Philippe Hansen-Fstruch Tlya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. orXiv preprint arXiv:2304.10573 2023.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lilicrap,, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. Advances neural information processing systems, 28, 2015.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advonces neural information processing systems, 33:6840-6855 2020.
Zhang-Wo Wei Hong, Ge Yang, and Pulkit Agrawal. Bilinear value networks. arXiv preprint arXiu:2204,13695 2022.
Ehsan Imani and Martha White, Improving regression performance with distributional losses. In International conference on machine learning, pp. 2157-2166. PMLR, 2018.
Ehsan Imani, Kai Luedemann, Sam Scholnick- Hughes, Esraa Elelimy, and Martha White, Investigating the histogram loss in regression. arXiv preprint arXiv:2402. 13425, 2024.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. International Conference on Machine Learning, pp. 9902 9915. PMLR, 2022.
Paul Jeha, Will Grathwohl, Michael Riis Andersen, Carl Henrik Ek, and Jes Frellsen. Variance reduction of dif- fusion model's gradients with taylor approximation-basee control variate. orXie preprint arxiv:2408. 12270, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space diffusion-based generative models. Advances in Neural Information Processing Systems, 1-26565-26577, 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-21 217077 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1812.6114 2013.
Alexander T Lai, German Espinosa, Gabrielle E Wink, Christopher F Angeloni, Daniel A Dombeck, and Malcolm A MacIver. A robot-rodent interaction arena with adjustable spatial complexity for ethologically relevant behavioral studies. Cell reports, 43(2), 2024.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint XrXim:1805. 00909, 2018.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Contimuous control with deep reinforcement learning. arXiv preprint Xr 1rXi::1509.00. 2015.
Tong Liu, Yinuo Wang, Xujie Song, Wenjun Zou, Liangla Chen, Likun Wang, Bin Shuai, Jingliang Duan, and Shengbo Eben Li. Distributional soft actor-critic with diffusion policy. arXie preprint XXXi:2557700181.. 2025.
Πya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXio preprint Xrx::11111.0510 2017.
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Soft diffusion actor-critic: Efficient online reinforcement learning for diffusion policy, orXiv preprint orXiv:2502.00361, 2025.
Paul Marrow, Ulf Dieckmann, and Richard Law. Evolutionary dynamics of predator-prey systems: an ecological perspective. Journal of mathematical biology, 34:556 578, 1996.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 -532, 2015.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Amusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1112. PMLR, 2020.
Razvan Pascamu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. Pmlr, 2013.
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference Machine learning, pp. 745 750, 2007.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrow, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for rescarch. arXiv preprint arXiv: 1 1802.09444 2018.
Ralf Römer, Lukas Brunke, Martin Schuck, and Angela P Schoellig, Safe offline reinforcement learning using trajectory-level diffusion models. In ICRA 2024 Workshop {(teetemdash) Back to the Future: Robot Learning Going Probabilistic, 2024.
Julian Schritlwieser, Joannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839))604 609, 2020.
John Schulman, Sergey Levine, Picter Abbecl, Michael Jordan, and Philipp Moritz. Trust region policy oplimization. In International conference on machine learning, pp. 1889 1897. PMLR, 2015.
John Schulman, Xi Chen, and Picter Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprinl ar Xrxi: 704.06440, 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (rXiu:1007006347, 2017b.
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pp. 30365 30380. PMLR, 2023.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. ar preprint arXiv:2010. 02502, 2020a.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXie preprint 2rxiv:2310. 0...23100.188... 14189, 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neurol information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishck Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint ar 2ixi:2201 2020b.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. arXiv preprint arXin:2308.01469, 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: NXX:1880000099 2018.
Hado Van Hasselt. Double q-learning. Aduances in neural information processing systems, 23, 2010.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAT conference on artificial intelligence, volume 30, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compulation, 23 (7):1661-1674, 2011.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint 2rxi1:2311. 12908, 2023.
Tingwu Wang, Xuchan Bao, Tgnasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint ar PrXf::107.02005 2019.
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling, ar preprint arXiv:1509.014 00090, 2015.
Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou. Aggressive driving with model predictive path integral control. In 2016 IEEE International Conference Robotics and Automation (ICRA), pp. 1433-1440. IEEE, 2016.
Long Yang, Zhixiong Huang, Fenghao Lci, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint 2rx 2023.
Mingqi Yuan, Bo Li, Xin Jin, and Wenjum Zeng. Ultho: Ultra-lightweight yet efficient hyperparameter optimization in deep reinforcement learning. arXio preprint ar xxXi:2550306100 2025.
A Analysis of Computational Cost
To analyze the computational cost of D'AC, we compare its wall-clock training time against both the model-based TD-MPC2 and the model-free SAC (Figure 10 The results highlight D2AC's efficiency profile and its relationship with sample efficiency, as established in our main results (Figure 3
Computational Profile vs. Baselines: Our analysis reveals two distinct computational charactoristics. First, as a model-free method, D2AC is dramatically more computationally efficient than the model-based TD-MPC2, It achieves superior or comparable performance fraction of the wall-clock time by avoiding the significant overhead of world model learning and online planning.
Second, when compared to the simpler model-free SAC, clear trade-off emerges. SAC's lightweight architecture gives it a speed advantage in wall-clock time on less complex tasks. This is an expected consequence of D2AC's more powerful components, the diffusion actor and the distributional critic, which inherently require more computation per training step.
Justifying the Computational Overhead with Sample Efficiency: The crucial question whether this additional computational cost is justified. The answer lies in the significant gains in sample efficiency that A2 achieves. As shown in our primary results (Figure 3), D2AC consistently extracts more information per environment interaction, leading to higher asymptotic performance.
This trade-off is most evident on challenging (asks like Humanooid Walk. While SAC is faster per iteration, its poor sample efficiency prevents it from solving the task. D2AC's computational investment per step, in contrast, enables level of sample efficiency that is sufficient to master the task. Therefore, D²AC's design makes a highly effective trade-off: it modestly increases par-sample computation in exchange for the capability to solve complex problems that are intractable for simpler, faster methods.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Quadruped Walk: Run Time Comparison
Dog Trot:
Humanoid Walk Run Time Comparison
Run Time Compartson
800
300
800
600
600
000
Rgisode
400
280
200
200
2
10
12b
12.
16
20
2.2
12
16
20
24h
Puan Time
Rua Time
Run Time
TD-MPC2
SAC
Figure 10: Wall-clock runtime comparison against TD-MPC2 and SAC on a single GPU, The figure highlights the differing computational profiles across tasks of varying complexity. While D'AC is substantially faster than the model-based TD-MPC2, its imitial learning speed is surpassed by the simpler SAC on some tasks, illustrating a trade-off between computational overhead and the capacity for sustained learning on complex problems.
B Number of Diffusion Steps at Training vs the Final Model Performance
With respect to the diffusion actor, we are also interested in understanding the relationship between the number of diffusion steps at training vs the final model performance. In particular, we can consider both Ktrasn, the number of diffusion steps used during training, and K, the number of diffusson steps used at inference. In Table 1, we that there some moderate improvements to the final results when training longer diffuusio horizons. Our results agree with the common practice in diffusion models (Song al., 2020a; Karras et al.) 2022) of using more diffusion steps for training and for inference. In addition, we also find that setting #ref = T as done in D2AC policy loss equation 15 rather than just using the actions from the replay buffer &el + To slightly improves results.
Table 3: Performance Statistics at 500K Environment Steps
Configuration
Quadruped Walk
Humanoid Walk
Ktrašn K
778.6 227.6
378.7 + 52.5 425.1 39.9
Ktrain K
941.5 9.7
Ktrain 5. K =
928.8 = 15.9
386.9 + 59.8
Kttran K = 5 where Fea + n
829.9 + 127.5
314.2 + 76.5
C On Two-hot Functions
The function hr, called the two-hot function. Mathematically, hz, (2) defined as:
2 < Vmin and A = O,
2 S nn kk 1).
2 n
Zylk: 1|
Zg [k]
he, [k] =
2,
2g
2, k+
2, [k] < = E 2.y 1};
2, 14 +
2
0
z n, lk 1}:
= 2 Vmax and k = N.
Which is a piece-wise linear function with the following properties. Vz, if Vmnnn S 2 S Vmaxs
N
NV
hz, (2)[b)= =1
Mz (z)[s 2g[5 =
(16)
0=
1=0
18

---

Published in Transactions on Machine Learning Research (10/2025)
Which means that for any value, the two-hot function will distribute probability mass in the [Vmin: Vmax] region that sums up to 1, with weighted mean equal to itself. Note that the two-hot function is not the only function that satisfies those properties; for instance, the HL-Gauss parameterization proposed in dlmani & White 2018 can also satisfy those two conditions.
Lately, different type discrete critic based on two-hot representation has gained popularity from the success of MuZero (Schrittwieser et al. 2020), Dreamer-v3 (Hafner al., 2023), and TD-MPC2 (Hansen To draw the connection between the two types of discrcte critics, first notice the following:
et al.
2023
N E j=0
N Σ + +2,))) p(A j=0
Σ pbu =1 ==
r+
pll
Tt follows that: two-hot discrete critic learning can be reformulated as simply swapping the order between the sum and the two-hot function in the projection operator of distributional RL:
N
dlist (Zp, P. [b]]
(ZpLi)MA pbj
f=0
N
twohot (pp, P. [b)) hay
2 > 5=0
p(1] [M]
And the two-hot label is:
a' TTl (s') lewohot = + Ptwohoo (r + 22g 97(s', a'), 2n))
L(0) - e log (s(s a)
In comparison, distributional RL can be seen as using soft labels rather than hard labels; it allows the uncertainty about future returns to propagate through Q-learning. Two-hot discrete critic learning does not allow such uncertainty to propagate between different timesteps through Bellman backup.
D Policy Gradients for Explldce- (VE) Diffusion
Let one step of the diffusion process a + a (k-1) for policy TH be:
To (a (k-1) (k) s) = N(aa (k-1) (b-11
( k;s),, - o
(s(a k;s) = (N) + +o Va logp(a!)) (ox
V. logo(pa o,s) (Ds(a,o;s) a)/o²
where we can see that
(ot - o%-
(k(a(k), k::s) a
(al(a())oxxs0 - aa(
o
T a
(K)
(0 oh-
Do (k) ,x;8))
ok
oz
In addition, let the latent variables a (1)...
(K)
(K)
bc:
a(a (k) a) = (a( a, o
Then we have the marginal distribution denoted as:
p(a ((a)))= / Pdata (k) aada
19

---

Published in Transactions on Machine Learning Research (10/2025)
Under the assumption that max is sufficiently large, the initial noise distribution is pre-defined as: p(x(K)) = NO, o max I)
The full policy distribution is:
K
Ta a 8) = f p(a(k)) II
ne (a((a))))
8, (*)) (d)a.
The distribution of (*) given a0),, for k > i, is given by:
((a(k) (K) a (5) .² |a (t), (o - I
Since the forward process is Markov, have
a(a (k) a (0)
a) = q()
a) q(a()) a)
The posterior distribution can be written as:
a(a (3)
a (k) a)
Which can computed in closed form using the conjugate prior of Gaussian distributions:
q(a()) (0) a (K) a) = =N(:
of
(o of)
{ of) a,
a
(b)
og
o
o
Then Jensen's inequality gives us the following lower bound (Sohl-Dickstein et al.
2015
Ho al. 2020):
E gl DEL (9(a (a( a) To (a (M) .s)
log no (a s) S Lprior
prior DKz(g(a a) p(a (k))
From the result of Kingma et al. 2021 we know that
of
-o--1
(k))aa) D (()) a) n (a (k) s))
a Do(a(h) s); {))))
20
of
1
||a Do(a(1) ,xx:)]1
2
0k
Allowing us to arrive the following result:
log T (als) exp Q(s, a) Un ( Do(a (xx,, Q(s, a)]
where the weightings for policy gradient objective Apg are:
Apg (k) (1/02_1 1/07) (K/2)
E Proof of Proposition 1
Proposition Define the data distribution under diffusiion step le 2 0 be (with à 2 amin):
Pa(a|s)= /
(17)
(a(a)s) Put (411s)) 6T)ju
(18)
20

---

Published in Transactions on Machine Learning Research (10/2025)
Under the assumptions that (t) the entropy of phs and Py strictly increases with k; (8) the KL distance from P& and Pu to the optimal policy p' is non-decreasing with k; (iii) à is sufficiently large such that (Ro ll p1) 2 DRL(PO l| p²). Then for any state and diffusion step & 2 O, we have
B [Q] 2 Bp [Q]
Proof. Fix an arbitrary state E S. Because the Boltzmann optimal policy satisfies
pd pa s) cxp(Q(s,a))
Z(s) - exp(Q(s. da',
((s)
Tt follows that Q(s, a) log ((s) log (a 8). For any distribution g(: () this identity yields
E [Q] log (()) (K a ll 1²) -H(g):
(19)
19 to Po and = Px and taking their where H(-) denotes differential entropy. Applying equation
difference gives
Eo [Q) Ep, [Q] = DRL (fx Ilp') DEL (po l1 H(fr) H(po)
(20)
(Ak((A)
(A)
KL term. Assumption (ii) states that the function k + DEL (pellp") is non-decreasing. Thus, for any A 2 0, DEL (pxll"))) 2 DKL (Polle"" Assumption (iii) provides that DKL (po llr') 2 DKL (poll"))" Combining these inequalities yields the following chain:
DKL. (fell"))) 2 DKL. (foll")lp") 2 DKL. (poll"")
which implies that (KL(k) 2 0 for all h 2 0.
Entropy term. Assumption (i) states that entropy strictly increasing with k which ensures H(px) 2 H(Po). The analysis thus reduces to showing H((p)) 2 (po).
Let A x(- s), and let so (N0, and é (2) be independent Gaussian noises. By definition, the distribution of ( en pl, and the distribution of Y D(A(0) Eo; 01; 8) + 61 is po-
For independent random vectors X and Z, Cover 1999. Lemma 17.2.1) asserts that
H(X Z) 2 H(X).
(21)
Applying equation 21 with X = D(A + s) and Z = gives
M(()) H(Y) 2 H (DA (0) + 5 E))o)))))
Furthermore, since a deterministic mapping cannot increase entropy, and by Assumption (i), we have:
H(D(A + Eo. 01;5)) 2 H(P)) > H(Pv).
Chaining these results together yields H((p)) H(po), which in turn implies that (H(k) = H((p)) - H((p) 2 0 for all k > 0.
Since both terms on the right-hand side of equation 20 are non-ncgative, their sum is non-negative. Consequently,
Conclusion.
E [Q) Epp [Q] 2
Ep. QQ)
Ep, (Q),
Wk 2 0.
This completes the proof.
21

---

Published in Transactions on Machine Learning Research (10/2025)
F MPPI
Model Predictive Path Integral Control refinement (Williams et al. 2015 2016 sampled from noise distribution:
(MPPT) applies an iterative process of sampling and return-weighted
Nagabandi et al,, 2020 Hansen al. 2022). Initial trajectories are
00 K- ~ ~- Mi=0, Po
=
1 = 0
0 = d max
At the i-th ileration, MPPI samples K action sequences x
k=0-K-1 from
N(: p;
uses stochastic (virtual) rollouts to estimate the K empirical returns each action sequences { It then updates the sampling distribution according to
K-1
[K-
exp(Ra/At A))x
Σ
exp(Rx/A))
#++1
of
exp(Rx/A))(x) 1=+1)²
Σ
exp(Re/A)
MPPI uses a distribution of returns to update the action proposals to higher-relurn regions.
G On Tanh Squashing of Actions
The most commonly used action distribution for continuous control is Gaussian distribution with tanh squashing (Haarnoja et al. 2018 The policy T outputs the mean pg and log sigma, which we exponentiate to get dg.
u N( (s),,,, (s))
tanh(u)
(22)
where u is sampled using the reparameterrization trick (Kingma & Welling 2013 = ph teMo ~ N(O0
Applying the change of variable formula, The change of variable formula says
log ((a s) logNN(u pho 2 Σ log(1 - tanh?(u.))
And the second part can be written as:
log(1 tanh (ann2(u))) (u))) log u
2og 2 2log(e" + e =)
2og 2log(e + 2u))
=2 - 2u log((1 + 22))
2log 2 - 2u softplus( -2u)
2[log u - softplus(-225
Thus, we use an easy-lo-compute and numerically stable form of the log likelihood of the Tanh Gaussian distribution:
pent (u, μ1, σr) = log.NN((( μ, 02)22= - 2 1 [log 2 u softplus(- -2u)]
(23)
Thus, we can use the squashing enforce the action boundary, and apply entropy regularization (Haarnoja et al. 2018):
L(a) = -alssat Aene [A]] We apply to policy network similar to other model-free baselines.
22

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm Policy Optimization (under Tonh Action Squashing)
procedure UPDATE(Ó, a 8, a,
Input: policy parameters 9, entropy coefficient a.
Input: state s, continuous action (-1,1).
Input: differentiable ((s, a), noise level a.
I), u = tanh - (a) t6.0 As = (s, ú, 0), σΦ (s 0, X ~ N(. Pe. 0%)
x target = + VxQ(s, tanh(x)) x=x,
Eq equation 15
font- font (xp, Op)
Eq equation
SGD(o, Vollxx x xtarget| 12 fo
a + (a, Vol- Val-a(fer foutt Aent (l)l)
Update temperature
end procedure
Algorithm 2 : D2 Actor Critic
procedure ACTORCRITIC(D, 0)
Input: (s, a, r') ~
Input: critic parameters 8, policy parameters ø.
Input: entropy coefficient a.
a' <oTo s')
Diffusion Sampling
Optimize with clipped double distributional RL equation ??
Uniff1.... Ktrain ). j Unif{1 K}
< a guamm_) oj x a x=)
Select noise levels in EDM D Alg
Update é, a on (s, a, o.) and (s' a',,) Target network update: + T + + (1-11p T)
b Exponential moving average.
end procedure
H Policy Network Parameterization
Let Pa (ū) PPataa (ū u. o I.du. The min miin and max maxx of noise levels are selected such that Pa s Pdata (u) and Paaax (ū) NOO, o We adopt the EDM denoiser parameterization (Karras
ct al. 2022
( o) = Caiil
(24)
+ Cout (o) Fg Ci (in(o)u, (o))
With the specific functions being:
Cakip (a) = 02 ddatallo" odura)
Cout (o) = a - v 0² + 02
Cin (o) 1/ 1/ 02
Croise = logg
This parameterization offers powerful inductive bias for an iterative denoising process that has been shown to work well. (s, a) uses the same network Fa except for the last linear layer. The conditioning of (noise(a)
is done via positional embedding (Vaswani et al. 2017
A sequence of noise lcvels is used Omin = 01 < 02 < < GMM = Omax- We follow EDM in setting o; = = ((-
for = (1,... .) and p =
o(1) =
111
+
2/pp 1/p)
max mim)
)
(25)
23

---

Published in Transactions on Machine Learning Research (10/2025)
Biology-inspired Environment Experiment Details
The experimental environment was designed to examine strategic escape behaviors in mice by approximating predator prey interactions within a hexagonal arena (Lai et al. 2024), similar but not identical to the spt described in Han et al. (2025). In our implementation, a custom robotic device acted as a predator-like pursuer, delivering brief air puffs as an aversive stimulus when the simulated mouse was "caught", while the mouse's task was reach a designated goal location without interception.
The agent operated in a continuous two-dimensional action space, where each action specified target coordinates (x,)) within the arena. Each simulation step corresponded to 0.25 seconds of behavior in the real-world mouse task.
The state was represented by a ten-dimensional vector that included the simulated mouse's position and heading, the robot's position when it was visible, the distance to the goal, and a binary variable indicating whether the mouse had just received an air puff.
And receive reward of +1 for reaching the goal and -1 when being caught. At the start of each episode, the predator initialized outside the prey's field of view at randomly chosen position. This initialization remained stochastic even when a fixed random seed was used, ensuring variability in the predator's 's starting state across episodes.
For all baselines, we adopted their official implementations and default hyperparameters (Haarnoja et al. 2018 Hansen et al., 2023). Specifically, both SAC and our method D2AC utilized standard episodic setting, while TD-MPC2 was configured using its original non-episodic formulation as presented in its peer-reviewed paper.
J Hyperparameters
Hyperparameter Tuning. D2AC demonstrates considerable robustness to most hyperparameters. For the majority of our experiments, we adopted standard values from prior literature for common parameters such as learning rates and network architectures, without performing extensive, task-specific grid searches. The consistent use of single core set of hyperparameters across a diverse set of domains highlights the general applicability and stability of our algorithm.
The most critical task-specific consideration the value range [Vmin Viax] for the distributional critic. These parameters define the support for the return distribution and must be set appropriately cover the plausible range of cumulative discounted rewards for given task family. As detailed in hyperparameter table set wider range for the high-reward DeepMind Control Suite tasks (Tass et al. 2018). whereas narrower range was used for the sparse-reward, goal-conditioned manipulation tasks (Plappert et al. 2018
Beyond setting a reasonable support for the return distribution, D2AC does not require laborious, per- environment tuning, underscoring its potential as a general-purpose reinforcement learning algorithm.
24

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Hyperparameters for D2AC
Parameters
Value
Batch size
256
Optimizer
AdamW
Loshchilov Hutter 2017
Learning rate for policy
0.001
Learning rate for critic
0.001
Learning rate for temperature
0.0001
a initialization
0.2
Weight Decay
0.0001
Number of hidden layers (all networks)
Number of hidden units per layer
256
Non-lincarity
Layer Normalization
Ba et al. 2016 + ReLU
Y
0.99
Aent
0.0
Polyak for target network
0.995
Target network update interval
for dense rewards and predator-proy
10 for sparse rewards
Ratio between optimization steps
for dense rewards and predator-proy
for sparse rewards
Initial random trajectories Number of parallel workers
200
4 for dense rewards and predator-pree prey
20 for sparse rewards
Update every number of steps in environment Replay buffer size
samne as Number of parallel workers
10 for dense rewards and predator-prey
2.5 x 106 for sparse rewards
[/mia \max]
[-1000, 1000] for DM control
200, 200] for predator-prey
[-50, 0) for Multi-Goal RL
Number of bins (size of the support zn)
201 for DM control
201 for predator-prey
101 for Multi-Goal RL
Umin
0.05
Tmax
2.0
Odata
1.0
P
M
2
Mtman
Noise-level conditioning
Position encoding on (10 log o)/4
Embedding size for noise-level conditioning
32
25


Reviews:
[{"review_id": "CC8EP4nXc0", "reviewer": "Reviewer_Dmak", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros:**\n\nThe design of the actor-critic scheme is based on a novel application and combination of existing techniques.\n\nThe authors demonstrate that the proposed algorithm has good performance on a range of RL tasks and has nice properties such that yielding policies that induce more diverse trajectories.\n\n**Cons:**\n\nI believe that some statements do not follow from the experimental results. For instance:\n- The last sentence of paragraph 3 in the introduction is not empirically verified. Unless I missed it, there's no ablation study where the distributional critic is changed to a point-estimate critic, or where the clipped double Q-learning mechanism is removed.\n- How is the claim of the last paragraph of page 8 suggested by the experiments?\n\nThe theoretical result presented in Proposition 1 seems strange to me. Notably, assumption (ii) doesn't make sense to me (how can larger variance leads to smaller KL divergence?). Isn't it the reverse? Also, it's not clear how assumption (iii) can always hold in practice.\n\nThe proposed algorithm is probably more computationally costly than other model-free algorithms. However, this point is not discussed in the paper.\n\nThe presentation of the experimental results should be complemented with a discussion on hyperparameter tuning (if any) and a discussion on computational times (with relevant information of compute specifications).\n\nThe writing could be improved to make it easier to read for a general audience and could be more polished to remove typos.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops an actor-critic algorithm using a distributional critic and diffusion-based actor. The novelties of the approach include:\n- Applying the idea of clipped double Q-learning to the distributional setting\n- Applying the monotonic improvement theorem of TRPO to the training of the diffusion-based actor", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe the paper would be strengthened as follows:\n- The connection between the proposed method and model-based RL method, mentioned in the conclusion, should be explained in more details. \n- The related work on model-free RL could be more comprehensive to include the baselines used in the experiments (e.g., SAC, DDPG), but also other classic algorithms (e.g., TD3). In particular, why use DDPG instead TD3, which is, I believe, a stronger baseline and also includes a mechanism to reduce overestimation?\n- The notations should be explained the first time they appear(e.g., Eq. 7).\n- I would suggest the authors to be more pedagogical and also recall the intuitions of the equations.\n- More discussion of Proposition 1 is needed.\n- In the experimental part, the authors should discuss hyperparameter tuning (if any) and computational times (with relevant information of compute specifications).\n- An ablation study is needed to confirm the usefulness of the distributional critic with the clipped double Q-learning mechanism.\n- In general, the paper should be more polished, since there are quite a few typos (e.g., \"opinated\"; \"where\" after equations doesn't need capitalization; \"give us give us\"; \"equation equation\"…)", "broader_impact_concerns": "There is no Broader Impact statement. However, given this type of work, I believe this is fine.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YbaFRPADYu", "reviewer": "Reviewer_EjNa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- The paper is mostly well-written, and the contributions are clear. \n- Combining distributional RL with diffusion policy is a novel and promising idea. \n- The paper extensively evaluated their algorithm across several domains and against several baselines, including model-based RL methods.\n- The proposed algorithm shows strong performance across several domains explored in the paper. \n### Weaknesses:\n-  I am still unclear about the computational complexity of the proposed algorithm. I know there are some run-time experiments in Figure 9, but it was only comparing with model-based, not model-free RL. \n- There are different ways and losses to learn a distributional critic that has been used for RL before, such as in [1,2]; the paper lacks discussion on those other options. \n\n[1] Imani, Ehsan, et al. \"Investigating the histogram loss in regression.\" arXiv preprint arXiv:2402.13425 (2024).\n[2] Farebrother, Jesse, et al. \"Stop regressing: Training value functions via classification for scalable deep rl.\" arXiv preprint arXiv:2403.03950 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new model-free actor-critic method that combines a distributional critic with clipped double Q learning and a parameterized actor based on a diffusion policy. \nThe paper also derives some lower bounds on their policy objective and introduces some simplifications to the diffusion policy to reduce the sampling steps required. \nThe paper evaluates the proposed algorithm across a wide range of benchmarks: DeepMind Control Suite, a biologically inspired environment, and some robotics tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would like to see more discussion on the computational complexity of D2AC with a runtime comparison to other model-free algorithms such as SAC. \n- I didn't fully understand the lower bound discussion Eq. 13/Eq.14, I think clarifying these parts more with more discussion on how to interpret this lower bound would make the paper better.", "broader_impact_concerns": "no concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fE4DokvHA1", "reviewer": "Reviewer_WCJ3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The proposed method is technically sound, although it is not new.\n- The writing is clear and easy to follow.\n- The experiments are rigorous, including classical dense rewards, sparse rewards, and a new biologically motivated environment to investigate the exploration coverage, which looks interesting. \n\n## Weaknesses\n- The proposed method has limited novelty. When it comes to distributional policy gradient methods, it is very common to consider the distributional critic loss, such as [1,2,3]. Also, the double learning technique is also used in the implementation of [2] as far as I know. It is easy to expect the performance improvement given the success of the distributional RL. The categorical distributional loss is also used in D4PG. On the other hand, in the model-free RL in the robotics literature, oftentimes, the actor is parameterized by a diffusion policy for an actor trajectory planning by leveraging the powerful expressiveness of the diffusion model. Therefore, combining these two ideas is not new, making this paper lack technical contribution. For example, it seems that [1] also considers the combination of diffusion policy and distributional critic. Although it is an arXiv paper, it is still useful to have a discussion about the difference from it. \n\n- The technical descriptions are inaccurate or even erroneous. In the third paragraph of Section 1, (Bellmare 2017) did not consider the distributional critic. The concept ‘distribution’ is used to characterize the return random variable instead of Q function, which is the expectation of the return. Still in this paragraph, it is thus incorrect to say ‘distributional Q estimation.’ In section 2, it is also not accurate to say ‘value distribution’. This error occurs many times throughout this paper. A lack of understanding of the mechanism behind distributional reinforcement learning (RL) is problematic in writing and explanation of the empirical results in a deep way. \n\n- It is not clear what the proposed diffusion policy's contribution is. Although I do not think there is anything new in the distributional critic part, I am trying to find some technical contributions in this section of the proposed diffusion policy. It seems that the content before equation (12) is known. The proposed simplifications may be new, although it seems straightforward by only conducting a one-step diffusion process. I kindly suggest the authors highlight the real contribution or the difference in the diffusion policy compared with others, which is more helpful to help me and readers to understand this paper. A large concern here is that the combination of the two techniques in this paper seems separate. For example, in the diffusion policy section, the authors still use the language of Q function, instead of explaining the mechanism in the context of learning the whole return distribution. This also reduces the technical contribution as far as I can tell. \n\n\n### Reference\n[1] Liu, Tong, et al. \"Distributional Soft Actor-Critic with Diffusion Policy.\" arXiv preprint arXiv:2507.01381 (2025).   \n[2] Duan, Jingliang, et al. \"Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors.\" IEEE transactions on neural networks and learning systems 33.11 (2021): 6584-6598.  \n[3] Ma, Xiaoteng, et al. \"DSAC: Distributional Soft Actor-Critic for Risk-Sensitive Reinforcement Learning.\" Journal of Artificial Intelligence Research 83 (2025).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper combines the distributional critic and diffusion actor to enhance the actor critic algorithm. This distributional critic is also equipped with clipped double Q-learning. Extensive experiments are conducted, including the classical locomotion tasks with dense rewards, manipulation tasks with sparse rewards, as well as a biologically motivated environment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please make changes regarding the weaknesses I found above. There are other comments on the changes.\n\n$q_\\theta(s, a)$ in Section 3.3 should be the probability of each bin, which is not clearly explained. \n\nSince D4PG also uses a categorical distributional critic loss, the only difference in the algorithm shown in Figure 3 between D2AC and D4PG is the diffusion policy. Is that correct?\n\nWhat is the relationship between a larger visit coverage rate and better performance? It seems that the methodology does not involve any explanation of the exploration. Be aware that a high exploration coverage ratio does not always imply superior performance. \n\nIt is challenging to parse the results from Figure 8. More detailed explanations are expected in the revised version.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)