Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning
Authors: Haque Ishfaq
Guangyuan Wang
Sami Nur Islam
Doina Precup
Venue: ICLR 2025 Poster (2025)
Keywords: Actor-Critic
Exploration
Reinforcement Learning
Thompson Sampling
Langevin Monte Carlo
Deep Reinforcement learning
Continuous Control
Primary area: reinforcement learning
TL;DR: We propose a principled exploration approach for actor-critic algorithms through critic learning using approximate Thompson sampling

Abstract: Existing actor-critic algorithms, which are popular for continuous control reinforcement learning (RL) tasks, suffer from poor sample efficiency due to lack of principled exploration mechanism within them. Motivated by the success of Thompson sampling for efficient exploration in RL, we propose a novel model-free RL algorithm, \emph{Langevin Soft Actor Critic} (LSAC), which prioritizes enhancing critic learning through uncertainty estimation over policy optimization. LSAC employs three key innovations: approximate Thompson sampling through distributional Langevin Monte Carlo (LMC) based $Q$ updates, parallel tempering for exploring multiple modes of the posterior of the $Q$ function, and diffusion synthesized state-action samples regularized with $Q$ action gradients. Our extensive experiments demonstrate that LSAC outperforms or matches the performance of mainstream model-free RL algorithms for continuous control tasks.
Notably, LSAC marks the first successful application of an LMC based Thompson sampling in continuous control tasks with continuous action spaces.

Published as a conference paper at ICLR 2025
LANGEVIN SOFT ACTOR-CRITIC: EFFICIENT Ex- PLORATION THROUGH UNCERTAINTY-DRIVEN CRITIC LEARNING
Haque Ishfaq", Guangyuan Wang", Sami Nur Islam, Doina Precup
Mila. McGill University
(haque. ishfaq, guangyuan. wang | @eaal. m wew.emmmggllll...mmmm magill.cca
ABSTRACT
Existing actor-critic algorithms, which are popular for continuous control rein- forcement learning (RL) tasks, suffer from poor sample efficiency due to lack principled exploration mechanism within them. Motivated by the success Thompson sampling for efficient exploration in RL. we propose novel model-free RL algorithm, Langevin Soft Actor Critic (LSAC), which prioritizes enhancing critic learning through uncertainty estimation over policy optimization. LSAC employs three key innovations: approximate Thompson sampling through distri- butional Langevin Monte Carlo (LMC) based Q updates, parallel tempering for exploring multiple modes of the posterior of the Q function, and diffusion syn- thesized state-action samples regularized with Q action gradients. Our extensive experiments demonstrate that LSAC outperforms or matches the performance of mainstream model-free RL algorithms for continuous control tasks. Notably, LSAC marks the first successful application of an LMC based Thompson sampling in continuous control tasks with continuous action spaces.
INTRODUCTION
We introduce a practical and efficient model-free online RL algorithm termed Langevin Soft Actor- Critic (LSAC), which incorporates distributional Langevin Monte Carlo (LMC) (Welling & Teh, 201 1) critic updates with parallel tempering and action refinement on diffusion synthesized trajectories. Our approach employs a distributional Q objective and allows diverse sampling from multimodal Q posteriors through the use parallel tempering (Chandra et al., 2019), making LSAC especially well-suited for continuous control tasks in environments like MuJoCo control tasks (Brockman et al., 2016) and DeepMind Control Suite (DMC) (Tassa et al., 2018).
Although Langevin-styl update is powerful for learning posteriors by performing noisy gradient descent updates to approximately sample from the exact posterior distribution of the Q function, when naively applied to continuous control settings, it meets with the following three challenges:
(C1) Multidimensional continuous action spaces. In continuous control settings, actions are typically continuous and multidimensional tensors. This makes computationally intractable naively select the exact greedy actions based on Q posterior approximations, often leading to sub-optimal performance and inefficient exploration.
(C2) Value approximation errors. While LMC update helps in better exploration (Ishfaq et al., 2024a;b). it may also lead to instability issues when coupled with deep neural networks (Dauphin al., 2014) due to overestimation bias of Q-function. Moreover, naive LMC might lead to similar actions being overly explored due to high correlation among samples from the LMC Markov chain nearby steps which in turn can lead to value approximation error (Holden, 2019; Vishnoi, 2021).
(C3) Low sample efficiency. Naive LMC updates, akin to many actor-critic frameworks, use a Update-To-DD Dotaa (UTD) ratio of which is the number of network updates actual environ- ment interactions. This limited UTD ratio often leads to underfitting the complex state-action
"Equal contribution

---

Published as a conference paper at ICLR 2025
representations in continuous control (Chen et al., 2021; Dorka et al., 2023). Relying solely on a single critic update per iteration from on on-policy experience is insufficient, as fails to leverage the diversity of the data space, thereby hindering critic learning.
Recently there have been several works (Dwaracherla & Van Roy, 2020; Ishfaq et al., 2024a;b) that provide provably efficient RL algorithms that rely on LMC-style updates. However, these algorithms are scalable only to pixel-based Deep RL environments with discrete action spaces. One challenge arises from the multidimensional continuous action spaces in continuous control environments, as detailed in (C1). LSAC addresses this challenge by eliminating the need to compute the maximum of Q values over the entire action space to select greedy actions (Ishfaq et al., 2024a). Instead, it employs distributional critic learning framework using LMC along with a Maximum Entropy (Max-Ent) policy objective (Eysenbach & Levine, 2022). Learning distributional critic further mitigates Q-value overestimation issue as detailed in (C2). To address (C3), LSAC further incorporates Q action gradient refinement (Yang et al., 2023) in diffusion synthesized state-action samples during the critic update. This introduces diverse and potentially high-valued synthetic acteonction pairs into the collected trajectories, thereby enhancing critic learning. The actor in turn benefits from more accurate value estimations and thus improved policy learning. We further use parallel tempering (Chandra et al., 2019) to allow sampling from multi-modal Q function posterior.
Furthermore, traditional continuous control benchmarks such as DSAC-T (Duan et al., 2023), REDQ (Chen et al., 2021), SAC (Haarnoja et al., 2018a), 8a), and TD3 (Fujimoto et al., 2018), while benefiting from heuristics such as noise perturbed actions sampling and entropy maximization, do not sufficiently emphasize principled and directed exploration in their design principles. On the flip side, LSAC highly exploratory nature and effectively increases state-coverage during training by virtue of using theoretically principled LMC based Thompson sampling.
1.1 KEY CONTRIBUTIONS
To address the aforementioned challenges, we propose Langevin Soft Actor-Crite (LSAC) that endows traditional Max-Ent actor-critic algorithms with LMC-style updates and multimodal posterior sampling techniques for Q function. We summarize our algorithmic contributions as follows:
Distributional Adaptive Langevin Monte Carlo. Incorporating LMC for updating the Q function significantly boosts exploration while simultaneously maintaining a crucial balance with exploitation. To address (C2), we first define a distributed Max-Ent critic objective inspired by Duan et al. (2023). Then, we employ distributional critic learning with the addition of adaptive LMC samplers.
Multimodal Q Posteriors. One downside of naive LMC updates is that potentially homogeneous
posterior samples are generated at adjacent gradient steps. This translates to sampling similar critic for adjacent areas of Q-function posterior and this high correlation restricts exploration within the policy space while using a single critic for policy update. Consequently, a much longer burn-in period (Roy, 2020) is necessary to ensure adequate mixing of the Markov chain. However, this extended burn-in period conflicts with the frequent updates required by the agent in complex exploration tasks. To overcome this challenge, we introduce a simplified version of parallel tempering or replica exchange method (Geyer & Thompson, 1995; Chandra et al., 2019) that helps with exploring different modes of the Q-posterior more effectively. This in turn diversifies the actions sampled by the Max-Ent policy.
Diffusion Q Action Gradient. In the off-policy model-free RL setting, directly sampling from diffusion policies can be prohibitively expensive (Chen et al., 2024), often requiring tens hundreds of iterative inference steps per action, particularly in the absence of pretraining with a diffusion behavior model. To address this challenge and resolve (C3), we explore an alternative approach to introduce diversity and multimodality without relying solely on policy learning. Our strategy incorporates diffusion synthetic data, as proposed by Lu et al. (2024), to enhance critic updates. By blending online data with synthetic trajectories, and refining actions within the diffusion synthetic buffer, we leverage the Q action gradient to effectively constrain synthetic actions within the support set of optimal actions. This approach reduces computational costs and ensures that synthetic actions effectively contribute to the stability and quality of critic learning, resulting in more accurate and robust value estimates.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARY
Markov Decision Process and Maximum Entropy RL. We consider Markov Decision Process (MDP) defined as tuple (S, A. Po. P. R) where S is a continuous state space, A is a continuous action space. Po is the initial state distribution, P
x A + S is the transition probability, R : S A + A(R) is the reward distribution function and y E (0, 1) is the discount factor. At each timestep t. the agent observes a state S E S and takes an action O₁ ~ x(a |st) E A following policy and transitions to the next state 81-1 according to $t+1 P(s $t, at) while receiving
the reward R(#; st: ot). For simplicity, we use the notation (s, () and (6'.a') as the current and the next state-action pairs respectively. Furthermore, we adopt ry to denote R(ss; 0z) and use (*( and Pa (s, a) to denote the state and state-action occupancy measure induced by policy IT.
While standard RL aims to find a policy that maximizes the expected cumulative return, in this work, we consider maximum entropy RL (Ziebart, 2010; Haarnoja et al., 2017) in which the objective function is augmented with the entropy of a policy each visited state sy:
Jx
E (e). + aH(((- [s))];
where H- s)) := Ba |s) |-] log ा (a s)] is the policy entropy and a 0 is a temperature coefficient. We denote the entropy augmented cumulative return from 8t, also known as soft return, by G, Ei= 7'Ir,
log π [a; s.)]. The soft Q-value of policy π, which deseribes the expected soft return of policy π upon taking action a at state 8t, is defined as Q" (8t, ax) == ra + BBBG.+ where the expectation is taken over trajectory distribution under policy n.
Langevin Monte Carlo (LMC). LMC is a popular sampling algorithm in machine learning that leverages Euler discretization method to approximate the continuous-ting time Langevin diffusion process (Welling & Teh, 2011). Langevin diffusion (Rossky et al., 1978; Roberts & Stramer, 2002) is a stochastic process that defined by the stochastic differential equation (SDE) dw, VV ddt v B2, where L R R is a wwice-iifferentiable function and B, is standard Brownian motion in IRd. Taking Huler-Murayama discretization of the SDE, we obtain the iterative update rule for LMC:
W++1= w₁ 1 ((w)) + V 3
(2)
where 77 is a fixed step size, 3 is the inverse temperature and 6, ~ (O), LMC update generates a Markov chain whose stationary distribution converges target distribution p(x) xx exp (w)) (Roberts & Tweedie, 1996). Intuitively, LMC can be thought as a version of gradient descent perturbed by Gaussian noise. Replacing the true gradient (((w)) with some stochastic gradient estimators results in the celebrated stochastic gradient Langevin dynamics (SGLD) algorithm (Welling & Teh, 2011).
Diffusion Models. Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015) are a class of generative models that were inspired by non-equilibrium thermodynamies and first used in image synthesis. They have recently emerged as a powerful framework for RL to enhance multimodal recision-making process (Wang al., 2023; Mlansen-Estruch et al., 2023). Given data marginally distributed as (xo), we sample from it by first defining a stochastic differential equation (SDE) dx: = f(() g(()dwr, 35n ~ ()
where We is the standard d-dimensional Wiener process. Diffusion models gradually add Gaussian > ou 0 such that Ie ~ qe (z,; ot) and gr ~ ~N, o I I) is indistinguishable from pure Gaussian noise. Equation admits equivalent reverse denoising process starting with the fully noised distribution (T()):
noise from t = 0 to T by setting noise schedules o Tmax = or 0r-1
dz, (f()) (t) = log (xx))) + g(t)dū,,
FT (xT),
(4)
Since the score function V log qe (a:)) each time step t is unknown, Karras et al. (2022) considers training a noise predictor cp (2,, t) on the score matching objective
(f) min ex. Do + + - x112. x
to predict the added noise that converts xo to zT. Then, the score function can be expressed as V, log Pa(:; (x: σ) = (D,(s; a) - x)/o², and we can generate synthetic samples by solving either the backward SDE in Equation 4 or using DPM solvers (Lu et al., 2022).
3

---

Published as a conference paper at ICLR 2025
3 ALGORITHM DESIGN
In this section, introduce Langevin Soft Actor Critic (LSAC), shown in Algorithm 1, which builds off three main ideas. First, during critic learning, we want to learn and efficiently sample Q-value function from its approximate posterior distribution. We leverage Langevin Monte Carlo (LMC) to perform this. This is a natural adaptation to posterior sampling or Thompson sampling that widely used in RL for efficient exploration. Second, we couple LMC based posterior sampling with distributional value function learning (Duan et al., 2023; Ma et al., 2020) that helps with mitigating the well-known overestimation issue. Third, ensure LMC can sample from different modes of Q-posterior, we use parallel tempering (Chandra et al., 2019). Fourth, to improve sample efficiency and the UTD ratio, during critic update, we synthesize diverse and potentially high-valued state-action samples using a diffusion model and Q action gradient refinement.
Algorithm 1: Langevin Soft Actor-Critic (LSAC)
Input:
Policy networks a a.. critic networks ZΦi, Ziz, and diffusion model M. Replay buffer D < 0. diffusion buffer D 0.
Collection of posteriors V/ z (v)(() 1== entropy factor o, initialize weights w0(1) for (1b0)) fi=1 Set step size 1ι2 > 0 and temperatures Sa, Sr, Br, 8Q, 8M.
while policy has not converged do for each sampling step do
L Online interaction with मे the environment, DU (s,,,, for each update step do for i i 1, n do
Sample mini-batch BD. from D and BM. {sMMsM aaa.aMMMM TM, s dM)} from D'.
Refine am with aM & aM - Ba Va Q (sM. aa) in BM.. Update Z on B; = Bp, U Bas with Algorithm 2.
Sample (+()) ( (z) at random and recover Bi.
10
Compute a with Equation 12 and update T& on B, with Equation 11.
Update M on D with Equation 10 and fill up D'.
Polyak update (6() Tw(() + (1 T) (() t TO + -)Φ.
12
Distributional Critic Learning with Adaptive Langevin Monte Carlo. To describe distributional critic update, first define few terminologies. We first define soft state-action return, random variable, given by 2d (st, az) == + GG which is a function of policy π and state-action pair (St, az). It is easy observe that (s) ([2" a)]. Instead of the expected state-action return (* a), we aim to model the distribution of the random variable (= a). We define 2 (" (s, a) s, a) x A P (s, a)) as a mapping from a) distribution over the soft state-action return 2 (s, a). We refer to this mapping as value distribution function. We define the distributional Bellman operator in the maximum entropy framework as
PP(
D T T" r (ZT a') - a log (a' (()))
(5)
We model the value distribution function and stochastic policy as diagonal Gaussian distribution and parameterize as (- (-(11 a) and | where 1/ and & are the neural network parameters. Due to Gaussian assumption, Zy can be expressed as [- s, a) = (QQ (s, a). (s a)?). where s,
Qu(s, a) and (s, a) are the mean and standard deviation of value distribution respectively. The distributional critic is updated by minimizing the following loss function:
Lz(v) == (. DKL (T (2(())))
(6)
where DKL is the Kullback-Leibler (KL) divergence, B is the replay buffer, and 0 and o denotes the target network parameters of Z and n. respectively. The gradient scalar w := E(s.a) (s, a depends the variance of the distribution function.
4

---

Published as a conference paper at ICLR 2025
Following Duan al. (2023), we decompose the critic update gradient into two components: mean- related gradient L2 and variance-related gradient (yLzz
V = JQ Qu(s, a)
V)Re(s,a)
av(s,a)²+ + co
- V (v,w(, V VoLz,((1) == - -
( + Eo
where the target terms yo and are defined pQ == r (Q,5('', a log ( (a'ls)) and yz := (Z d') -a log Tz(aa|s") (s)). yz is further clipped with a clipping function (Vip, = clip(yz. (x( b, Q (s, a) + b) where x ((s,a)BBo a) is an automated boundary.
Finally, we can express the sample-baser based critic update gradient
V ( 2 ((aa))BB V (t) + V VoLz,
In Appendix B, we show that, under some mild assumptions, the posterior over Q₂ is of the form exp(- T (()))T/)))))// (1)))/Z, where Z is the partition function, and that (s a) (() Z (s, a). However, exactly sampling from this distribution is non-trivial as we do not know the partition function. To this mean, we can use LMC based sampling algorithm. In place of vanilla LMC described in Equation 2, following Ishfaq et al. (2024a); Kim et al. (2022), we use adaptive Stochastic Gradient Langevin Dynamics (aSGLD), where an adaptively adjusted bias term is included in the drift function to enhance escape from saddle points and accelerate the convergence to the true Q posterior, even in the presence of pathological curvatures and saddle points which are common in deep neural network (Dauphin et al., 2014). Concretely, we use the following update rule
the nQ V (1)) + aGo.) V2n ck. 6k ~0, Ta),
(8)
for a step size nQ > O. bias factor adaptive preconditioner Sr. and inverse temperature $q- Inspired from the Adam optimizer (Kingma & Ba, 2014), the adaptive preconditioner So is defined as Syz m2 2 V + Xl where,
m& = a- + (1 VL2
and 2k = 02 + (1- (1- O V2 (VA), (9)
with 01, a2 [0. 1) being the smoothing factors of the first and second moments of the stochastic gradients, respectively. Each sampled d following Equation 8 parameterizes a possible distributional Q function and thus is equivalent to sampling from the posterior over distributional function.
While our critic update rule is motivated by Ishfaq et al. (2024a), we are the first to apply aSGLD based parameter sampling for continuous control task along with distributional critic. Moreover, while in each critic update step, we perform one aSGLD update, Ishfaq et al. (2024a), in their LMCDQN algorithm, performs this update ((K) times, where K is the episode number. This significantly increase the runtime of LMCDQN compared to that of LSAC.
Algorithm 2: Distributional Adaptive Langevin Monte Carlo
Input: Policy Ne and target Tzz critic weight &i and target v data batch B.
Sample ~ (0, Ia).
Update Zg, Zz and T (s, a) by Equation 5.
Set clipping boundary x E (s,))) (s, a), temperature w x mlo a)?].
Set Lz(v) := (. (T (s, (s, a)l Z (s, a)) by Equation 6.
Compute the adaptive drift bias Go = m 2 e + Al using Equation 9.
Update 2 posterior weights with 1/2 < 3 (V (xb)) + ac + +/ 2 Le.
Polyak update b rb + - a)). w Tw + (1 ()- (ll=))/ca)=Bll00(000)
Parallel Tempering and Multimodal Q Posteriors. Despite the scalability of LMC, its mixing rate often extremely slow, especially for distributions with complex energy landscapes (Li et al., 2018). Performing naive LMC to approximately sample from multimodal Q posterior can thus converge very slowly which in turn will affect the performance of the algorithm. Parallel tempering (also known
5

---

Published as a conference paper at ICLR 2025
replica exchange) (Marinari & Parisi, 1992; Geyer & Thompson, 1995; Chandra et al., 2019) is a standard approach for exploring multiple modes of the posterior distribution while performing LMC. In parallel tempering, multiple MCMC chains (known replicas) executed at different temperature values. It allows global and local exploration which makes it suitable for sampling from multi-modal distributions (Hukushima & Nemoto, 1996; Patriksson & van der Spoel, 2008).
We use a simplified version parallel tempering where for all replicas we use the same temperature. To reduce complexity, we also do not perform replica exchange. Even though, in principle, it limit the exploration of the parameter space, as we initialize each replica with different starting points, achieves enough exploration for our purpose while maintaining a simple implementation. By running sample Q-functions for crities from distinct modes
multiple LMC chains V {vb(() ) of the multimodal posterior distribution while ensuring faster convergence and mixing time.
Diffusion Q Action Gradient. Our approach begins with Tg. which approximates a Max-Ent policy (Eysenbach & Levine, 2022) used for collecting online trajectories т. To enhance the diversity of state-action pairs and increase the UTD ratio for critic updates, we first randomly sample a batch BD, from the online buffer. Next, we sample synthetic data BM, from a diffusion (Wang et al., 2023; Lu et al., 2024) generator M.
However, between the periodic updates of M, synthetic data generated by M can become stale, potentially limiting its effectiveness in dynamic environments. Hence, each action sample in Bar, is then refined through gradient ascent 3, V V.Quic Qucc (s, d), targeting improved alignment with high-value regions. While this action gradient approach shares conceptual similarities with DIPO (Yang et al., 2023), which replaces the original actions from the samples in the replay buffer by performing gradient ascent for policy optimization, our dual focus on diversity and quality of mini-batch data used for critic and policy update is distinct. We utilize Q action gradient specifically for critic updates, ensuring that the synthetic actions are not only diverse but also accurately reflect regions of high Q value, all while remaining within the valid support of the action space. Finally, increment the UTD ratio, we mix Bp, and BM, into single parallel data batch B; and for each 1 S i S n, update 20()) using Algorithm 2.
The diffusion model M is trained with the score matching loss
(M( ==
colvasa + vI 022 8, f)ll
(10)
where U([T)) denotes the uniform distribution over a finite collection of reverse time indices {I...., T}. The weights ā. == II ag a; := B, are computed via predefined diffusion temperatures (8.)
Policy Improvement. For each actor update step, we randomly sample a Z weight 46(() from V and retrieve the mixed replay data batch B,. Soft policy improvement maximizes the usual Max-Ent objective
(Ø) Eg aror arwes
(s, [aH(s)(alss)
(11)
Eg B ao
s. E [2(0,0)22 s. (s) + (
Following Haarnoja et al. (2018a;b), the entropy coefficient a is updated with
Ba V log (a(s) H),,
(12)
where H. is the expected entropy. Finally, update the diffusion model M periodically with on-policy data and generate [D'| copies synthetic transitions into D', while target networks are updated using the Polyak averaging approach.
4 EXPERIMENTS
4.1 EXPERIMENTS IN MUJoCo AND DMC
Main Results. We present empirical evaluations of LSAC on the MuloCo benchmark (Todorov et al., 2012; Brockman et al., 2016) and the DeepMind Control Suite (DMC) (Tassa al., 2018),

---

Published as a conference paper at ICLR 2025
showing that LSAC is able to outperform match several strong baselines, including DSAC-T (Duan et al., 2023), the current state-of-the- art model-free off- off-po RL algorithm. Other baselines include DIPO (Yang et al., 2023), SAC (Haarnoja et al., 80). TD3 (Fujimoto et al., 2018), PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015); REDQ (Chen et al., 2021) and QSM (Psenka et al., 2024). Our code is available at https: //gi thub. wwwm/miishaaq/LSS..
We emphasize that, for implementation simplicity and fair comparisons, both policy and critic networks sizes are kept the same for our algorithm and all of the baselines. After an initial warm-up stage of 1e5 steps, we gradually anneal LMC step size TiQ from the initial 1e-3 down to 1e-4, For computing the adaptive drift bias Çg. we use fixed values of a1 0.9, a2 0.999 in Equation and X 10-8 without tuning them. To prevent gradient explosion during training, we clip the sum of the gradient and the adaptive bias term using clip,( (V LQ + açp) by constant c 0.7.
We accelerate training following the SynthER (Lu al., 2024) implementation and update the diffusion generator M using the data DD every 1e4 time steps. During the critic updates, for each 2i(() E V Q. where 1 1 S n, the sampled replay buffer data is mixed with a synthetic batch M, with ratio of 015. This synthetic batch is generated from the diffuser, with its state-action samples immediately optimized through gradient ascent with respect to the Q function, parameterized by the current weight ())())
From Figure and Table 1, we that LSAC outperforms other baselines in 5 out of 6 tasks from MuJoCo. Humanoid-v3 even though DSAC-T outperforms LSAC, the difference is marginal. For space constraint report the DMC result in Appendix C and Figure 8.
20000
150
Average Retur
15000
Avcrago Return
6000
Averuge Return
100-
10000
5000
2000
50
0.0
0.2
0.4
0.6
0.0
0.4
0.8
1.0
0.4
1.0
Steps (million)
Stops (million)
Stops (million)
(a) Halfchectah-v3
(b) Ant-v3
(c) Swimmer-y3
10000
6000
4000
Average Rotura
Average Return
Average Return
4000
2000
5000
2000
2500
0
0.0
0.2
0.4
0.6
0.3
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
Steps (million)
Steps (million)
Steps (emillion)
(d) Walker2d-v3
(c) Hopper-v3
Humanoid-v3
LSAC (ours)
DSAC-T
DIPO
TRPO
REDA
Figure/ I: Training curves for six MuJoCo continuous control tasks over 1e6 time steps. Results are averaged over window size of 11 epochs and across 10 seeds. Solid lines represent the median performance, and the shaded regions correspond to 90% confidence interval.
(ooted
LSAC (ours)
DSAC-T
DIPO
SAC
TD3
PPO
TRPO
REDQ
HallChestah
17948 + 1724
12703 # 1711
9329
1798
10543 + 1422
9034 + 1350
6560 +: 1189
6534 # 1345
10122
1298
155
3271
Swimmer
151
120
102
Walker2d
6143 :: 394
5880
411
4921 + 549
+ 402
4625
399
3182 -: 233
2228 302
4598 318
Hopper
3839 : 537
3327 588
3138 + 731
2919 +: 165
2604 140
2315 :: 152
2006 201
3002
512
Hummnodd
8545 740
9028
792
5012
1018
102
4459 564
7213
Table 1: Maximum Average Return across 10 seeds over 1e6 time steps. corresponding 90% confidence interval for each task are shown in bold.
Maximum value and
7

---

Published as a conference paper at ICLR 2025
Sensitivity Analysis. In Figure 2, present the learning curves of LSAC for different
values of learning rates n1 E {10 {10-2.10 201003,, 00-3,3.
10 100 ,,10+1 inverse temperature 5Q E {10%; 100, 10", 109, 10"), and bias factor E 1,0,1,011 0.013. We observe that our algorithm most sensitive to the step size nQ in the LMC update and the bias factor from Equation 8. On the contrary, LSAC less sensitive to the choice the inverse temperature SQ-
20000
20000
20000
Average Recurss
15000
Average Resurn
55000
Avenge Retur
15000
10000
10000
5000
5000
0-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps (million)
Steps (million)
Steps (million)
(a) Bias factor
(b) Learning rate dдQ
(c) Inverse temperature 3o
Figure 2: Sensitivity analysis different parameters on HalfCheetah--v environment. A comparison of LSAC with different bias factors a, step sizes I/Q, and inverse temperature parameters BQ.
We now present comprehensive ablation analysis of LSAC by systematically removing individual algorithmic contributions while maintaining optimal parameters for the remaining components. We refer the readers Table 4 for a complete list of hyperparameters used for each model.
Impact of distributional critic on performance and overestimation bias. To understand the impact of distributional critic, run ablation studies where we replace our distributional critic with standard critic implementation in SAC (Haarnoja et al., 2018a). Figure 3 shows that the performance of LSAC declines significantly when distributional critic is replaced by standard critic. To find out what might be driving such performance gap. following the same evaluation protocol as Chen et al. (2021), we compare the normalized Q estimation biases Figure We observe that throughout most of training, LSAC with distributional critic has much smaller and often near-constant under- estimation bias compared to LSAC with standard critic. indicates that distributional critic allows more stable learning and increased performance by lowering estimation bias.
20000
150
Avergge Rotum
15000
Average Retum
4000
100-
10000
Mum
2000
5000
50
0.2
0.4
0.8
L.
Steps (million)
Steps (million)
Steps (mmillion)
(a) Halfcheetah-v3
(b) Walker2d-v3
(c) Swimmer-v3
distributional critic
with
Figure 3: Ablation on MuJoCo environments comparing the replacement of the distributional critic component with standard critic. LSAC with distributional critic is more performant than the variant where standard critic is used.
Usefulness of the synthetic experience replay and action gradient ascent. Figure indicates that the performance of LSAC experiences only a marginal decline in HalfCheetah--y v3 when the diffusion Q action gradient is excluded. However, a more pronounced drop is observed in Ant-v3 (Figure 5b) and Swimmer (Figure 5c) when action gradient excluded.
Number of parallel critics. Figure 6a, we observe that when LSAC is equipped with too few or many parallel crities Noll the performance drops. This is due when the parallel critic number
8

---

Published as a conference paper at ICLR 2025
IQM Norm Q Bias
Norm Bius
10M Norm Q Bias
2
1
0.2
0.8
L.0
Steps (iiillion)
Steps (million)
Steps (million)
(a) HalfCheetah--3
(b) Walker2d-v3
(c) Swimmer-v3
LSAC with distributional critic
LSAC with standard critic
Figure 4: Normalized Q bias plots for ablation study of the distributional critic component in LSAC. The Q bias value is estimated using the Monte Carlo return over 1e3 episodes on-policy, starting from states sampled in the replay buffer.
too low, the LMC sampler cannot explore different modes of the posterior distribution. On the other hand, when the critic number is high, it may hamper the actor learning as during each policy update may encounter some critics only very few times due uniform sampling of the critic. This may cause drop in the performance.
Usefulness aSGLD sampler. In Figure 6b, we observe that approximate Thompson sampling through aSGLD sampler boosts the performance compared to when the critics are simply trained with the Adam (Kingma & Ba, 2014) optimizer. When only Adam is used, collection critics can be thought of as ensemble akin bootstrapped DQN (Osband et al., 2016a).
20000
8000
Avarage Return
15000
Return
Average Return
100-
10000
5000
2000
Difiision arudest. web diffisint praciene
Diftusion godden diffusion
0-
0
diftusion gradien
0.2
0.6
1.0
0.0
0.4
Steps (million)
Steps (million)
Steps (million)
(a) HalfChectah-v3
(b) Ant-v3
(c) Swimmer
Figure 5: Ablation study of Q action gradient regularization synthetic acte-action samples on the effect of average return in three MuJoCo environments.
20000
20000
20032
Averago Rommn
5500
Average Reium
Average Retum
15000
10000
10000
20000
0.88
5000
5000
toy
0-
0.0
0.2
0.4
0.8
0.2
0.4
0.6
1.0
Steps (millionn)
Steps (million)
Update Erquueccy (steps)
(a) Number of parallel critics
(b) Different Q samplers
(c) Update frequencies of M
Figure 6: Ablation study on HalfCheetah-v3 environment. Performance of LSAC is affected (a) the choice of parallel critics number and (b) the use of LMC (aSGLD) sampler. (c) The performance difference between each update frequency of diffusion model M is not significant.

---

Published as a conference paper at ICLR 2025
Learning is stable in practice. While off-policy deep RL algorithms often challenging to stabilize, we found that LSAC is fairly stable as shown in Figure 18. This is likely due to the KL objective which parallel distributional crities optimized, where the stochastic soft state-action value Z. remains close to the value target distribution T Zz. Moreover, distributional critic stabilizes learning by mitigating overestimation bias.
4.2 EXPLORATION CAPABILITY LSAC
To further evaluate the exploration ability of LSAC, we test our method on two types of maze environ- ments, custom version of PointMaze MediMeMium Mm-i and AntMaze-v4 from de Lazcano al. (2024), which are implemented based on the D4RL benchmark (Fu et al., 2020). In PointMaze Medium-v3, the agent is tasked with manipulating ball reach some unknown goal position in the maze. The initial state of the ball is at the center of the maze and we define two potential goal states for the ball the top right and the bottom left corner of the maze. Please refer Appendix for further details on the environments. We first train the agent for 500k environment steps, and then use its oracle to complete 200 evaluation episodes. The agent has better exploration ability solves the task by reaching multiple goals or finding out multiple paths leading goal.
To quantify the exploration ability of LSAC and baseline methods, we discretize the maze and track the cell visitation to visualize the exploration density map and track the cell visitation. We set the maximum density threshold to be 100 visits per cell to reduce the dominance high-density areas such the agent's start location, which may otherwise interfere with measuring the true trajectory densities. In Figure and Figure 9, we see that LSAC is capable of discovering multiple paths leading both goals while all other baselines, except for DIPO (Yang et al., 2023), either fail solve the task or only manage to discover single path. While DIPO manages to find multiple paths toward the goal, LSAC offers state coverage that is comparable or greater than that of DIPO, as shown in Figure 10.
T
F
E
(a) LSAC (ours)
(b) DSAC-T
DIPO
(d) SAC
E-E
F
PPO
(g) TRPO
Figure 7: Exploration density maps of LSAC and baseline algorithms tested on the PointMaze Medium-v3 environment. The two goals are located in the upper-right and lower-left corners, as shown by the triangle markers. The starting position is at the center of the maze map.
5 CONCLUSION
In this paper, we introduced LSAC, off-policy algorithm that leverages LMC based approximate Thompson sampling to learn distributional critic. We observe that distributional critic learning coupled with LMC based exploration boost performance while mitigating overestimation issue commonly seen in other model-free methods. Future work includes trying more advanced approximate samplers such underdamped Langevin Monte Carlo (Ishfaq et al., 2024b).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data, International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
(p. 23.)
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint wrXiv:1606.01540 2016. (pp. 1, 6, 20, and 21.)
Rohitash Chandra, Konark Jain, Ratneel V Deo, and Sally Cripps. Longevin-gradiinn parallel tempering for Bayesian neural learning. Neurocomputing, 359:315-326, 2019. (pp. 1, 2, 4, and 6.)
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In The Twelfth International Conference on Learning Representations, 2024. (p. 2.)
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double Q-learning: Learning fast without model. In International Conference on Learning Representations, 2021. (pp. 2, 7, 8, 21, and 25.)
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019. (p. 16.)
Djork-Aréé Clevert. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 1e.iiv:/51I.00 1.07289, 2015. (p. 22.)
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in -ighnniimnsional non-convex optimization. Advances in Neural Information Processing Systems, 27, 2014. (pp. I and 5.)
Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymna- sium robotics, 2024. URL http://gt thub. commFarama-Founddea Fonnaaaatttntttinn/n/nmmmaaa www.eetiiinnnnnnnnnnnnniniaa Roboticcs.
(p. 10.)
Nicolai Dorka, Tim Welschehold, and Wolfram Burgard. Dynamic update-to-data ratio: Minimizing world model overfitting. In The Eleventh International Conference on Learning Representations, 2023. (p. 2.)
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. JEEE Transactions on Neural Networks and Learning Systems, 33(11):6584-699 2021. (p. 21.)
Jingliang Duan, Wenxuan Wang, Liming Xiao, Jiaxin Gao, and Shengbo Eben Li. DSAC-T: Distributional soft actor-critic with three refinements. arXiv preprint arXiv:2310005888, 2023. (pp. 2, 4, 5, 18, 21, and 25.)
Vikranth Dwaracherla and Benjamin Van Roy. Langevin DQN. arXiv preprint 2rX1v:2002.07282. 2020. (pp. 2 and 16.)
Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022. (pp. 2 and 6.)
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learning Representations, 2018. (p. 16.)
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020. (p. 10.)
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018. (pp. 2, 7, 18, and 21.)
11

---

Published as a conference paper at ICLR 2025
Charles Geyer and Elizabeth A Thompson. Annealing Markov Chain Monte Carlo with applications to ancestral inference. Journal of the American Statistical Association, 90(431):909 1995. (pp. 2 and 6.)
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014. (p. 16.)
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017. (p. 3.)
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning. pp. 1861-1870. PMLR, 2018a. (pp. 2, 7, 8, 18, 21, and 25.)
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05900 2018b. (p. 6.)
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference Learning Representations, 2020.
(pp. 18, 21, and 22.)
Philippe Hatsen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as actor-critic method with diffusion policies. arXiv preprini arXiv:2304. 10573, 2023. (p. 3.)
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020. (p. 3.)
Lars Holden. Mixing of MCMC algorithms. Journal of Statistical Computation and Simulation, 89: 1-19, 05 2019. doi: 10. 00t.:00994496552211911......
(p. 1.)
Vincent Huang, Tobias Ley, Martha Vlachou-Konchylle and Wenfeng Hu. Enhanced experience replay generation for efficient reinforcement learning. arXiv preprint arXiv:1705.08245 2017. (p. 16.)
Koji Hukushima and Koji Nemoto. Exchange Monte Carlo method and application to spin glass simulations. Journal of the Physical Society of Japan, 65(6):1604-1608, 1996. (p. 6.)
Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub, Zhuoran Yang. Zhaoran Wang. Doina Precup, and Lin Yang. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning, pp. 4607-4616. PMLR, 2021. (p. 16.)
Haque Ishfaq, Qingfeng Lan, Pan Xu, A Rupam Mahmood, Doina Precup, Anima Anandkumar, and Kamyar Azizzadenesheli, Provable and practical: Efficient exploration in reinforcement learning via Langevin Monte Carlo. The Twelfth International Conference on Learning Representations, 2024a. (pp. 1, 2, 5, and 16.)
Haque Ishfaq, Yixin Tan, Yu Yang, Qingfeng Lan, Jianfeng Lu, A. Rupam Mahmood, Doina Precup, and Pan Xu. More efficient randomized exploration for reinforcement learning via approximate sampling. In Reinforcement Leurning Conference, 2024b. (pp. 1, 2, 10, and 16.)
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffuusion- based generative models. Advances Neural Information Processing Systems, 35:26565-26577, 2022. (p. 3.)
Schwan Kim, Qifan Song, and Faming Liang. Stochastic gradient Langevin dynamics with adaptive drifts. Journal of Statistical Computation and Simulation, 92(2):318-333 2022. (p. 5.)
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1411 1412,6980 2014. (pp. 5, 9, 22, and 23.)
12

---

Published as a conference paper at ICLR 2025
Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends@ in Machine Learning, 1774)3307-392, 2019. (p. 16.)
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic:
Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33:741-752, 2020. (p. 16.)
Hao Li, Zheng Xu, Gavin Taylor. Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in Neural Information Processing Systems, 31, 2018. (p. 5.)
Zechu Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki.
Learning multimodal behaviors from scratch with diffusion policy gradient. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. (p. 16.)
Ziniu Li, Yingru Li, Yushun Zhang, Tong Zhang, and Zhi-Quar Luo. HyperDQN: A randomized exploration method for deep reinforcement learning. In International Conference on Learning Representations, 2021. (p. 16.)
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: SSKG:v:/5090097777 2015, (p. 22.)
Cheng Lu, Yuhao Zhou, Fan Bao. Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022. (p. 3.)
Cong Lu, Philip Ball, Yee Whye Teh, and Jack Marker-Holder Synthetic experience replay. Advances in Neural Information Processing Systems, 36, 2024. (pp. 2, 6, 7, 16, and 23.)
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004.14547, 2020. (pp. and 18.)
Enzo Marinari and Giorgio Parisi. Simulated tempering: a new Monte Carlo scheme. Europhysics letters, 19(6):451, 1992. (p. 6.)
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. Advances in Neural Information Processing Systems, 29, 2016a. (pp. and 16.)
Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In International Conference on Machine Learning, pp. 2377-2386. PMLR, 2016b.
(p. 16.)
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018. (p. 16.)
Alexandra Patriksson and David van der Spoel. A temperature predictor for parallel tempering simulations. Physical Chemistry Chemical Physics, 10(15):0033-2077, 2008. (p. 6.)
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbecl, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations, 2018. (p. 16.)
Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via Q-score matching. In Forty-first International Conference on Machine Learning. 2024. (pp. 7, 16, and 25.)
Gareth O Roberts and Osnat Stramer. Langevin diffusions and castropoli--hasting algorithms. Methodology and Computing in Applied Probability, 4(4):337-357, 2002. (p. 3.)
Gareth O Roberts and Richard L Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, pp. 341-363, 1996. (p. 3.)
13

---

Published as a conference paper at ICLR 2025
Peter Rossky, Jimmie D Doll, and Harold L Friedman. Brownian dynamics smart Monte Carlo simulation. The Journal of Chemical Physics, 69(10):4628 4633))44666--43,, 1978. (p. 3.)
Vivekananda Roy. Convergence diagnosties for Markov Chain Monte Carlo. Annu. Rev. Stat. Appl, 7: 387-412, 2020. (p. 2.)
Daniel Russo. Worst-case regret bounds for exploration via randomized value functions. In Advances in Neural Information Processing Systems, pp. 14410-14420, 2019. (p. 16.)
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. International Conference on Machine Learning, pp. 1899,187,, 2015. (pp. 7, 18, and 21.)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (cXivv1707.06477 2017, (pp. 7, 18, and 21.)
Jascha Sohl- Dickkstiin, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pp. 2256-2265. PMLR, 2015. (p. 3.)
Malcolm Strens. A Bayesian framework for reinforcement learning. In ICML, volume 2000, pp. 943-950, 2000. (p. 16.)
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00660, 2018. (pp. 18, 19, 20, and 21.)
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000. (p. 22.)
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. (p. 6.)
Nisheeth K. Vishnoi. An introduction to Hamiltonian Monte Carlo method for sampling. arXiv preprint arXiv:2108.12100 2021. (p. 1.)
Yinuo Wang, Likun Wang. Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang. Liming Xiao, Jiang WU, Jingliang Duan, and Shengbo Eben Li. Diffusion actor-critic with entropy regulator. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. (p. 16.)
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. (pp. 3 and 6.)
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICML-1I), pp. 681-688. Citeseer, 2011. (pp. 1 and 3.)
Zhihan Xiong, Ruoqi Shen, Qiwen Cui, Maryam Fazel, and Simon Shaolei Du. Near-optimal randomized exploration for tabular Markov decision processes. In Advances in Neural Information Processing Systems, 2022. (p. 16.)
Long Yang. Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13122 2023, (pp. 2, 6, 7, 10, 16, 18, 21, and 25.)
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-frree reinforcement learning from images, 2020. URL hnttps://opett review.
(p. 21.)
14

---

Published as a conference paper at ICLR 2025
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. International Conference on Learning Representations, 2022. (pp. 18, 21, and 22.)
Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics, pp. 1964-1964. PMLR, 2020. (p. 16.)
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010. (p. 3.)
15

---

Published as a conference paper at ICLR 2025
A
RELATED WORK
Posterior sampling. Our research is closely aligned with approaches that utilize posterior sampling, specifically Thompson sampling, within the reinforcement learning (RL) framework (Strens, 2000). Notably, Osband et al. (2016b), Russo (2019), and Xiong et al. (2022) introduced randomized least- squares value iteration (RLSVI), which incorporates frequentist regret analysis in the context of tabular MDPs. RLSVI strategically adds carefully calibrated random noise the value function to promote exploration. Building on this. Zanette et al. (2020) and Ishfag et al. (2021) extended RLSVI linear MDP settings. Although RLSVI achieves favorable regret bounds in both tabular and linear scenarios, its reliance on predefined and fixed features during training limits its applicability to deep RL environments (Li al., 2021).
To address this limitation, Osband et al. (2016a; 2018) proposed training an ensemble of randomly initialized neural networks, treating them as approximate posterior samples of Q functions. However, this ensemble approach incurs significant computational overhead. Alternatively, some studies have explored directly injecting noise into network parameters (Fortunato et al., 2018; Plappert et al., 2018). For instance, Noisy-Net (Fortunato et al., 2018) learns noisy parameters through gradient descent, while Plappert et al. (2018) introduced constant Gaussian noise to the network parameters. Nonetheless, Noisy-Net does not guarantee an accurate approximation of the posterior distribution (Fortunato et al., 2018).
Dwaracherla & Van Roy (2020); Ishfaq et al. (2024a:b) propose using Langevin Monte Carlo for approximate Thompson sampling which is the most related work to ours. Furthermore, Ciosek al. (2019) explored bootstrapped DON-inspired actor-critic algorithms but were unable manage scenarios involving multimodal Q posterior distributions.
Upsampling in RL training. Prior RL studies that augment existing datasets typically employ Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) or Variational Auto-Encoders (VAEs) (Kingma et al., 2019). For example, Huang et al. (2017) utilized GANs to generate synthetic data for pre-training RL agents, thereby accelerating training in production environments. Similarly, Lee et al. (2020) applied sequential latent variable models like VAEs to perform amortized variational inference in partially observable Markov Decision Processes (POMDPs). However, as highlighted by Lu et al. (2024), these methods often face limitations in achieving rapid training in online proprioceptive settings and scalability in data synthesis.
Online reinforcement learning with diffusion. Recently, there has been growing interest in using diffusion model to represent policies in online reinforcement learning due to its inherent ability in learning complex and multimodal distributions. One of the earliest works, that employ diffusion policies for online RL is DIPO (Yang et al., 2023). DIPO uses the critic to update the sampled action from the replay buffer using action gradient before fitting the actor using the updated actions from the replay buffer. Psenka et al. (2024) argues that optimizing the likelihood of the entire chain of denoised actions can be computationally inefficient and instead proposes Q-Score Matching (QSM) that iteratively aligns the gradient of the diffusion actor (i.e. score) with the action gradient of the critic. Li al. (2024) proposes DDifiPG an actor-critic algorithm that learns multimodal policies parameterized as diffusion models from scratch. To discover different modes in the policy, DDifiPG uses novelty-basen intrinsic motivation along with off-the-shelf unsupervised hierarchical clustering methods. More recently, Wang et al. (2024) proposes DACER that uses the reverse process of the diffusion model as a policy function. To perform adaptive adjustment of the exploration level of the diffusion policy, DACER estimates the entropy of the diffusion policy using Gaussian mixture model. We emphasize that while these works utilize diffusion models to parameterize policies, we use diffusion model to create synthetic data to enhance critic learning.
B THEORETICAL INSIGHTS
Without considering the gradient scalar w, the objective function of the critic update from Equation 6 can be written as
Iz()) = B(a DKL
(13)
16

---

Published as a conference paper at ICLR 2025
Using Proposition B. 1, it can be further shown that the objective function in Equation 13 is equivalent to the following:
L2(v)- = -B B.a 2 10:09
[log JogP((ZZZ(ssdd) {.()s,)) +
(14)
where e a term independent of db.
Since En is assumed to be Gaussian model, it can be expressed as (,(-18,a) N(Qu(s, a), (3, where (s, a) and (s, a) are the outputs of the value network. Then ignoring the & independent term c, Equation 14 can be written as
à ((s,a) ((s.))-Qulaa Qu(a,a))
exp(
Lz(() = -E B.. log
200(0,))
(2,0((
Zid (
Z(s a) Qu(s,a)) log( (
E (.
(15)
(0,
(Uz as)
E B.
+ log( (TTo
where we used the definition yz = +
log (a(')))))
Now, let's assume the prior for parameters Gaussian distribution with and variance o2. Then, by Bayes rule, we have
- -oggp(1) [B = log D (b) log logr(B)
(16)
E B(sans))= Qu((s 2 1 +C,
2
where C' is constant and X = 1/o². For simplicity of the analysis, let us assume that the variance of a constant. Then, Equation can written as
(uz Qu(s,a))
L2
E B. 20 20
C2
(17)
C1
where C and C2 are constants.
Combining Equation 16 and Equation 17, we have that L2(b) x - loggpppp B) and thus conse- quently we have:
p B) = z exp(-Lz(V)).
(18)
where Z is the normalizing constant.
Proposition B.I. The objectivefuuction in Equation 13 for learning distributional critic is equivalent the following:
Lz (w) = -B (e.arra)))
log PTT* (
8, s,)))
17

---

Published as a conference paper at ICLR 2025
Proof. The proof is adapted from Ma al. (2020). From Equation 13, the loss function for distribu- tional critic update is given by
Lz (v) (s,aa) (T 2 (s)
= =Bs
Σ
P( a) T. a)) log P(T**2(8, 2 (.(,,a))
('
=sB(s
P(T"'Z(s,a) ( 2 (gg (z,(ss,a) tc
-Bi
By ((aa))=)
log log ((3 2 tc
-
E (o
log ((s.a) (.(11s.))) tc
=-E (na.r))) 2(2),a))23))) (i.)))
log ((s (-(-1s,0) +c
where e a term independent of tb. This completes the proof.
C DMC EXPERIMENT RESULTS
For DMC (Tassa et al., 2018), consider 12 hard exploration tasks with both dense and sparse rewards. We refer the readers to Table 3 for the list of these 12 tasks and their corresponding properties. From Table 2 and Figure 8, see that LSAC outperforms both model-free (DSAC-T (Duan et al., 2023), DIPO (Yang et al., 2023), TD3 (Fujimoto et al., 2018), PPO (Schulman et al., 2017), SAC (Haarnoja et al., 20) TRPO (Schulman et al., 2015), DrQ-v2 (Yarats et al., 2022)) and model-base (Dreamer (Hafner et al., 2020)) in 9 out of 12 tasks.
(Metook
LSAC (ours)
DSAC-T
DIPO
SAC
TD3
PPO
TRPO
Dreamer
DrQ-v2
Cheetah Run
967 + 98
540 + 173
521 + 112
693 + 191
549 + 138
492 - 76
603 +
7924 168
747 + 172
Curtpole Bal, Sp.
1000
1000 :
1000 1:
8
991 12
997 2
996 2
1900 =
1000 1:
Cartpole Swi. Sp.
610 41
573 + 102
112 24
32
164 =
42
751 + 142
783 52
Reacher Easy
941 23
928 + 46
931 59
912++41
908 20
249 +
331 30
481 42
938 3:
Reacher Hard
981
182
152
109
Fish Swim
892 + 122
716 59
816 65
531 +
748 = 101
117 12
283 +
509 38
870 74
Hopper Hop
587 71
144 89
27
10
13++8
7+33
10 2
407
392 113
Finger Torn Easy
64
751 +
548 + 107
525 73
373
965 +
072--70
Finger Tum Hard
949 + 34
743++43 42
271 + 161
251 +-27
252 21
250 + 11
253 34
790 21
832 79
Walker Run
869 +: 67
731 9:93
391
+ 227
465
+
71
388
86
89
21
221
33
726
85
717
Quadruped Run
41
16.13
157
94
1.
1
113
1:
11:55
788
139
753 1.
Table 2: Maximum average return of LSAC and baselines across 10 seeds over 3e6 training steps on selected DMC (Tassa al., 2018) tasks, which consist of complex control tasks that feature complex dynamics, sparse rewards, and hard explorations. Maximum value and corresponding 90% confidence interval for each task shown bold.
D GOAL--REACHING MAZE EXPERIMENTS
In this section, we describe the PointMaze Medium-v3 and the AntMaze-v4 environments in detail.
PointMaze MMdiuu--a Environment. In PointMazeeMedium-va3 the agent is tasked with manipu- lating a ball to reach some unknown goal position in the maze, with each observation as a dictionary, consisting of an array of the ball's kinetic information, an achieved_gee _goal key representing the current state of the green ball, and a desired desired_goal key representing the final goal to be achieved.
The action is force vector applied to the ball. The initial state of the ball is at the center of the maze, and we define the goal positions to be at the upper-right and lower-left corners. The reward function is defined to be the negative Fuclidean distance between the desired goal and the visited state.
18

---

Published as a conference paper at ICLR 2025
1000
1000
1000
Aveage Retum
750
Avezage Return
Average Retum
730
500
500
250
250
250
0
2.5
2.0
3.0
0.5
25
3.0
Steps (million)
Steps (million)
Steps (million)
(a) Cheetah Run
(b) Finger Turn Easy
Finger Turn Hard
1000
1000
1000
Aveage Recurn
750
Average Retur
Averuge Retum
.00
500
500
250
250
250
01
2.0
0.5
2.5
Steps (million)
Steps (million)
Steps (million)
(d) Reacher Hard
(e) Reacher Easy
(l) Walker Walk
1000
1000
800
Avernge Rehirn
600
Avernge Return
750
Return
750
400
200
0
0.0
0.5
1.0
15
2.5
3.0
0.0
0.5
1.0
2.0
2.5
3.0
0.5
1.5
2.5
Steps (million)
Steps (million)
Steps (million)
(g) Walker Run
(h) Fish Swim
(i) Cartpole Balance Sparse
1000
1000
600
Avenge Retum
750
Averaeo Retur
Avenuge Retur
750
500
500
200
0.0
1.0
1.5
2.0
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Steps (million)
Steps (million)
Steps (million)
(j) Cartpole Swingup Sparse
(k) Hopper Hop
(l) Quadruped Run
ISAC
DSAC-T
DIPO
TDE
PPO
SAC
TRPO
Deeuner
DiQ+2
Figure 8: Training curves for 12 DMC (Tassa et al., 2018) continuous control tasks. Results are gathered throughout 3e6 steps of online interaction with the environments, averaged over window size 20 and across seeds. Solid lines represent the median performance, and the shaded regions correspond to 90% confidence interval.
AntMaze-v4 Environment. Similar to 3.3 AntMaze-v4 is also a navigation task, in which the agent is tasked with controlling complex 8 degree-of-frredom (DOF) quadruped robot. The objective is to reach one of the two goal positions where the red balls are located. Each goal can be accessed through two routes. The agent bypass the obstacle the right by either going up or down, and similarly for the two obstacles on the left. The episode length is set to 700. A sparse 0)1 reward is applied upon reaching the goal.
19

---

Published as a conference paper at ICLR 2025
(a) L.SCC (ours)
(b) DSAC-T
(c) DIPO
(d) SAC
66
62
60
(c)
(f) PPO
g TRPO
Figure 9: Exploration density maps of LSAC and baseline algorithms tested the AntMaze-v4 environment. The two goals are depicted by the triangle markers which are located on the left side of the maze. The starting position is the center of the right side of the maze.
LSAC (ours)
DSAC-T
DIPO
TD3
SAC
PPO
TRPO
E
8. 2.0.5
50
0.50
Sizps (millon)
Nage (milkkn)
(a) AntMaze-v4
(b) PointMaze-v3
Figure 10: State coverage of LSAC and baselines in the maze environments. For state coverage, measure the binary coverage of each cell. The y-axis indicates the percentage of cells that have been visited.
E IMPLEMENTATION DETAILS FOR THE EXPERIMENTS
In this section, we provide further details on the environments and the algorithm implementation.
E. ENVIRONMENTS
MuJoCo and DMC. Our experimental evaluations mainly based the MuloCo environments (Brockman et al., 2016) and the DeepMind Control Suite (DMC) (Tassa et al., 2018). For MuJoCo, we pick the six most challenging vector-input-baase Walker2d, Hopper, and Swimmer). On the other hand, DMC offers environments of various difficulty levels, including complex multi-joint bodies and high degree of freedom settings. Hence, we select in particular sparse and hard environments to test the exploration ability of LSAC and basclines. No modifications were made to the state, action, or reward spaces. The action space AA in both continuous control benchmarks is by default the box [-ar, ag.)* 1d. where d = dim A is the dimension of the action space, and aj, ad are the low and high action limit, respectively. DMC environment observations, different from MuJoCo's vectorized states input, are stacks of three consecutive RGB images, each of size 84 x 84, stacked along the channel dimension to enable inference of dynamic information like velocity and acceleration. Thus, DMC places more emphasis than MuJoCo pixel-based proprioceptive tasks, which better tests an agent's learning in visual continuous control together with more challenging exploration tasks.
control tasks (HalfCheetah, Ant, Humanoid,
20

---

Published as a conference paper at ICLR 2025
Figure 11:
OpenAl MuJoCo (Brockman et al., 2016) benchmarks. From left to right: Humanoid, Ant, HalfCheetah, Walker2d, Hopper, and Swimmer.
DeepMind Control Suite (Tassa et al., 2018) domains. From left to right: Finger, Fish, Cheetah, Reacher, Cartpole, and Walker.
Figure 12:
For DMC, we consider 12 hard exploration tasks with both dense and sparse rewards, in which many other off-policy model-free RL algorithms often struggle. We refer the reader to Table 3 for detailed description of the DMC environments we use in our experiments. Figure 12 shows some examples of the DMC environments.
Task
Traits
dim(S)
dim(.A)
Cheetah Run
run, dense
18
Finger Turn Easy
turn, sparse
6
Finger Turn Hard
turm, sparse
6
Reacher Hard
reach, dense
4
Reacher Easy
reach, dense
4
Walker Walk
walk, dense
18
Walker Run
run, dense
18
Fish Swim
swim, dense
26
Cartpole Balance Sparse
balance, sparse
4
Cartpole Swingup Sparse
swing. sparse
4
Hopper Hop
move, dense run, dense
14
Quadruped Run
56
12
Table 3: A detailed description of each task used in the DeepMind Control Suite (DMC) (Tassa et al., 2018) experiment.
E.2 BASELINE METHODS
For the evaluation of the baseline algorithms on MuJoCo, we consider DSAC-T (Duan et al., 2023), DIPO (Yang et al., 2023), SAC (Haarnoja et al., 2018a). TD3 (Fujimoto al., 2018), PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015) and REDQ (Chen et al., 2021). Additionally, for the DMC benchmark we add two additional strong baselines for visual learning, DrQ-v2 (Yarats al., 2022) and Dreamer (Hafner et al., 2020). In particular, Dreamer is a leading model-based algorithm for visual continuous control setting, which tends to achieve better sample complexity in the expense of greater computational burden from learning separate state dynamics model. The highest reported performance on the continuous tasks is achieved by DSAC-T (Duan et al., 2023), an improved version of DSAC (Duan et al.. 2021) that uses expected value substituting, twin value distribution learning, and variance- based critic gradient adjusting. Note that since vanilla SAC is rather weak baseline, consider augmenting it with the convolutional encoder from SAC-AE (Yarats et al., 2020), similar
21

---

Published as a conference paper at ICLR 2025
to what has been done in (Yarats al., 2022). We use publicly available codebases for reproducing the baseline results."
The hyperparameters shared across all methods for the MuJoCo tasks are listed in Table 4. We adopted the best hyperparameters reported by the authors of the respective baseline methods, as all of them utilize MuloCo for continuous control evaluation and conducted extensive hyperparameter sweeps. For implementation simplicity, we maintain these key hyperparameters fixed when testing on the DMC benchmark.
Hyperparameter
(Sur()ur))
DSAC-T
DIPO
SAC
TD3
PPO
Num. hidden layers
3
3
3
3
Num. hidden nodes
256
256
256
256
256
256
Activation
GeLU
GeLU
Mish
ReLU
ReLU
Tanh
Batch size B
256
256
256
256
256
256
Replay buffer size [D]
1e6
1e6
1e6
1e6
le6
1e6
Diffusion buffer size D
1e6
N/A
1e6
N/A
N/A
N/A
Discount for reward Y
0.99
0.99
0.99
0.99
0.99
0.99
Target smoothing factor
0.005
0,005
0.005
0.005
0.005
0.005
Optimizer
aSGLD
Adam
Adam
Adam
Adam
Adam
Adaptive bias (a1,
(0.9, 0.999)
(0.9, 0.999)
(0.9. 0.99)
(0.9, 0.99)
(0.9, 0.99)
(0.9. 0.99)
Inverse lemperature BQ
1e-8
N/A
N/A
N/A
N/A
N/A
Actor learning rate 3,
3e-4
3e-4
3e-4
3e-4
3e-4
7e-4
Number of critics n
1
N/A
Critic learning rate no
1e- 3 + 1e-4
3e-4
3e-4
3e-4
3e-4
7e-4
Actor Critic grad norm
0.7
N/A
2
N/A
N/A
0.5
Replay memory size Entropy coefficient a
1e6
1e6
1e6
1e6
1e6
1e6
N/A
0.2
N/A
0.01
Expected entropy H
dim A
dim A
N/A
N/A
N/A
N/A
Diffusion training frequency
1e4
N/A
1
N/A
N/A
N/A
Table 4: Common hyperparameters used across all MuJoCo and 12 DMC tasks for LSAC and baselines.
E.2.1 ADDITIONAL BASELINES FOR THE DMC EXPERIMENT
The two additional baselines we use for the DMC experiments are Dreamer (Hafner et al., 2020) and DrQ-v2 (Yarats al., 2022). They use few different network configurations and additional model components. Dreamer (Hafner et al., 2020) uses a pair of convolutional encoder and decoder networks, with remaining functions implemented as three dense layers size 300 with ELU activations (Clevert, 2015). Action outputs are passed through a tanh mean layer, scaled by factor of 5 and applied with softplus transformation. The world model, value model, and action models are all trained on batches of 50 sequences of length 50, using the Adam (Kingma & Ba, 2014) optimizer with learning rates 6e-4, 8,-5, and 8e-5, respectively. Gradient norms over 100 are scaled down. We note that Dreamer uses an imagination horizon of H = 15, which is exclusive to itself, while the value targets VA are updated with discount factor 7 = 0.99 and X = 0.95. The first five episodes used as warm-up period, where the random actions noise. The latent dynamics model is trained on an information bottleneck objective (Tishby et al., 2000): max ($1:7; (01:r, F1:r) (1:))
sampled with (N00, 0.3) exploration
BI(( f1:27 (1:r), where 3 is scalar temperature and 4 are dataset indices such that p(o, 1t) = 8(); oz). On the other hand, DrQ-v2 (Yarats et al., 2022) uses DDPG (Lillicrap
al., 2015) as backbone and augment it with n-step returns for estimating the TD error. Image encoders J are used to embed augmented image observations into cow-dimensional latent vector by CNN. Exploration noise is scheduled according (t) Fini + (1 - min(t/ 1)) Fmit) at different states of training. A bigger batch size of 512
DIPO: httts::/gitth:i com/Be lman TimeHut/DIPO: SAC: http://///.tttuuuumommm/aarr/aaa TD3: https://github..cm com/sffjim//m.. PPO: http://hhhh...ooo/./l/./bbaeeee/
TRPO: https: h/tp::::///hhtbb com/cko///////// toost
rikov/pytorch- trpo: REDQ: httt:tttsss/////ttttiit http:///giiiuuccccooooot/aaaaaaa QSM: https://githu/:
om/ mm/Allescontrellsssreeetttttniie Dreamer: httpps://gt thub. wtt/:///////aa///e/eaaaaaeaaa
http://///att/././oo/////////rrr DrQ-V2:
22

---

Published as a conference paper at ICLR 2025
and smaller learning rate of 1e-4 is used. Evaluation frequency is set to be every 10 episodes, same for each method.
E.3 MORE ON DIFFUSION Q ACTION GRADIENT
The diffusion generator M in Algorithm plays crucial role in synthesizing a batch Ba,, of full environment transitions for each critic Qu These synthetic transitions consist of concatenated states sM. actions ax,, episodic rewards rMr, next observations 6'rr and binary terminal masks dv- We set the mixing ratio of BD, U BM, to be 0.5 following the design choice of Ball et al. (2023), where [Bp, = Ba, = 128. Thus, the overall batch size stays the same as baseline methods at |B| = 256. We train M normalized transition tuples from the collected online trajectories within the replay buffer D, ensuring that each entry has a mean of zero and standard deviation of one, except for the "done" signals which remain unnormalized and thresholded to either 0 based cutoff of 0.5.
To speed up online RL training, we update M using data from D every 1e4 online interaction steps and generate 1e6 transitions right after each update of M, which are then stored in the diffusion buffer. The diffusion buffer D' differs from the online replay buffer D in that the log probability of Max-Ent actions are not collected and that its action samples are normalized, such that E [-1, 1]dd suit for more effective Q action gradient regularization. Our implementation of M uses the default training hyperparameters in Lu et al. (2024).
Although Lu al. (2024) has observed the fidelity these synthetic samples by comparing their high-level statistics to those of the on-policy data, potential limitation arises from the generated data becoming stale during the 1e4 online interaction steps, due to the fact that M remains unchanged and the uncertainty in critic function updates. Hence, we apply Q action gradient
normalized synthetic action samples by gradient ascent along the Q gradient field. We make use of the Adam optimizer (Kingma & Ba, 2014) with Polyak coefficients (a1, a2) (0.9, 0.99) and learning rate of 3e-4. After Q action gradient update on a& ~ D', we replace these state-action pairs in the diffusion buffer update belief about the current high-value regions in the action space and mitigate the risk of data staleness.
In Figure 14, we show distribution heatmaps of sampled actions in the online replay buffer D, synthesized action samples a in the diffusion buffer D', as well as Q gradient-optimized actions am V.Q. We observe that one update of M every 1e4 steps is adequate to match the high-valued region in the sample distributions. We also plot the best average return corresponding to each choice of updating frequency in three MuJoCo environments Figure 13, which supports our choice of update frequency of M.
9000
Time
1.00
Normal ioed Training Time
Time
5000
Average Return
6000
Normal ized Fraining
Average Relum
10.5
Averuge Retum
5000
0.95
10.00
3000
4000
4000
10.25
2000
10
3030
+0.85
0.75
3000
1000
0.30
0.70
1,000
10.000
100,000
1,000
16,100
100
10,000
100,000
Update froquency (steps)
Update frequency (stops)
Update frequency (steps) (c) Walker2d-v3
(a) Ant-v3
(b) Hopper-v3
Figure 13: Sensitivity analysis on the performance and training time of LSAA for different choices of update frequencies of diffusion generator M, in three MuJoCo environments. Results are averaged over 10 seeds. The performance difference between each update frequency of M is not significant on Ant-v3 and Walker2d-v3. However, LSAC scores less with the least update frequency (one in 1e5 steps) in H3pper-v3. The purple dots represent normalized training time.
23

---

Published as a conference paper at ICLR 2025
0
GM V.Q, 1e5 steps
aM
VaQ, 1e4 steps
aM VaQ, 1e3 steps
0M ~ V Q, 1e2 steps
D, 1e5 steps
D, 1e4 steps
a D, 1e3 steps
D, 1e2 steps
-1
0M ~ D', 1e5 1e5 steps
0.6 ~ D', 1e4 steps
0.3 D', 1e3 steps
0. ~ D', 1e2 steps
Figure 14: Comparison of Q gradient optimized sample distribution ((a-~VV v..2 (first row), replay action sample distribution d(awD) (second row), and diffusion buffer sample distribution d(ax (third row). The number of steps in the caption means the update frequency of M. Darker regions denote higher density. We randomly sample batch of 1e3 transitions from all three distributions at 2e5 time steps the Ant-v3 environment. The 2D kernel density estimate of these distributions taken from action space dimensions 2 and 3. We see that updating M every 1e4 steps is suitable for replicating the on-policy buffer action distributions with optimal total training time. More frequent updates M during training (frequency I 1e3 steps) leads to higher training time cost as shown in Figure 13. Q gradient optimization further refines the synthesized actions as D' better align with d(awD):
24

---

Published as a conference paper at ICLR 2025
F COMPUTATIONAL EFFICIENCY OF LSAC COMPARED TO BASELINES
Ensuring computational efficiency critical if we want to make application of RL. in the real world problems practical, As LSAC introduces parallel distributed critic learning with diffusion synthesized state-action samples for diverse critic learning, it naturally raises a question of whether the processing speed (wall-clock runtime) and the memory efficiency (number of learnable parameters in each model) become bottleneck for training LSAC. To answer this question, we compare LSAC to other baselines representative of two other RL training paradigms namely single-critie learning with policy entropy and diffusion policy inference. Among our baselines, we use DSAC-T (Duan et al., 2023), SAC (Haarnoja et al., 2018a), DIPO (Yang et al., 2023), QSM (Psenka et al., 2024), and REDQ (Chen et al., 2021). Our experiment indicates that, besides demonstrating superior sample efficiency and outperforming the baselines in most environments in Figure and Figure 8, LSAC also achieves comparable better computational efficiency than that of the baselines, as shown in Figure 15. To facilitate fair wall-clock time comparison, all algorithms are trained on the same hardware (i.e a single NVIDIA Quadro RTX 8000 GPU machine).
From Table and Figure 15, we see that the processing speed of LSACC is somewhat slower than that of DSAC-T, primarily due to the time spent parallel critic learning and diffusion upsampling, which is amortized over every 1e4 steps. However, LSAC training is significantly faster than ensemble based method- REDQ. As for diffusion-based policy methods like DIPO and QSM, while they offer the benefit of multimodal action distributions, they do so at the cost of expensive diffusion policy inference steps during online trajectory rollouts, which makes them considerably slower during training. A clear comparison of total time taken by these algorithms can be found in Table 5 and Figure 15.
Environment
Algoritim
LSAC (ours)
DSAC-T
DIPO
SAC
QSM
REDQ
Ant-v3
1128 (1095)
876 (857)
2119(911)
751 (722)
1898 (735)
1623 (1589)
HalfChectah-v3
1157 (1112)
858 (799)
2253 (946)
781 (737)
1912 (779)
1740 (1694)
Walker2d-v3
1141 (1101)
901 (879)
2191 (966)
761 (721)
1850 (753)
1828 (1625)
Humanoid-v3
1209 (1162)
904 (857)
(911)
768 (719)
1937 (732)
1794 (1633)
Hopper-v3
1124 (1078)
884 (792)
2391 (898)
772 (721)
1893 (778)
1879 (1417)
Swimmer-v3
1197 (1049)
(891)
2201 (913)
764 (703)
1829 (716)
1803 (1544)
Table Process times in ms per each actor-critic update loop in the MuloCo environments. Process times per Q functions update are shown in the parenthesis.
Envinonment Algozithm
I.ACC (ours)
DSAC-T
DIPO
SAC
QSM
REDQ
Ant-v3
2.992M
301K
5.109M
168K
4.829M
1.066M
HallCheetah-v3
2.591M
269K
4.307M
177K
3.516M
961K
Walker2d-v3
2.447M
255K
4.092M
146K
3.978M
795K
Humanoid-v3
4.807M
472K
7.041M
166K
6.125M
1.840M
Hopper-v3
2.944M
243K
4.121M
146K
3.772M
796K
Swimmer-v3
2.874M
239K
4.908M
135K
4.063M
771K
Table 6: Number of learnable parameters in each method. LSAC has less parameters compared to diffusion baaicy-based model-free algorithms (DIPO, QSM) but more than those of double or ensemble-based actor-critic methods (DSAC-T, SAC, REDQ). LSAC uses the same network structure and ensemble size across tested environments. The difference in the number of parameters is due to different observation and action dimensions in each environment, which affect the layer sizes of model networks.
25

---

Published as a conference paper at ICLR 2025
15000
6000
6000
Average Rtturn
Average Retum
Average Retum
10000
4000
2000
2000
0
0-
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
Steps (million)
Steps (million)
Steps (million)
(a) AR at 10h, Ant
(b) AR at 10h, Chectah
(c) AR at 10h, Walker
ESSC:
20000
LSAC
9000
LSAC:
Wallelok Runs ine 68
DRAG-T
DSACTT
DRAC-T
Wallelock Rustinu
Walleloc Rustine
00000
REDO OSM
10000
REPO QSN
REDO USM
40000
+200
2000
20000
Algorithms
Algoritams
Algorithms
(d) Runtime 2e5 steps, Ant
(e) Runtime at 2e5 steps, Cheetah
(f) Runtime at 2e5 steps, Walker
4000
150
Average Retum
3000
Average Retum
6000
4000
2000
Average
50
1000
2000
0-
0.1
0.2
0.3
0.5
0.0
0.1
0.3
0.4
0.5
0.0
0.1
0.2
0.4
Stops (million)
Steps (million)
Stops (million)
(g) AR 10h, Hopper
(h) AR 10h, Swimmer
(i) AR at 10h, Humanoid
$2000
18AC
1000
LSArT
DSAC-T
USAC-T
DRAC-I
Runtime
60000
SAC
Wallclnck Kintime
03000
SAC T RETIQ QUM
Runtime
RIDOO OSM
40000
RUDQ ORM
Wallelock
42000
41000-
33000
Q5M
Algorithms
Algorilhms
Algorithms
Runtime at 2e5 steps. Hopper
(k) Runtime at 2e5 steps, Swimmer
Runtime 2e5 steps, Humanoid
LSAC (ours)
DSAC-T
DIPO
SAC
REDQ
QSM
Figure 15: The first and the third show the number of steps completed in 10 hour of training and the corresponding average return (AR) in corresponding environments. The second and the fourth row show the average wall-clock runtime in seconds to complete 2e5 steps in respective environments.
26

---

Published as a conference paper at ICLR 2025
4000
Averge Remmm
6000
Average Retum
Averzye Retum
7500
4000
2000
2500
2000
0
0.6 Steps (millionn)
0.4
0.8
1.0
0.2
0.6 Steps (emillion))
Stops (million)
(a) Ant-v3
Hopper-v3
(c) Humanoid-v3
LSAC with distributional critic
LSAC with standard critic
Figure 16: Ablation on MuJoCo environments comparing the replacement of the distributional critic component with standard critic. LSAC is more performant with improved stability than the ablated counterpart.
IQM Norm 1ias
IQM Norm Q Bias
JQM Norn Bius
0.0
0.2
0.6 Steps (million)
0.4
0.8
1.0
0.0
0.2
0.6 Steps (millioa)
0.4
0.8
1.0
0.0
022
0.4
0.6
0.8
1.0
Steps (million)
(a) Ant-v3
(b) Hopper-v3
(c) Humanoid-v3
LSAC with distributional critic
LSAC with standard critic
Figure 17: Normalized Q bias plots for ablation study of the distributional critic component in LSAC. The Q bias value is estimated using the Monte-Carlo return over 1e3 episodes on-policy, starting from states sampled in the replay buffer.
Average Return
15000
Average Retum
6000
Avenge Return
200
10000
4000
5000
2000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps (million)
Stops (million)
Stops (million)
(a) Halfcheetah-v3
(b) Ant-v3
(c) Swimmer-v3
6000
4000
Arerage Retum
Average Return
Return
4000
AM
5000
2000
2000
2500
0.0
0.4
0.6
0.8
1.0
0.0
0.4
0.8
1.0
0.2
0.6
0.8
Steps (million)
Stops (million)
Steps (emillion)
(d) Walker2d-v3
(e) Hopper-y3
(f) Humanoid-v3
LSAC (ours)
DSAC-T
DIPO
Figure 18: Training curves for the six most challenging MuloCo continuous control tasks over 1 time steps and over 10 individual runs. LSAC almost similarly stable as DSAC-T, and more stable compared to DIPO in most cases.
27


Reviews:
[{"review_id": "lTfG8Kt5gO", "reviewer": "Reviewer_TJsH", "summary": "This paper aims to address inefficient exploration and bad sample usage in actor-critic algorithms for continuous control problems. They propose Langevin Soft Actor-Critic (LSAC) which uses adaptive Langevin Monte Carlo (aSGLD) to sample from a Q-value distribution, improving exploration by capturing uncertainty. A diffusion model is also applied to generate diverse synthetic state-action pairs, refined by Q-action gradients to focus on high-reward areas, thereby enhancing sample efficiency. The algorithm demonstrates improved or comparable empirical performance on MuJoCo and DeepMind Control Suite benchmarks compared to standard RL algorithms. An ablation study is also provided.", "questions": "see weakness", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. They propose a new method that models each Q(s, a) value as a distribution. Then, an LMC-based sampling algorithm is used to sample the posterior of the Q value, which encourages exploration. The accordingly distributional Bellman operator is used.\n2. They propose using a diffusion model to generate more state-action pairs to update the critic function, where each pair's action is replaced with its Q action gradient updated ones. The use of generative models helps increase the sample efficiency.\n3. Thorough experiments are conducted, with the algorithm implemented on two benchmarks, demonstrating improved or comparable empirical performance.", "weaknesses": "My primary concerns are with novelty and clarity.\n\n**Contribution:**\nBased on my reading of this submission, this work extends Langevin Monte Carlo (LMC) [2] from discrete to continuous action spaces (e.g., MuJoCo tasks). It builds on [2] with the same critic modeling as [1] and integrates a diffusion model for generating synthetic replay data [4].\n\nThe main novelties introduced in this work seem to include: \n1. extending the use of multiple Q-values from Langevin-gradient-based Bayesian neural learning [3] to RL.\n2. using gradient ascent-refined actions in diffusion-based policy optimization [5] to improve data diversity in experience replay [4].\n3. reducing the number of aSGLD updates for the critic in [2] from $O(K)$ to a single iteration. However, I am uncertain whether this improvement arises from the diffusion model generating more data, which increases the size of batch in Algorithm 2.\n\nDespite the outlined contributions, I have several **questions** regarding the arguments and claims made in the submission:\n1. the paper claims multiple critics act as parallel tempering to address slow mixing in LMC. However, LMC is described as an optimization rule, not a distribution approximation. Does the issue exist in the LMC-inspired critic update? And does the gain in Figure 6 come from parallel tempering or simply reduced TD target variance, as in SAC, since all critics share the same temperature?\n2. from the ablation study, the distributional critic modeling seems most impactful. Is this due to exploration from the variance in the Gaussian Q-values distribution? My previous question regarding the distributional TD loss is that: for the distributional TD loss $L(\\psi)$ and normal deterministic TD loss $TD_c(\\psi)$, $L(\\psi) = \\frac{TD_c(\\psi)}{\\sigma_{\\psi}} + \\log(\\sigma_{\\psi})$ as shown in Eq(15), higher $\\sigma_{\\psi}$ increases exploration (second term) but reduces TD loss (first term). Therefore, I am curious whether this $\\sigma_{\\psi}$-$TD_c(\\psi)$ trade-off inside the training of $L(\\psi) $ is related to the improvement in Figure 3,4. \n\n\n**Presentation:**\n many techniques or claims are applied without sufficient interpretation or explanation. For example, in section 3, the paragraph **Distributional Critic Learning with Adaptive Langevin Monte Carlo** mostly explains [1] and [2], and Equation (9) is the same as lines 9-11 of Algorithm 2 in [2]. It is better to move them to the preliminary section. As a reader, it is hard to understand the paper without reading other references. When presented in the Algorithm Design section, I would expect the work to provide a self-contained explanation of its logic and methodology.\n\nI truly appreciate that the author included comparisons of running time and parameter numbers. But when a paper adapts old algorithms to a new task, I would expect this work can provide deeper analysis and insights specific to the task at hand. This would enhance the impact.\n\n\n[1] Jingliang Duan, Wenxuan Wang, Liming Xiao, Jiaxin Gao, and Shengbo Eben Li. DSAC-T: Distributional soft actor-critic with three refinements.\n\n[2] Haque Ishfaq, Qingfeng Lan, Pan Xu, A Rupam Mahmood, Doina Precup, Anima Anandkumar, and Kamyar Azizzadenesheli. Provable and practical: Efficient exploration in reinforcement learning via Langevin Monte Carlo. In The Twelfth International Conference on Learning Representations\n\n[3] Rohitash Chandra, Konark Jain, Ratneel V Deo, and Sally Cripps. Langevin-gradient parallel tempering for bayesian neural learning\n\n[4] Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. Advances in Neural Information Processing Systems, 36, 2024.\n \n[5] Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bcQb5fkZCz", "reviewer": "Reviewer_ETnf", "summary": "Authors of this paper introduces a model-free online RL algorithm named Langevin Soft Actor-Critic (LSAC). The algorithm incorporates  Langevin Monte Carlo (LMC) updates for critic learning with parallel tempering and action refinement on diffusion-synthesized trajectories. it also utilizes distributional Q objective, allowing diverse sampling from multimodal Q posteriors. Through experiments on MuJoCo benchmarks, the algorithm is shown to outperform/match the performance of SOTA baselines.", "questions": "As all the techniques introduced and incorporated into LSAC and complexity of the implementation, any analysis why LSAC is not working better than DSAC in the humanoid environment (which is also regarded as the one of the most complex environments among the benchmarks) ? \nWhat is the impact of each key component (LMC updates, parallel tempering, diffusion models) on the performance of LSAC? Have you tried to remove one of these components and see the affect the overall effectiveness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel ideas of combining LMC updates, parallel tempering, and diffusion-synthesized action refinement in the LSAC algorithm.\nBy efficiently exploring state-action spaces through LMC based Q updates and multimodal Q posteriors with parallel tempering, LSAC has the potential to learn a better policy in complex environments.\nLSAC demonstrates competitive or superior performance against several established baselines on standard continuous control benchmarks.\nThe methodological choices in LSAC are grounded in theoretical analysis presented in the paper.", "weaknesses": "The implementation seems very complex even by reading through the pseudo code. \nThe computational cost of LSAC lacks discussion in the paper.\nAdditional parameters and hyper-parameters are introduced, including 10 parallel critics, additional diffusion buffer, inverse temperature, etc, and ablation study shows that LSAC is sensitive to some of the parameters, which may limit its scalability to different applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XAzUvwv3VR", "reviewer": "Reviewer_iNev", "summary": "This paper introduces a new model-free RL algorithm Langevin Soft Actor Critic (LSAC). It uses Langevin Monte Carlo (LMC) for updating the Q function to help exploration and improve performance, the authors pointed out several key challenges in taking this approach and propose solutions that show benefit to performance in the ablations. When compared to a number of other baseline RL algorithms on MuJoCo and DMC benchmarks, LSAC shows stronger overall performance. This is the the first successful application of a Langevin Monte Carlo (LMC) based Thompson sampling in RL continuous control setting.", "questions": "- It seems to me a main part of the strong performance comes from a better exploration technique. In the baselines you compared to, are some of them focused on exploration techniques? If we replace the exploration in proposed method with some other recent exploration technique, will we obtain a similar strong performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**originality**\n\n- authors study and show how to do Langevin Monte Carlo (LMC) based Thompson sampling in RL continuous control setting properly and achieve strong performance, this is a novel contribution. \n\n**quality**\n\n- overall good quality of presentation and writing. \n- fairness of comparison: I appreciate that the authors discuss comparison fairness and provide full details of the hyperparameter setting in the appendix and give source code. \n\n**clarity**\n- overall clear \n\n**significance**\n- strong results: Figure 1, table 1 show that proposed method has fairly consistent stronger results compared to other baselines. \n- results show how to properly use Langevin Monte Carlo (LMC) based Thompson sampling in DMC and MuJoCo. \n- ablations: it is good to have ablations to understand the effect of different design choices.", "weaknesses": "Some of the arguments made in the paper seems not adequately supported, some more explanations or experiments will help:\n\n- Line 346: the authors argue LSAC has good performance while being simple in implementation. Why is it simple in implementation? It seems to me it has a number of additional components/hyperparameters compared to a baseline such as SAC, and these extra components seem not trivial to implement correctly. The complexity of the method seems a weakness to me. \n- Line 460: The authors argue that Figure 5 shows LSAC as more stable. I don't feel convinced. The fluctuation in its performance seems similar to DSAC-T, and even when compared to DIPO, LSAC has some wild changes in Figure 5 (c). \n- Is there a comparison on computation efficiency? With the additionl components in the proposed method, how much computation/wall clock time does it take to finish a run compared to other algorithms? \n- Although a lot of details and discussion are given to show an effort to compare to alternative algorithms in a fair manner, it is a bit unclear to me how 2 things are made fair: (1) network capacity, the proposed approach seems to have more critics and additional networks (such as in the diffusion part), how does it compare to other baselines in terms of e.g. number of parameters? And (2) UTD ratio, if the proposed approach is doing more updates, is it still fair to compare it to baselines that take limited updates? I understand that some of them might not work well with more updates due to instability, but in that case, one might argue it would be good to compare to algorithms that do benefit from ensemble of networks and higher UTD such as REDQ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OAvDTJYwRC", "reviewer": "Reviewer_j9KN", "summary": "This paper presents a novel model-free RL algorithm, Langevin Soft Actor Critic (LSAC), which enhances critic learning through uncertainty estimation rather than just optimizing the policy. LSAC features three innovations: approximate Thompson sampling via distributional Langevin Q updates, parallel tempering for exploring multiple posterior modes, and diffusion-synthesized state-action samples regularized with Q action gradients. Experiments show that LSAC outperforms or matches mainstream model-free RL algorithms in continuous control tasks and is the first successful application of Langevin Monte Carlo-based Thompson sampling in continuous action spaces.", "questions": "- In line 9 of Algorithm1, you directly replaced $a_M$ with $a_{M'}$, which destroyed the Markov chain. What problems exist in fusing such data with $B_D$? Please analyze it theoretically.\n\n- Algorithm1 Why doesn't the strategy also choose to update on $B_i$?\n\n- Line341 Is the frequency of updating the diffusion model too low? Does the data generated in this way really have many benefits for sample distribution? Please analyze the sample distribution, as well as the performance and sample distribution results corresponding to different update frequencies.\n\n- Why do other algorithms in Table1 have no variance? It seems that you didn't say how you calculated the data in the table.\n\n- Do you also sample 20 samples like DSAC-T for each interaction with the environment?\n\n- In the high-dimensional action space task Humanoid-v3, why is your performance much lower than DSAC-T?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper uses LMC combined with distributed Q to learn critic, and also uses the action gradient method to improve the diversity of samples in the buffer. It is an interesting work;\n- The experiments in this paper are very comprehensive, and it has achieved very good performance in many environments, and the curve rises very quickly;\n- Compared with diffusion as a policy, this paper does not have the risk of too long sampling time in actual use.", "weaknesses": "- In the related work, I strongly suggest you add the work on combining diffusion with online reinforcement learning. Although they focus on the policy, they are also directly related to your paper. Eg. Policy Representation via Diffusion Probability Model for Reinforcement Learning (Yang et al.), Learning a Diffusion Model Policy from Rewards via Q-Score Matching (Psenka et al.), Diffusion Actor-Critic with Entropy Regulator (Wang et al.), etc. Please write a paragraph in the paper to summarize.\n\n- I don't agree with the paragraph around line 68. Methods like SAC and DSAC-T are soft policies. They balance exploration and utilization by introducing policy entropy. How can this be classified as a trick? I think their core is also heuristic, which is essentially the same category as your method.\n\n- In Algorithm 1, there are 4 for loops for each policy update, and the time-consuming action gradient is used, but the time efficiency is not analyzed, and the performance difference brought by different action gradient steps is not analyzed. Please add time efficiency analysis: for example, when training for 8 hours at a time, compare the performance of the algorithms (draw a curve with time as the horizontal axis and TAR as the vertical axis).\n\n- You don’t seem to have experimentally proved some of the problems caused by high-dimensional action space. Compare your current method with the previous method to see why this problem disappears (analyze the experimental results).\n\n- Based on the citations of DSAC-T, I found a work Diffusion Actor-Critic with Entropy Regulator (Wang et al.). Please compare their performance numerically.\n\n- I think the core work in Q-update is to combine Provable and practical: Efficient exploration in reinforcement learning via Langevin Monte Carlo with DSAC-T: Distributional Soft Actor-Critic with Three Refinements. This way of updating Q can indeed alleviate overestimation while bringing better performance, but the novelty may be less.\n\nI'll consider score changes based on your response.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)