Double Gumbel Q-Learning
Authors: David Yu-Tung Hui
Aaron Courville
Pierre-Luc Bacon
Venue: NeurIPS 2023 spotlight (2023)
Keywords: deep reinforcement learning
Q-Learning
TD-Learning with function approximation
extreme value theory
maximum-likelihood estimation
moment-matching
TL;DR: DoubleGum is a well-performing Q-Learning algorithm that models noise with two heteroscedastic Gumbel distributions.

Abstract: We show that Deep Neural Networks introduce two heteroscedastic Gumbel noise sources into Q-Learning.  To account for these noise sources, we propose Double Gumbel Q-Learning, a Deep Q-Learning algorithm applicable for both discrete and continuous control.  In discrete control, we derive a closed-form expression for the loss function of our algorithm.  In continuous control, this loss function is intractable and we therefore derive an approximation with a hyperparameter whose value regulates pessimism in Q-Learning.  We present a default value for our pessimism hyperparameter that enables DoubleGum to outperform DDPG, TD3, SAC, XQL, quantile regression, and Mixture-of-Gaussian Critics in aggregate over 33 tasks from DeepMind Control, MuJoCo, MetaWorld, and Box2D and show that tuning this hyperparameter may further improve sample efficiency.

Double Gumbel Q-Learning
David Yu-Tung Hui Mila, Université de Montréal dythu12+dr180ma1l com
Aaron Courville Mila, Université de Montréal
Pierre-Luc Bacon Mila, Université de Montréal
Abstract
We show that Deep Neural Networks introduce two heteroscedastic Gumbel noise sources into Q-Learning- To account for these noise sources, we propose Double Gumbel Q-Learning, Deep Q-Learning algorithm applicable for both discrete and continuous control. discrete control, we derive closed-form expression for the loss function of our algorithm. In continuous control, this loss function is intractable and we therefore derive an approximation with hyperparameter whose value regulates pessimism in Q-Learning. We present default value for pessimism hyperparameter that enables DoubleGum to outperform DDPG, TD3, SAC, XQL, quantile regression, and Mixture-of of of-Gaussian Critics in aggregate over 33 tasks from DeepMind Control, MuJoCo, Meta World, and Box2D and show that tuning this hyperparameter may further improve sample efficiency.
Introduction
Reinforcement Learning (RL) algorithms learn optimally rewarding behaviors (Sutton and Barto, 1998, 2018). Recent RI algorithms have attained superhuman performance Atari (Mnih et al., 2015), Go (Silver al., 2016), Chess (Schritwieser et al., 2020), StarCraft (Vinyals al., 2019), simulation racing (Wurman et al., 2022), and have also been used to control stratospheric balloons (Bellemare et al., 2020) and magnetic fields in Tokamak nuclear fusion reactors (Degrave al., 2022). All these aforementioned agents use Q-function (Watkins and Dayan, 1992) to measure the respective quality of their behaviors.
Many algorithms (Lillicrap et al., 2015; Mnih et al., 2016, 2015; Schulman et al., 2015; Fujimoto al., 2018; Haarnoja et al., 2018b) learn Q-function by minimizing the Mean- Mcan-Squaree Squared Bellman Error (MSBE). The functional form of the MSBE was motivated by an analogy to the popular Mean-Squared Error (MSE) in supervised learning regression problems, but there is little theoretical justification for its use in RL. (Bradtke and Barto, 1996; Ernst et al., 2005; Riedmiller, 2005).
Training deep neural network to minimize the MSBE is empirically unstable (Irpan, 2018; Henderson et al., 2018; Van Hasselt et al., 2016) because deep neural networks induce overestimation bias in the Q-function (Thrun and Schwartz, 1993; Fujimoto et al., 2018). 8). A popular method to reduce overestimation 'pessimistically' selects less positive Q-values from an ensemble of Q-functions, effectively returning low-quantile estimates (Fujimoto et al., 2018; Kuznetsov et al., 2020; Ball et al., 2023). However, an ensemble is computationally expensive and quantiles only provide discrete- grained control over the degree of pessimism. Curiously, pessimism has mainly been applied to "contimuous control' problems that require continuous-valued actions and not discrete control.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

This paper analyzes noise in Deep Q-Learning We introduce loss function for Deep Q- Q-Learning and scalar hyperparameter that adjusts pessimism without an ensemble. These two innovations form Double Gumbel Q-L learning (DoubleGum), an off- Deep Q-Learning algorithm applicable to both discrete and continuous control. Our paper is structured as follows:
1. Section 2.1 shows that the MSBE implicitly uses noise model of the Bellman Optimality Equation with additive homoscedastic noise. Section 4. shows that this noise model is empirically too coarse to fully capture the distribution of noise in Deep Q-Learning.
2. Section 3 derives a noise model of the Soft Bellman Equation with additive heteroscedastic logistic noise derived from two heteroscedastic Gumbel noise sources. Section 4.1 shows that our noise model is a better empirical fit to the noise distribution Deep Q-Learning.
3. Section 3.3 shows that when the soft value function in the Soft Bellman Equation is in- tractable in continuous control, its approximation leads to methods that adjust pessimism. Section 3.1 presents a method to adjust pessimism with a single scalar hyperparameter, and its effecc is empirically verified in Section 4.2.
4. Section 5 discusses related work, notably presenting DoubleGum as a special case of Distri- butional RL and generalization of existing Maximum Entropy (MaxEnt) RL algorithms.
5. Section 6 benchmarks DoubleGum, showing marginal improvements over DQN baselines on 3 classic discrete control tasks and improved aggregate performance over DDPG, SAC, TD3, XQL and two Distributional RL algorithms over 33 continuous control tasks.
2 Mathematical Background
Reinforcement Learning agents learn to make maximally rewarding decisions in an environment (Sutton and Barto, 1998, 2018). In Markov Decision Process (MDP) environments, Q-function specifies the expected amount of reward an agent obtains (Watkins and Dayan, 1992) and is also known state-action value function. MDPs define transition probability p(s' a) and reward r(s, a, s for every pair of states s, s' E and action a E A (Bellman, 1957; Howard, 1960). Decisions are made by a policy Ta 8) at every discrete timestep and the Q-function of 8, is
Qx(s,a)= = E Σ
rry 8 5,0;
denoting r(sj, Ujs 85+1) as Tj.
where the discount factor y EJO, 1[ ensures finite Q for infinite time-horizon (Samuelson, 1937). The expectation in the Q (-ffucctoon is computed over trajectory sequence of s and induced by the coupling of the policy and the transition function.
In every MDP, an optimal policy πt makes maximally rewarding decisions following s + arg max, Q* a) (Watkins and Dayan, 1992). Q* is the optimal value function specified as the unique fixed point of the Bellman Optimality Equation of the MDP (Bellman, 1957)
as(
Ir r y max ('(' over all s and a, denoting r(s, a, s') as r. (1)
p(='s,a)
In Deep Q-Learning, Q" is approximated by neural network Qo with parameters 6. Following Munos and Szepesvári (2008), is updated by sequence of optimization problems whose 7th stage iteration is given by
0 = arg min L(Qo, Va.), where (s a)
r [ max Qa (s,(5','))
p(s"[s.a)
where the 'stop-gradient" operator (+) denotes evaluating but not optimizing an objective with respect its argument and y is dubbed the bootstrapped target. In this work, we restrict our environments to deterministic MDPs, so the resultant expectation over s' is computed over one sample.
Typically, the loss function L(Qe. va.) E.,a (Qo(s, a) ya, (s, a))² Error (MSBE). At every i, minimized with stochastic gradient descent by fixed number of gradient descent steps and not optimized to convergence (Mnih et al., 2015; Lillicrap al., 2015).
2 is the Mean Mean-Squared Bellman
'Code: tttps://githin /tobb/lyythouuueuuumu

---

2.1 Deriving the Mean-Squared Bellman Error
Thrun and Schwartz (1993) model Qs(s, a) as a random variable whose value varies with 6, and a. They set up generative process Qo(s, a) = Q* a) c0,s,a showing that the function approximator Qi is a noisy function of the implicit true value Q", but they do not make any assumptions about e.
We derive the MSBE with additional assumptions: c0.e.a tic normal due to the Central Limit Theorem (CLT) and secondly Q*(s,a) a) ya(s, a),, where are the parameters at an arbitrary optimization stage. Incorporating these two assumptions into the noise model yields yā(s, a) - Qa(s, a) + 60,s,ax uncovering the implicit assumption that Temporal- Difference (TD) errors Ue - Qo follow a homoscedastic normal. Maximum Likelihood Estimation (MLE) of the resultant noise model yields the MSBE after abstracting away the constants
lid N0 o²), where W is a homoscedas-
max E logp(ua(s, 0 s,0
a)) min E 0 s,a
logo 2 + 02 (Qa(s,a) a) uo(s, us(),a))²
2.2 The Limiting Distribution Bootstrapped Targets
Both assumptions used to derive the MSBE theoretically weak. First, the CLT states that iid samples of an underlying distribution tend towards a homoscedastic normal but do not account for the form of the underlying distribution. Secondly, there analytic justification for Q" (s,s, a) s yo(s, a). Substituting the Thrun and Schwartz (1993) model in the RHS of the Bellman Optimality Equation yields
(s) p(s)(s,a)
r y max Qo(s', d') + co.st
[r + 190.0/ * ya(s,
p(='(sa)
When the max over a finite number of iid samples with unbounded support is taken, the Extreme Value Theorem (EVT) gives the limiting distribution of the resultant random variable as a Gumbel distribution (Fisher and Tippett, 1928; Gnedenko, 1943), defined here go, G(a, [8). Expressions relating a and 3 to the parameters of the underlying iid random variables are rarely analytically expressible (Kimball, 1946; Jowitt, 1979) and there guarantee Q* (*(s,a) (s,
3 Deriving Deep Q-Learning Algorithm from First Principles
We find an analytic expression that relates the limiting Gumbel distribution to parameters of the underlying noise distribution from function approximation. To do so, assume that noise in the Thrun and Schwartz (1993) model a heteroscedastic Gumbel noise with state-dependent spread
Qo(s, a) (s, (s., (-,a(-) G(O, Bo(+),, for all A
(2)
Appendix A. 1 restates the finding of McFadden al. (1973) and Rust (1994), which results in the following expression with the soft value function 1/2aft
[Q(s,a)) da + go(s)
max (s, = max [Qo(s, a) + g0.a ) = (e(s) log exp
Ba(s)
Vi (s) + go (s), where g0.a (.). id g(., (.)
(3)
Substituting Equations 2 and 3 into 1 yields new noise model of the Soft Bellman Optimality Equation with two function approximators and two heteroscedastic Gumbel noise sources
Q0(3,a) a) + (s) p(=1ssa)
E
[r + vV2 (s') )gg(('))
(4)
We now develop this noise model into a loss function used our algorithm, DoubleGum.
"This equation derives from an unnumbered equation at the top of Page 3 of Thrun and Schwartz (1993), and we have renamed their variables to fit our notation. The authors add the noise e to Q', but we subtract it for notational clarity later on without loss of generality. We additionally subscript the noise by 0 to clarify that the random variable is resampled for different 0s in the function approximator.
'Ernst et al. (2005) and Riedmiller (2005) introduced the MSBE to Q Q-Learning, arguing that MSE was popular for regression problems in supervised learning. In regression, the MSE arises from MLE parameter estimation of homoscedastic normal, which is assumed to exist because of the CLT.
3

---

3.1 Merging Two Distributions
We replace Qo with new function approximator (s. Q0((s,) go(s), resulting in new soft value function on the RHS of Equation 4
cxp (800(,,))) Bo(s)
(s,
V (s) + go = (s) log
da Vg new (s) (Appendix A.2),
and LHS with a heteroscedastic logistic random variable obtained in Appendix A.3 from the difference of two Gumbel random variables with equation spreads. Note that this operation is only permissible in MDPs without terminating states. Equation 4 simplifies into
Q8 a) (s)
E p(=)laa)
+ Vo new (s)) (e,a()) (0,
(5)
the Soft Bellman Equation with heteroscedastic logistic noise. The bootstrapped targets in this equation the RHS, which will denote by (s,
3.2 Parameter Estimation with Moment Matching
Equation 5 does not allow MLE because the heteroscedastic logistic is not a member of the exponential family and lacks sufficient statistical estimator. We learn 0 with moment matching by MLE a heteroscedastic normal y/ (s. ~NQ (s, a), ou(s)) given by
y6 (s, a)
neo (s.a)
min
log) +
(6)
(8())
and recovering the logistic location and spread as Q8 (s, a) and Bo(s) = (o)s) v3
3.3 Computing the Bootstrapped Targets
soft is calculated with one sample from s' and a' where appropriate. In discrete control, A is finite yo
and the expression Vrooft new is sum. The resultant bootstrapped target in discrete control is
discrete (s, a, T, = V Vo
V/stt,
soft, new (s')
In continuous control, V-soft, new is intractable because A contains infinitesimally small quantities. Appendix A.4, adapted from Equation 19 Haarnoja al. (2017), variationally approximates the integral with respect to policy neural network aa s) with parameters Φ, producing
Q8 (s, a,
soft, new (s)=
(Q (s, a)] + Bo(s) C(T Po]. where (a s) x exp (a|ss)
E
(7)
(o(s)
Denoting CiT Po] as constant
yields bootstrapped targets of
v
yo
continuous
(s, a, r.
17% (s',a') (s0) (a(aa(s))
(8)
We treat as a hyperparameter named the pessimism factor defined before training and held constant throughout. Appendix F.1 empirically finds that a default value of -0.1 and details how may be adjusted. Appendix B.3 shows that the pessimism factor performs the same role as pessimistic ensembles but with more computational efficiency and fine-grained control.
3.4 Algorithm
Algorithm presents pseudocode for DoubleGum, an off-policy Deep Q-Learning algorithm that learss in both discrete and continuous control. DoubleGum is named after the two Gumbels used in the derivation of its noise model, Equation 4. We outline some implementation details that empirically improved the stability of training.
Target Networks: As is standard, discrete and continuous are computed with target networks, whose parameters we denote by z.. We used target networks with exponential moving weight updating. This

---

Algorithm 1: DoubleGum: An Off-Policy Deep Q-Learning Algorithm
Input: MDP A,,, replay buffer D (Lin, 1992), initial parameters 0g (and do in continuous control), initial state 8, learning rates 7o and 1/a, target network EMA rate Th
batch size n.
Output: Parameters 0 (and in continuous control)
discrete control then define (s)) max, Q8, (8, a) at iteration i
define y y piiserete
4 define o = de, (s) at iteration i
5 else if continuous control then define (ss) F.; (s) at iteration i
define y = y cominnuuu
8 define o
(s, a) at iteration i
9
2/0 < 80
for iteration i do
11
a - π (s})
8' pls 8)
13
+ DU {(8, (8,,,,)), s s
14
Oc+1 + +0; -0;-10 11 Vo
E
Jo Jo
(s.a.r,'
6.
16
Vi+1 + - + n 0 1441
if continuous control then
18
Di+1 <- 0: + n V
E (s,0 nm
QQ (s, TT(())]]
D
commonly used in continuous control (Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018b), and has been recently been shown to improve discrete control (D'Oro et al., 2022).
O-Functions with Variance Networks: We learn a with variance networks (Nix and Weigend, 1994;
Kendall and Gal, 2017). In discrete control, (s))) is variance head added to the Q-network, and is approximated as an (s, a) in continuous control. The variance head is single linear layer whose input is the penultimate layer of the main Q-network and output is single unit with SoftPlus activation function (Dugas et al., 2000) that ensures its positivity. Seitzer et al. (2022) improves stability during training by multiplying the entire loss by og. the numerical value of the variance network, yielding loss function for which for continuous control is
Q (s,a) (0,0(000000 )3]
oo(s,a) a) (10g
Ho (s, (o a (0(8.0)
Jo
Note that in the above equation ag(s, a) would be (s)s) in discrete control. We do not have any convergence guarantees for this loss function, and Appendix C discusses this issue in more detail.
Policy: In discrete control, actions are taken by max, QIE"(s a) as standard. Exploration was performed by policy churn (Schaul et al., 2022). In continuous control, the policy Fy. is separate network. We use a DDPG Exee-variance actor because Yarats et al. (2021) showed it trained more stably than SAC actor with learned variance. Following Laskin et al. (2021), the actor's 's outputs were injected with zero-mean Normal noise with a standard deviation of 0.2 truncated to 0.3.
Network Architecture: All networks had two hidden layers of size 256, ReLU activations (Glorot et al., 2011), 1), orthogonal initialization (Saxe et al., 2013) with gain of /2 for all layers apart from the last layer the policy and variance head, which had gains of This initialization had been shown to be empirically advantageous in selicyy-rrdienttt and Q-Learning methods in Huang et al. (2022a) and Kostrikov (2021), respectively. Ball et al. (2023) improves stability continuous control using LayerNorm (Ba et al., 2016). We find that the similar method of GroupNorm (Wu and He, 201 8) with 16 groups without shift and scale (Xu et al., 2019) worked better, but only continuous control. All parameters were optimized by Adam (Kingma and Ba, 2014) with default hyperparameters.
5

---

Frequency Density
Prequency Density
0.6
244
0.0
83
1.5
0:4
0.3
0.2
0.0
0.5
0.
0.2
0.4
0.6
TD Error
Standardized TD Error
Timesteps millions)
Empirical (Histogram)
Hetero-L (expected) Homo-Gumbel (Garg et al., 2023)
Moment-Matcher Hetero-Normal
Homo-Normal
Figure 1:
The empirical distribution of noise in discrete control CartPole-v1. (a, b): Fitted
normal, logistic, and Gumbel distributions against (a) unstandardized and (b) standardized empirical distributions in Deep Q-Learning at the end of training. (c): Negative Log-Likelssoods (NLLs) of the noise in Deep Q-Learning under different distributions throughout training (lower is better). Appendix D.I presents further results in more discrete and continuous control tasks.
acrobot-swingup
MuloCo: Hopper-ve
Box2d: Bipedal Walker//alker-v3
Meta World: button- pittoonrrees-22
1500
1900
430
1600
Magnitude
1330
1200
900
800
300
400
20
60
100
0
20
40
60
80
100
0
20
40
60
80
100
20
40
62
100
Timestops (in millions)
c=00.5
c=-0.1
c=0.0
c=0.1
c=0.5
Figure 2: The effect of changing pessimism factor the target Q-value in continuous control. Appendix D.2 presents further results in more tasks.
4 Empirical Evidence for Theoretical Assumptions
4.1 Noise Distributions in Deep Q-Learning
Figure 1 shows the evolution of noise during training DoubleGum in the classic control task of CartPole-vi. Histograms in Figures 1(a) and 1(b) were generated from 10 samples from the replay buffer after million training timesteps. Figure 1(a) computes TD- by Be (s, a, r') Q (8, a), while Figure 1(b) computes standardized TD-errors by dividing the previous equation by (s(s) (a(s, a.) where appropriate. The normal, logistic, and Gumbel distributions were fitted by moment matching, and homo(scedastic) and hetero(scedastic distributions were respectively fitted to unstandardized and standardized data. Figure (cc) shows the mean and standard deviation 12 Negative Log-Likelihoods (NLLs), each computed from a different training run with a different (randomly chosen) initial seed. Every NLL computed 10 samples from the replay buffer every 1000 timesteps of training.
Figure I(a) shows that homoscedastic normal coursely fits the empirical distribution, forming good estimate of the mean but not the spread. Our result contradicts Garg et al. (2023) that fitted Gumbel to the empirical distribution. We discuss this discrepancy in Appendix E. Figure 1(b) shows that heteroscedastic Logistic captures both the mean and spread of the TD-errors, validating Equations 5 and its derivation. Finally, Figure 1(c) shows that moment-matched heteroscedastic normal used in DoubleGum is a suitable approximation to the heteroscedastic logistic throughout training, validating Equation 6. Appendix D.I shows that the trend in Figure ((c) holds for other discrete control environments as well as continuous control.
4.2 Adjusting The Pessimism Factor
1 Figure 2 plots a 12-sample IQM and standard deviation over 256 E Q (s,, a;), the average magnitude of the target Q-value used in bootstrapped targets. Figure 2 and Appendix D.2 show that the average magnitude increases the pessimism factor increases, validating its effectiveness.
256

---

5 Related Work
5.1 Theoretical Analyses of the Noise in Q-Learning
Logistic Q-Learning Bas- Bas-Serrano et al. (2021) presents a similar noise model to DoubleGum but with homoscedastic logistic instead of a heteroscedastic logistic in Equation 5. Their distribution derived from the pnnaamnrogramming perspective of malue-function learning described in Section 6.9 of Puterman (2014)) and Peters et al. (2010). While DoubleGum learns Q-function off off-policy, Logistic Q-Learning uses on- on-policy rollouts to compute the Q-values of the current policy. We do not benchmark against Logistic Q Q-Learning because their method is on-policy and only uses linear function approximation.
Extreme Q-Learning (XQL) Garg et al. (2023) presents a noise model for Deep Q-Learning with one homosecdastic Gumbel noise source, as opposed to the two heteroscedastic Gumbels in Equation (LINEX) loss Varian (1975) formed from the log-likelihood of a Gumbel distribution. XQL is presented in more detail in Appendix B.4.
4. Parameters are learned by the
Gumbel Noise in Deep Q-Learning: Thrun and Schwartz (1993) argued that the max-operator in bootstrapped targets transformed zero-mean mean noise from function approximation into statistically biased noise. The authors did not recognize that the resultant noise distribution was Gumbel, and this was realized by Lee and Powell (2012). Unlike DoubleGum, these two works did not make assumptions about the distribution of function approximator noise but instead focused on mitigating the overestimation bias of bootstrapped targets. In economics, McFadden et al. (1973) and Rust (1994) assume the presence of Gumbel noise from noisy reward observations (unlike DoubleGum, which assumes that noise comes from function approximation) and derives the soft value function we present in Appendix A.1 for static and dynamic discrete choice models. XQL brings the Rust-McFadden et al. model to deep reinforcement learning to tackle continuous control.
The soft value function was introduced to model stochastic policies (Rummery and Niranjan, 1994; Fox et al.. 2015; Haarnoja al., 2017, 2018b). The most prominent algorithm that uses the soft value function is Soft Actor-Critic (SAC) (Haarnoja et al., 2018a,b), which Appendix B.2 shows a special case of DoubleGum when Q7 learned by homoscedastic instead of heteroscedastic normal regression and the spread is a tuned scalar parameter instead of learned state-dependent standard deviation. Appendix B.2 also shows that Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al., 2015) is simpler special case of SAC that has recently been shown to outperform and train more stably than SAC (Yarats et al., 2021).
5.2 Empirically Similar Methods to DoubleGum
Distributional RL models the bootstrapped targets and Q-function as distributions. In these methods, a Q-function is learned by minimizing the divergence between itself and a target distribution. The most similar distributional RL method to DoubleGum is Mixture-of-Gausse al. (2018). DoubleGum is special case of MoG-
(MoG) Critics,
introduced in Appendix A of Barth-Maron Critics with only one Gaussian (such as in Morimura et al. (2012)) and mean samples of the target distribution. Curiously, Shahriari al. (2022) shows that sample-efficiency of training improves when bootstrapped target sampled are increasingly near the mean but did not try mean sampling. Nevertheless, Shahriari et al. (2022) show that MoG-Critics outperforms a baseline with the C51 distributional head (Bellemare et al., 2017) popular in discrete control, obtaining state-of-the-art results in DeepMind Control. In discrete control with distributional RL, C51 has been superseded by Quantile Regression (QR) methods (Dabney et al., 201 8b,a; Yang et al., 2019) that predict quantile estimates of a distribution. Ma et al. (2020); Wurman et al. (2022) and Teng et al. (2022) apply QR to continuous control.
Adjusting pessimism greatly improves the sample efficiency of RL. Fujimoto et al. (2018) showed empirical evidence of overestimation bias in continuous control and mitigated it with pessimistic estimates computed by Twin Networks. However, Twin Networks may sometimes harm sample efficiency because the optimal degree of pessimism varies across environments. To address this, the degree of pessimism adjusted during training (Lan et al., 2020; Wang et al., 2021; Karimpanal et al., 2021; Moskovitz et al., 2021; Kuznetsov et al., 2020, 2022; Ball et al., 2023), often with the help of an ensemble. In contrast, DoubleGum uses one nentwork and one scalar hyperparameter fixed throughout training.
7

---

CartPole-vl
Acrobot-v1
Mountain Car-vO
-00
1QM of Retur
-10
4.0
-00
-150
150
190
210-
-600
80
40
Timesteps (in thousands)
DoubleGum (Ours)
DQN
Ducling DDQN
Figure 3: Discrete control, IQM of returns + standard deviation over 12 seeds.
Many other RL methods add or subtract the learned standard deviation to bootstrapped tar- gets. Risk-Aware RL subtracts learned standard deviation to learn low-variance policy (La and Ghavamzadeh, 2013; Tamar and Mannor, 2013; Tamar et al., 2016). Upper-Confidence Bounded (UCB) methods add learned standard deviation explore high-variance regions (Lee al., 2021; Teng et al., 2022). These methods use a combination of ensembles and variance networks, but it is also possible derive a Bellman equation for the variance following Dearden et al. (1998). The current state-of-the-ae method in RL with variance estimation is Mai et al. (2022), which uses both variance networks and ensembles. The of ensembles motivated by the need capture model uncertainty differences between Q-functions with different parameters. We believe that ensembles are unnecessary because model uncertainty will be expressed in bootstrapped targets from two different timesteps as parameters change through learning and that all variation in bootstrapped targets will henceforth be captured by variance networks.
Results
6.1 Discrete Control
We benchmarked DoubleGum on classic discrete control tasks against two baselines from the DQN family of algorithms (Figure 3). All algorithms were implemented following Section 3.4. Appendix E.3 discusses the baseline algorithms more detail.
Performance was evaluated by the InterQuartile Mean (IQM) over 12 runs, each one with a different randomly initialized seed. Agarwal al. (2021) showed that the IQM was a robust performance metric in RL. 12 was chosen because it was the smallest multiple of four (so IQM could be computed) greater than the 10 seeds recommended by Henderson et al. (2018). In each run, the policy was evaluated by taking the mean of 10 rollouts every 1000 timesteps, following Fujimoto al. (2018).
Figure shows that DoubleGum sometimes obtains better sample efficiency than baselines, but not significantly more to necessitate further discrete control experiments. In the remainder of this work, we focus on continuous control, where we found DoubleGum to more effective.
6.2 Continuous Control
We present two modes of evaluating DoubleGum because our algorithm has a pessimism factor pepraarraeeter we choose to tune per suite. We, therefore, benchmark all continuous control algorithms with default pessimism (Figure 4) and without pessimism-tuning per-suite (Figure 5). Table 5 presents default and per-suite pessimism values.
We compare against seven baselines. The first five are popular algorithms in the continuous control literature: DDPG (Lillicrap al., 2015), TD3 (Fujimoto et al., 2018), SAC (Haarnoja et al., 2018b), MoG-Critics (Shahriari et al., 2022) and XQL (Garg et al., 2023). Here. DDPG, TD3, and SAC represent MaxEnt RL, MoG-Crities represent Distributional RL, and XQL the algorithm with the most similar noise model to ours. We introduce two further baselines: QR-DDPG, a stronger Distributional RL baseline that combines quantile regression (Dabney et al., 201 8b) with DDPG, and Fin TD3 with ensemble of five networks opposed network of two ensembles

---

Untuned
64
DoubleCum, c 0.1 (Ours)
Twin-SAC
IQM
0.3
MoG-DDPG
Twin-XQL
DDPG
QR-DDPG
TD3
FinerTD3
03 Timesteps (in millions)
Figure 4: Continuous control with default parameters, IQM normalized score over 33 tasks in 4 suites with 95% stratified bootstrap Cls. Methods that default to use Twin Networks are dashed. Appendix F.4 presents per-suite and per-task aggregate results.
Tuned per suite
0.4
DoubleGum, best c, (Ours)
XQL (best of w/wo Twin)
IQM
MoG-DDPG (untuned)
QR-DDPG (best w/wo Twin)
0.2-
Best of DDPG/TD3
Fine/TD3 (best pessimism)
0.1
SAC (best w/wo Twin)
02
0.5
15
Timesteps (in millions)
Figure 5: Continuous control with the best pessimism hyperparameters tuned suite, IQM normal- ized score over 33 tasks 4 suites with 95% stratified bootstrap Cls. Appendix F.5 presents per-suite and per-task aggregate results.
in the original TD3 that enables finer control over the degree of pessimism with ensemble of five networks. All algorithms were implemented following design decisions outlined Section 3.4. Appendix E.4 discusses the algorithms in more detail.
The pessimism of DoubleGum was tuned by changing its pessimism factor. The pessimism of baseline algorithms was tuned by manually choosing whether to use Twin Networks (Fujimoto et al., 2018) or not. Note that we refer to DDPG with Twin Networks as TD3. The pessimism of FinerTD3 was tuned by selecting which quantile estimate to use from ensemble of five networks. The pessimism of MoG-Critics could not be tuned because critic does not support paired sampling between ensemble members. Appendices F.1 and F.2 respectively detail how the pessimisms of DoubleGum and baseline algorithms were adjusted.
We benchmarked DoubleGum on 33 tasks over 4 continuous control suites comprising respectively of 11 DeepMind Control (DMC) tasks (Tassa et al., 2018: Tunyasuvunakool al., 2020), 5 MuJoCo tasks (Todorov et al., 2012; 2; Brockman et al., 2016), 15 Meta World tasks (Yu et al., 2020) and 2 Box2D tasks (Brockman et al., 2016). We follow Agarwal et (2021) and evaluate performance with the normalized IQM with 95% stratified bootstrap confidence intervals aggregated over all 33 tasks. 12 runs from each task was collected by a similar method to that described in Section 6. Further details of the tasks and their aggregate metric are detailed in Appendix E.2.
Figure 4 shows that DoubleGum outperformed baselines in aggregate over 33 tasks when all algorithms used their default pessimism settings. Figure 5 shows that DoubleGum outperformed all baselines in aggregate over 33 tasks when the pessimism of all algorithms is adjusted per suite. Comparing the figures shows that adjusting the pessimism of DoubleGum per suite also attained a higher aggregate score than DoubleGum with default pessimism.
7 Discussion
This paper studied the noise distribution in Deep Q-Learning from first principles. We derived a noise model for Deep Q-Learnigg that used two heteroscedastic Gumbel distributions. Converting our noise model into an algorithm yielded DoubleGum, an off off-policy Deep Q Q-Lemmnng algorithm applied to both discrete and continuous control.
In discrete control, our algorithm attained competitive performance to the baselines. Despite having an numerically exact loss function in discrete control, DoubleGum was very sensitive to hyperparameters.

---

Practically, using Dueling DQN (Wang et al., 2016) to learn Q-function was crucial to getting DoubleGum to work. Appendix D.1 shows that our noise model fits the underlying noise distribution of Deep Q-Learning and we therefore suspect that instability in discrete control might be due to the training dynamics of deep learning and not our theory.
In continuous control, we introduced a pessimism factor hyperparameter to approximate our otherwise intractable noise model. We provided a default value for the pessimism factor that outprroomedd popular Q- Q-Learning baselines in aggregate over 33 tasks. Tuning this hyperparameter yielded even greater empirical gains. Our method of tuning pessimism was more computationally efficient and finer-grained than popular methods that tuned a quantile estimate from an ensemble.
In continuous control, DoubleGum outperformed all baselines in aggregate. We hypothesize that Dou- bleGum outperformed MaxEnt RI baselines because DoubleGum is a more expressive generalization of SAC, which is itself more expressive than DDPG shown in Appendix B.2. Our theory showed that TD-errors follow a heteroscedastic Logistic, and we believe that modeling this distribution should be sufficient for distributional RL. We hypothesize that more complex distributions considered by the Distributional RL methods QR and MoG overfit to the replay buffer and might not generalize well to online rollouts. FinerTD3 performs marginally poorer than DoubleGum, even when pessimism was adjusted per-suite. We believe this is because FinerTD3 adjusts pessimism finer than other baselines, but still not as fine as the continuous scalar in DoubleGum. Finally, we hypothesize that DoubleGum outperformed XQL because our noise model better fits the underlying noise distribution in Deep Q-Learning, as shown in Appendix D.1.
This paper shows that better empirical performance in Deep RL may be attained through a better understanding of theory. To summarize, we hope that work encourages the community to increase focus on reducing the gap between theory and practice to create reinforcement learning algorithms that train stably across a wide variety of environments.
7.1 Limitations
Theoretically. our work lacks a convergence guarantee for DoubleGum. This is exceptionally challenging because the best of our knowledge there are currently no convergence guarantees for heteroscedastic regression. Appendix discusses convergence in more detail.
Experimentally, there are many areas left open for future work. For speed in proof of concept experiments, we only swept over five values for the pessimism factor hyperparameter and only tuned per-suite. We anticipate that a more thorough search of the pessimism factor combined with per-task selection will improve our results even further. An obvious follow-up would be an algorithm that automatically adjusted the pessimism factor during training. Additionally, we only focused on environments with state observations to not deal with representation learning from visual inputs. Another obvious next step would be to train DoubleGum on visual inputs or POMDPs.
Acknowledgements
We would like to thank Clement Gehring, Victor Hui, Sobhan Mohammadpour; Tianwei Ni, Anushree Rankawat, Fabrice Normandin, Ryan D' Orazio, Jesse Farebrother, Valentin Thomas, Matthicu Geist, Olivier Pietquin, Emma Frejinger, and the five anonymous reviewers.
All algorithms and code were implemented in JAX (Bradbury et al., 2018) and Haiku (Hennigan et al., 2020) and cpprb (Yamada, 2019). Helper functions for running RL experiments were taken from Kostrikov (2021); Wu (2021); Hoffman et al. (2020) and Agarwal et al. (2021). Compute requirements are listed in Appendix E.5.
This research was funded by CIFAR and Hitachi and enabled in part by compute resources, software and technical help provided by Mila quebec), Calcul Québec (calculquebec. ca) and the Digital Research Alliance of Canada (alliance. can. ca).
Hui carried out the majority of his work in Section of Mila and would like express his wish for the silent (but vibrant) working conditions there to continue for many years to come.
10

---

References
Agarwal, R., Schwarzer, M., Castro, S., Courville, A. C., and Bellemare, M. (2021). Deep reinforcement learning at the edge of statistical precipice. Advances in neural information processing systems, 34:29304- 29320.
Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normaizaaion. arXiv preprint arXiv:1607.064:
Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. (2023). Efficient online reinforcement learning with offline data, arXiv preprint 21Xi::3302.02443
Barth-Maron, G., Hoffmmnn, M. W., Budden, D., Dabney, W., Horgan, D., Dhruva, T., Muldal, A., Heess, N., and
Lillicrap. T. (2018). Distributed distributional deterministic policy gradients. In International Conference Learning Representations.
Bas-Serrano, Curi, Krause, A., and Neu, (2021). Logistic q-learning. International Conference on Artificial Intelligence and Staristics, pages 3610-3618. PMLR.
Bellemare, M. G., Candido, S., Castro, P. S., Gong. J., Machado, M. C., Moitra, S., Ponda, S., and Wang. (2020). Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77- 82.
Bellemare, M. G., Dabney, W., and Munos, R. (2017). A distributional perspective reinforcement learning. In International conference on machine learning, pages 449-458. PMLR.
Bellman, R. (1957). markovian decision process. Journal of mathematics and mechanics, pages 679-684.
Bradbury, J., Frostig, R., Hawkins, Johnson, M. Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander- Plas, J., Wanderman-Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs.
Bradtke, J. and Barto, A. G. (1996). Lincar least-squares algorithms for temporal difference learning. Machine learning, 22(1-3)33--7.
Bridle, J. (1989). Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. Advances neural information processing systems,
Bridle, J. (1990). Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pages 227-236. Springer.
Brackman, G., Cheung, V., Pettersson, I.., Schneider, J., Schulman, J., Tang, J., and Zaremha, W. (2016). Openai gym.
Ceron, J. S. O. and Castro, P. S. (2021). Revisiting rainbow: Promoting more insightful and inclusive deep
reinforcement learning research. In International Conference Machine Learning, pages 1373-1383. PMLR.
Chen, X., Wang. C., Zhou, Z., and Ross, K. W. (2021). Randomized ensembled double q-learing: Learning fast without model. International Conference on Learning Representations.
Dabney, W., Ostrovski, G.. Silver, D., Munos, R. (2018a). Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096-1105. PMLR
Dabney, W., Rowland, M., Bellemare, M., and Munos, R. (2018b), Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 32.
Dearden, R., Friedman, N., and Russell, S. (1998), Bayesian q-learning. Auaifiaii, 1998: 099:76117668.
Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki. de Las Casas, D., et al. (2022). Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602((897):414 419.
D'Oro, P. Schwarzer. M. Nikishin, E. Bacon, P.-.,, Bellemare, M. G., Courville. A. (2022). Sample- efficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurlPS 2022.
Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., and Garcia, R. (2000). Incorporating recond-orded functional knowledge for better option pricing, Advances neural information processing systems, 13.

---

Ernst, D., Geurts, P., and Wehenkel, L. (2005). Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6.
Farama Foundation (2023). Drrama-Foundationn/yymmme
original-date: 0022-09-08T01:58800
Fisher, R. A. and Tippett, L. H. C. (1928). Limiting forms of the frequency distribution of the largest or smallest member of sample. In Mathematical proceedings of the Cambridge philosophical society, volume 24, pages 180-190. Cambridge University Press.
Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., et al. (2017). Noisy networks for exploration. arXiv preprint arXiv:1706. 10295.
Fox, R., Pakman, A., and Tishby, N. (2015). Taming the noise in reinforcement learning via soft updates. arXiv preprint 122.v225...
Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR.
Garg, D., Hejna, J., Geist, M., and Ermon, S. (2023). Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.0255 .03328.
Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315-323. JMLR Workshop and Conference Proceedings.
Gnedenko, B. (1943). Sur la distribution limite du terme maximum d'unc serie alcatoire. Annals of mathematics, pages 423 -53..
Gumbel, E. J. (1935), Les valeurs extrêmes des distributions statistiques. In Annales de l'institut Henri Poincaré, volume 5, pages 115-158.
Haamoja, T., Tang, H., Abbeel, and Levine, S. (2017). Reinforcement learning with doep energy- policies. In International conference on machine learning, pages 1352-1366 1361. PMLR.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018a). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMIR.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., ct al. (2018b). Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.05905.
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger. D. (2018). Deep reinforcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Hennigan, T., Cai, T. Norman, T., and Babuschkin, (2020). Haiku: Sonnet for JAX.
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D.,, Piot, B., Azar, M.. and Silver, D. (2018). Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence.
Hinton, G., Srivastava, N., and Swersky, K. (2012). Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2.
Hoffman, M. W., Shahriari, B., Aslanides, J., Barth-Maron, G., Momchev, N., Sinopalnikov, D., Stariczyk, P., Ramos. S., Raichuk, A., Vincent, D., Hussenot, L., Dadashi, R., Arnold G., Orsini, M., Jacq, A., Ferret, J., Vieillard, N., Ghasemipour, S. K. S., Girgin, S., Pietquin, O., Behbahani, F., Norman, T., Abdolmaleki. A.. Cassirer, A., Yang. F., Baumli, K., Henderson,
Friesen, A., Haroun, R., Novikov, A., Colmenarejo, S. G., Cabi, S., Gulcehre, C., Paine. T. L.. Srinivasan, S., Cowie, A., Wang. Z., Piot, B., and de Freitas. N. (2020). Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.000
Howard, R. A. (1960), Dynamic programming and markov processes. John Wiley.
Huang. S., Dossa, R. F. J., Raffin, A.. Kanervisto, A., and Wang, W. (2022a). The 37 implementation details of proximal policy optimization. In ICLR Blog Track. bttp:://icle-blogttaak.... gt0t implementation-detall.
Huang. S., Dossa, R. F. J.. Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Araújo, J. G. (2022b). Cleanrl: niihhquulity single-file file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research. 138
12

---

Huber, J. (1992). Robust estimation of location parameter. In Breakthreuuhs in statistics: Methodology and distribution, pages 492-518. Springer.
Irpan, (2018). Deep reinforcement learning doesn't work yet. 14/r1-hard.hh html.
Jowitt, P. (1979). The extreme-value type-1 distribution and the principle of maximum entropy. Journal of Hydrology. 42(1-2))23-38.
Karimpanal, T. G., Le, H., Abdolshah, M., Rana, S., Gupta, S., Tran, T., and Venkatesh, S. (2021). Balaneed q-learnin: Combining the influence of optimistic and pessimistic targets. arXiv preprint arXiv:2111.02787.
Kendall, A. Gal, (2017). What uncertainties we need bayesian deep learning for computer vision? Advances in neural information processing systems, 30.
Kimball, B. F. (1946). Sufficient statistical estimation functions for the parameters of the distribution of maximum values. The Annals of Mathematical Statisticx, 17(3):299-309.
Kingma, D. and Ba, (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv: arXiv:1412.6980. Kostrikov, I. (2021). JAXRI:: Implementations of Reinforcement Learning algorithms in JAX.
Kuznetsov, A., Grishin, A., Tsypin, A., Ashukha, A., Kadurin, A., and Vetrov, (2022). Automating control of overestimation bias for reinforcement learning.
Kuznetsov, A., Shvechikov, P., Grishin, A., and Vetrov, D. (2020). Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pages PMLR.
La, and Ghavamzadeh, M. (2013). Actor-critic algorithms for risk-sensitive mdps. Advances neural information processing systems, 26.
Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. (2019). Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.080 1910,09700.
Lan, Q., Pan, Y., Fyshe, A., and White, M. (2020). Maxmin q-learning: Controlling the estimation bias of q-learning. arXiv preprint arXiv:2002.06497,
Laskin, M., Yarats, D., Liu, H., Lec, K., Zhan, A., Lui, K., Cang, C., Pinto, L., and Abbeel, (2021). Urlb: Unsupervised reinforcement learning benchmark. Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Lee, D. and Powell, W. (2012), An intelligent battery controller using bias-corrected q-learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 26, pages 316-322
Lee, K., Laskin, M., Srinivas, A., and Abbcel, (2021). Sunrise: A simple unified framework for ensemble learning in deep reinforcement leamning. In International Conference on Machine Learning, pages 6131-6141. PMLR.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y.. Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint 150.00979...
Lin, L.J. (1992). Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 833:293-321.
Ma, X., Xia, L., Zhou, Z., Yang. J., and Zhao, Q. (2020), Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004. 14547.
Mai, V., Mani, K., and Paull, L. (2022). Sample efficient deep reinforcement learning via uncertainty estimation. arXiv preprint arXiv:2201.01666
McFadden, D. al. (1973). Conditional logit analysis of qualitative choice behavior. Institute of Urban and Regional Development, University of California.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap. T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning. pages 1928-1937 PMLR.
13

---

Mnih. V. Kavukcuogiu, K., Silver, D., Rusu, A. A., Veness, J. Bellemare, M. G., Graves, A., Riedmiller, M.. Fidjeland, A. K., Ostrovski, G., al. (2015). Human-level control through deep reinforcement learning. nature, 518(7540):529-533
Morimura, T., Sugiyama, M., Kashima, H., Hachiya, H., and Tanaka, T. (2012). Parametric return density estimation for reinforcement learning. arXiv preprint arXiv:/208.334)
Moskovitz, T., Parker-Holder J., Pacchiano, A., Arbel, M., and Jordan, M. (2021). Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Syxtems, 34:12849- 12863.
Munos, R. and Szepesvári, C. (2008). Finite-time bounds for fitted value iteration. J. Mach. Learn, Res.. 9:815-857.
Nix, D. A. and Weigend, A. S. (1994). Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 ieee international conference neural networks (ICNN*94), 94), volume I, pages 60. IEEE
Peters, J., Mulling, K., and Altun, Y. (2010). Relative entropy policy search. Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1607-1612.
Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming, John Wiley Sons.
Riedmiller, M. (2005). Neural fitted iteration-ligg first experiences with data efficient neural reinforcement learning method. In European conference machine learning, pages 317-328. Springer.
Rummery, G. A. and Niranjan, M. (1994). On-line Q-learning using connectionist systems, volume 37. University of Cambridge, Department of Engineering Cambridge, UK.
Rust, J. (1994). Structural estimation of markov decision processes. Handbook of econometrics, 4:3081-3143.
Samuelson, A. (1937). note measurement of utility. The review of economic studies, 4(2):155-161
Saxe, A. M., McClelland, J. L., and Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120.
Schaul, T., Barreto, A., Quan, J., Ostrovski, (2022). The phenomenon of policy churn. In Advances Neural Information Processing Systems.
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized experience replay. Bengio, Y. and LeCun, Y., editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel. T., al. (2020). Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(789))604 609.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015). Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR.
Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, S.S. (2023). Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine PMLR.
Learning, pages
Seitzer. M., Tavakoli, A., Antic, D., and Martius, G. (2022). On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. arXiv preprint 1731
Scyde, T., Werner, P., Schwarting. W., Gilitschenski, I., Riedmiller, M., Rus, D., and Wulfmeier, M. (2022). Solving continuous control via q-learning. arXiv preprint arXiv:2210.125666 12566.
Shahriari, B., Ahdolmalcki, A., Byravan, A., Friesen, A., Liu, S., Springenherg. J. Heess, N., Hoffman, M.. and Riedmiller, M. (2022). Revisiting gaussian mixture critic off-policy reinforcement learning: sample-based approach. arXiv preprint arXiv:2204.
Silver. D., Huang. A., Maddison, C. J., Guez, A., Sifre. L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I. Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of with deep neural networks and tree search. nature, 529(7587)4484 489.
14

---

Sutton, R. S. and Barto, A. G. (1998). Reinforcement learning: An introduction. MIT press. Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduciion. The MIT Press, second edition.
Tamar, A., Di Castro, D., and Mannor, S. (2016). Learning the variance the reward-to-go. The Journal of Machine Learning Research, 17(1)3311-39.
Tamar, and Mannor, S. (2013). Variance adjusted actor critic algorithms. arXiv preprint 131.v73100.3697.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L.,, Budden, D., Abdolmalcki, A., Merel, J., Lefrancq, A., et al. (2018). Deepmind control suite. arXiv preprint arXiv:1801 00690.
Teng, M., van de Panne, M., and Wood, F. (2022). Exploration with multi-sample target values for distributional reinforcement learning. arXiv preprint 222.v:2202.02693.
Thrun, S. and Sehwartz, A. (1993). Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Frlbaum, volume pages 1-9.
Todorov, E.., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RS/ International Conference on Intelligent Robots and Systems, pages 5026-5033, IEEE.
Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S.. Bohez, S., Merel, J., Erez, T., Lillicrap. T., Heess, N., and Tassa, (2020), dm control: Software and tasks continuous control. Softwrre Impacts, 6:100022.
Van Hasselt, H.,, Guez, A., and Silver, D. (2016). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30.
Varian, H. R. (1975). A bayesian approach to real estate assessment. Studies in Bayesion econometrie and statistics Honor of Leonard J. Savage, pages 195-208.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R.,
Ewalds, T., Georgiev, P., (2019). Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):30-3344
Wang, H., Lin, S., and Zhang, J. (2021). Adaptive ensemble q-learnin:: Minimizing estimation bias via feedback. Advances in Neural Information Processing Systems, 34:24778-24790.
Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. (2016). Dueling network architectures deep reinforcement learning. In Internationall conference on machine learning, pages 1995-2003, PMLR.
Watkins, C. and Dayan, P. (1992). Q-learning. Machine learning, 8:279-292.
Wu, Y. (2021). ld3-jax. https:///ithur wow.///yaaaa/////daoaooaoo
Wu, Y. and He, K. (2018). Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19.
Wurman, P. R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T. J., Capobianco, R., Devlic. A., Eckert, F., Fuchs, F,, et al. (2022). Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):23-222.
Xu, J., Sun, X., Zhang, Z., Zhao, G., and Lin, J. (2019). Understanding and improving layer normalization. Advonces Neural Information Processing Systems, 32.
Yamada, H. (2019). cpprb.
Yang. D., Zhao, I.., Lin, Z., Qin, T., Bian, J., and Liu, T.-Y. (2019). Fully parameterized quantile function for distributional reinforcement learning. Advances in neural information processing systems, 32.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, (2021). Mastering visual continuous control: Improved
data-augmented reinforcement learning. arXiv preprint arXiv:2107.0966
Yu, Quillen, He, Z., Julian, R. Hausman, K.. Finn, C., and Levine, (2020). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. PMLR.
Zhang. Y., Lin, J., Li, F, Adler, Y., Rasul, K., Schneider, A., and Nevmyvaka, Y. (2023). Risk bounds on aleatoric uncertainty recovery. In International Conference on Artificial Intelligence and Statistics, pages 6015-6036. PMLR.
15

---

Appendix
Table of Contents
A Proofs
17
A.1 The SoftMax Location of Gumbel Distribution
17
A.2 Shifting the Value Function
18
A.3 The Difference between Two Gumbel Random Variables a Logistic
A.4 Soft Q-Learning Identity....
B Further Theory and Derivations
19
B.1 Actor Loss
19
B.2 Eakimum-Entroppe Reinforvement Learning
20
B.3 Interpreting the Cross-Entrops as a Pessimism Factor
20
B.4 Comparison between DoubleGum and XQL
21
Discussion The Convergence DoubleGum
21
C.I Tabular Q-Functions
21
C.2 Deep Q-Functions
23
D Further Empirical Evidence for Theoretical Assumptions
23
D.1 Noise Distributions Deep Q-Learning
23
D.2 Adjusting The Pessimism Factor
23
E Further Experimental Details
23
E.1
Noise Distribution Discrepancy with Extreme Q-Leamnng Continuous Control Benehmarks and Evaluation
23
E.2
23
E.3
Discrete Control Baselines
26
F,4
Continuous Control Baselines
29
E.5
Compute Requirements
29
F
Further Results
30
F.1
Adjusting the Pessimism of DoubleGum
................................
30
F2
Adjusting the Pessimism of Baseline Algorithms
................................
30
E3
Discrete Control
30
F4
Continuous Control with Default Pessimism
33
E.5
Continuous Control with Pessimism adjusted Per-Suite
33
16

---

A Proofs
A.1 The SoftMax Location of Gumbel Distribution
We refer to the lug-sum-exp operator as the SoftMax operator. This is not the same-named operator Bridle (1989, 1990), which suggest should be (re-)named SoftArgMax.
Theorem 1.
ai max + gi] = 3 log Σ exp
(
+ 9, where 9.9 ~ g (0,
where G a Gumbel distribution. Gumbel random variable ~ G(a, 3) specified by location TR and spread 0 has PDF p(g) exp(-2 exp(-2)) with (g -a) and CDF
P(g) exp( - exp(xxxp(=)) (Gumbel, 1935).
Proof. First note
0i + Di
9i ~ 9 (0, =
a4 + D: ~ 9: (a 3)
Then, denoting y max:|a; + gil
P(X v)
II P.(X <)
II G x))
II exp
(eexp
B
x
exp
exp
Σ
02
exp
exp
exp
3
B
a
exp
exp
log
exp
3
3
1
exp
exp
Blog
Σ exp
3
Boo
Σ exp
3
= log Σ exp
g.
where 9 ~ (~g(0) (0,3)
When applied to discrete Q-Learning we produce
max (Q)(s.a) + g9 = (s(s) log / exp (Qo(s.a) Bs(s) ) da + go(s)
(10)
where ().a()).go))) (0, (.)). for all A.
We assume that same result holds for the continuous case if |A| < co, an assumption first used in Lemma 1, Appendix 1. Page 11 of Haarnoja et al. (201 8a) to ensure boundedness. Here, we require the output of the max-operator to be bounded, which cannot the case when the number of its arguments is
17

---

A.2 Shifting the Value Function
Theorem 2.
8 log
exp
+
log
Σ exp
+ g
B
Proof.
B log
exp
04 + 9
- log Σ exp
g
(11)
3
3
3 log exp
9
Σ
exp
ai
(12)
B
3
g
ai
3 expexp exp
log
exp
(13)
B
B
- log
Σ exp
ai
+
(14)
A.3
The Difference between Two Gumbel Random Variables is a Logistic
Theorem 3.
I1, I2 ~ g(o, B) - ====0 =1 T2 2 ~ L(., 3)
where C is logistic distribution. A logistic random variable ~ L(xx 3) with location and spread exp (xp=-(())) and CDF 1+exp (-(-))))))//
has PDF
((++xx(-00-0)/88)
Proof. First, construct the convolution based on the joint PDF
((2) P(Z & 2)
= P(X, - X2 & 2)
P(X1 < x22)
g( g (22 da day
G(z+ x2) g(x2)) daz
Rewriting 22 yields
((z)
G(z + +x) g(x) dr
2 +
x exp
F dr
exp
exp
exp (-3
B
exp
-e
$/2 (1+0-*) e dr
+ - du, where a == - du
5dr
3
exp
11
J0
-
11t $ }) x
3
1
0
1
which is the CDF of ())
18

---

A.4
Soft Q-Learning Identity
Theorem 4. For an arbitrary p(x)
exp B(x)
(x)
B log
exp
dæ
E [+(]]+++ p'), where p²(x)) = x)p() ()p())
exp B(x) dx
Proof.
E [E(x)] - 3 / p(x) log
F(x)
exp
E [b()]]] + +38 11pl1>11= p(a)
da
p(z)
J exp B dr
E(x')
/ p(z))ogg log
E(x')
exp
dx' dr
3
/ exp B
E(x)
Blog
dx
When applied Q-Learning the following identity produces ((s) log / exp
Q0(s,a)) da (a(a's))) [Q((s, ((s)) C(Ta |l pol
3(s)
Qo(s,a)
where pa (s ʃ exp Cala.") da ((3)
exp
B(o)
B
Further Theory and Derivations
B.1
Actor Loss
The actor losses used in DoubleGum, SAC, and DDPG are all derived from the same principle. For given the actor loss function should minimizes the following (reverse) KL--Divergence, previously presented in Equation 7.
exp Q (s, a
min B [DKI[TTT To pol, where pa(a J exp ( (8,a')
This simplifies
min DxI. TT |l Pal min 3 / TT(a log (a da pa(a (s)
B [ + / Ta(a 8) sog
exp Q2ew (s, a)/B
max
f eexp Q (s, al da
da
/ To(al
QaeT (s.
max
8 HT&) + 3
da
B
which is then estimated by Monte- Carlo samples from To as
max [Q (s, a) 3 log (s(a 8)]
E
(15)
④ (a(ass)
SAC (Haarnoja al., 20120.082 Sa,b) has policy with learned variance and state-independent 3. DDPG (L (Lillicrap et al., 2015) has fixed-variance policy which removes the second term in Equation 15 as it constant with respect to the maximization. DoubleGum has state-dependent (s),, but uses the policy.
same actor loss DDPG because DoubleGum uses DDPG
19

---

B.2 Maximum-Entropy Reinforcement Learning
SACvl (Haarnoja et al., 2018b) is special case of DoubleGum and DDPG (Lillicrap et al., 2015) is special case of SAC. All three continuous control algorithms have an actor and critic loss derived from the same principle. Section B.1 shows this for the actor losses of DoubleGum, SAC, and DDPG. We now relate the critic losses each other, starting from the most general case, DoubleGum. In continuous control, DoubleGum uses the following noise model, formed from substituting Equation 7 into Equation 5:
Q0 + (s) p(a)(s,a)
r +
1000 "'), + 7B0(s) Clao ll pe] (a(1'))))
E
(16)
Here, (.) L(0, (a(-)) is logistic distribution and (a s) oc exp critic loss is derived from this noise model by approximating the RHS with Equation 8 and learning 0 with moment matching in Section 3.2.
2 (s.a)
The DoubleGum
Bu(s)
The SAC noise model is derived from Equation 16 in three ways. First, SAC approximates (s) [(O, ())))) as nggaa ((0, rr), motivated by the fact that both distributions have the same mean/mode. Secondly, SAC approximates the DoubleGum nete-dependent logistic spread Ba (s) as temperature parameter learned not as part of the critic but by itself with Lagrangian dual gradient descent.
Thirdly, SAC breaks down C(T || Pa] = TT) - DxL. [ || pel before assuming that the KL-
Divergence is negligible, given that it minimized by the actor loss. These three approximations yield the SAC noise model as
n8th (s. 18,a
E p(a
r +
E as
102 (s', Blog nald' |
(17)
MLE of 0 wrt the above noise model yields the MSBE critic loss.
DDPG a special case of SAC that assumes 3 - O, removing the last term in Equation 17. lims-0 (a s) becomes deterministie, To may be modelled by deterministic policy.
B.3 Interpreting the Cross-Entropy Pessimism Factor
In continuous control, Fujimoto al. (2018) introduced Twin Networks, method that improved sample-efficiency with pessimistic bootstrapped targets computed by returning a sample-wise min- imum from an ensemble of two Q-functions. Follow-up work selects a quantile estimate from an ensemble (Kuznetsov et al., 2020; Chen et al., 2021; Ball et al., 2023), which we demonstrate is equivalent to estimating Vo.
Suppose there is an ensemble of n networks where the ith network follows Qe. (s. a) Qo a) 21,, a). Here, Qo is an 'ideal' function approximator never instantiated nor computed and is an arbitrary noise source. When is sufficiently large,
min E (a/s)
(s, = min E (a(a)s)
[Qa(s,)) a) + (((s,a)] = E [Qo(s a)] + min (s)
na(ale)
[Q)(s,)] g(s), where g(s) ~ G(a(s),
*sta(e)
A Gumbel random variable 9 G(a, f) has E[gi + 3, where ýe is the Euler-Mascheron constant, so for a deterministic environment the bootstrapped targets become
T r+
(Qo(s',a)) >a(s') r (a
recovering Equation 8, the DoubleGum continuous control targets, up to an additive term yas('), while (ss)) recovers the spread yoop (s') up to a negative scaling factor, indicating that the default should be negative. Moskovitz et al. (2021) and Ball et al. (2023) showed that the appropriate ensemble size and selected quantile changes the overestimation bias, so appropriate values would ensure a(s') 0.
20

---

B.4 Comparison between DoubleGum and XQL
We present an explanation of Extreme Q-Learnngg (XQL) as presented in Appendix C.1 of Garg et al. (2023). XQL can be derived from Soft Bellman Equation backups given by
(((s',a')
a(s,a)
E p(a' s.a)
(r(s,a,s) + (V (s)], where V-of(s) B log
Σ exp
B
and B is fixed hyperparameter. Computing the log-sum-exp of / is intractable in continuous control, the sum over a' becomes an integral in continuous control tasks.
Garg et al. (2023) present a method of estimating its value using Gumbel regression. Given a (potentially infinite) set of scalars z X, Gumbel regression provides method to estimate the numerical value log-sum-exp- e = B log Σx exp x/3. Gumbel regression assumes x cu a(a, 3), where is a homoscedastic Gumbel distribution, and 3 is a fixed (hyper)parameter. a estimated by MLE tends towards log-sum-cxpa
MLE is performed by numerically maximizing the log- eogliikelhhodd of a Gumbel distribution, which recovers the LINear- LINear-EXponentt EXponential (LINEX) loss function introduced by Varian (1975).
Garg et (2023) incorporate Gumbel regression into deep Q-Learning two ways, which they name X-SAC and X-TD3. X-SAC combines Gumbel regression to estimate the soft value function used SACv0 (Haarnoja al., 201 (a). The soft value function V750t (s) is neural network whose parameters are learned by Gumbel regression from Qv a) (s), where & are target network parameters. A neural network Qo with parameters 0 may then be learned by the MSE between itself and Epte']s.a) [r(s, a, 5') + 7 ( (s)]. X-SAC vastly different from DoubleGum, because our algorithm does not estimate the soft value function with separate neural network.
Gumbel regression is directly used to learn the Q-values X-TD3. First, the bootstrapped targets are thusly rewritten
y (s,
E p(a"(s,a)
7B log Σ exp r +
(2ols)
(o(s),a)
3
a²
r + 1Qu Qo(s,))
E p(s"(s,a)
B log
exp
73
In environments with deterministic environments, which comprise all environments considered by Garg et al. (2023) and our paper, Lemma C. of Garg et al. (2023) provides a method of learning the soft value function with Gumbel regression on yo (s,a) ~ G(Qu(s, a), 78). The Gumbel regression objective used in X-TD3 to learn is vastly different from the moment matching with the logistic distribution DoubleGum uses learn 6.
To motivate their use of Gumbel regression, Garg et al. (2023) derived noise model which they use to present empirical evidence of homoscedastic Gumbel noise. In contrast, presented empirical evidence heteroscedastic logistic noise formed from a noise model with two heteroscedastic Gumbel distributions.
C A Discussion on The Convergence of DoubleGum
To the best of our knowledge, there two types of convergence analysis in Q-Learning: operator- theoretic analysis over tabular Q-functions, and 2) training dynamics of neural network parameters. We believe the second is more appropriate for DoubleGum, because our theory addresses issues in using neural networks (and not tables) for Q-learning. Nevertheless, for completeness, we discuss convergence guarantees for the tabular setting and the function approximation setting. While we can guarantee convergence for the former setting, we have no guarantees for the second.
C.1 Tabular Q-Functions
Appendices B.1 and B.2 present DoubleGum MaxEnt RI. algorithm. When Q-functions are tabular, Appendix A of Haarnoja al. (2018a) shows that MaxEnt RL algorithms may be derived
21

---

Algorithm 2: DoubleGum Soft Policy Iteration
Input: Finite MDP (S,A, r.), initial tables Q, 3. you initial policy x Output: Optimal Tabular Q-function Q"
for training iteration i do for all do for all a do
3
(s,) +
E
[r(s, +773,()))++33((())) E.. exp (al Bi(s")
Malla,a)
5
Qi+1(8,a) - y (8,0)
(s) t v3
V annt(als)
bysott (s,
7
define T(a (8)
explQue (S.4//aa
from soft policy iteration. We therefore present convergence proof for DoubleGum with tabular Q-functions based on soft policy iteration.
DoubleGum treats the return coming from a logistic distribution and learns its location and spread. In the tabular setting, two tables would need to be learned, Q(s, a) and ((s). An algorithm to learn these tables in finite MDP with soft policy iteration is presented in Algorithm 2. Policy evaluation done by Lines 4-6 while Line 7 performs policy improvement.
Proof of convergence of Algorithm 2 is similar to the SAC proof of convergence in Appendix B of Haarnoja al. (2018a). This should not be surprising, given that Appendix B.2 shows SAC as special case of DoubleGum. We first show that policy evaluation converges and that new policy found by policy improvement does not reduce the magnitude of the value function.
Lemma 5 (Soft Policy Evaluation). Consider the Soft Policy Evaluation operator given by
Qi+1(s,a)+ E p()))s,))
r(s,a,s') 7B((s') log Σexp exp
(
over all (s, a) pairs.
4im Qi converges to the soft Q-value.
Proof. Following Appendix A.4
((s,a)) B(s)
2 E (Q0(s, a)] B(s) C[ r|
((s) log Σ exp
=(als)
where p(a () E. exp(((s.cc)/883))
the bootstrapped targets may be thusly rewritten
(Q(s,d))
r(s, a s') + 73())) log Σ exp
p(*(ssa))
B(s')
E p(x'(s,a)
r(s, 8') + x()'ls))
(Q(s d')] + B(s') C| 11
E Ms)(s,))
(8, 8') + [(a'ss))
1Q(s''00 a')]
where as, s') r(s, as) + 3(s') C[T p|.
Following Lemma 1 in Haarnoja et al. (2018a), Sutton and Barto (1998) gives convergence of (8,1(8, < (p))) [r'(s, 8') 15 a!]]]
The proof Soft Policy Improvement should be identical to SAC, given that Appendix B. shows that DoubleGum and SAC use identical actor losses. As such, Lemma 5 can be used in place of Lemma Theorem 1 of Haarnoja et al. (2018a), thus showing convergence of DoubleGum the tabular setting.
22

---

C.2 Deep Q-Functions
Parameters the deep Q-function used by DoubleGum in Algorithm are learned by a loss function equivalent to that of heteroscedastic normal regression. Convergence of DoubleGum in the function approximation setting would therefore rely on convergence of heteroscodastic normal regression.
Zhang et al. (2023) introduces PAC-bounds for heteroscedastic normal regression, but on the condition that the mean-estimate is close to the ground truth mean, as mentioned in Paragraph 1 of Section This is empirically achieved by Seitzer et al. (2022), who analyze heteroscedastic normal regression and find that the mean-estimate frequently converges to an underfitting solution. This because the Negative Log- Log-Likelihood (NLL) of normal distribution is minimized when the variance becomes large Equation 6, this term denoted with oz. As such, changes in Qi will not change the loss function much. To rectify this, Seitzer et al. (2022) multiplies the NLL of the normal with the numerical value of the standard deviation, reducing the dominance ag on the loss function.
D Further Empirical Evidence for Theoretical Assumptions
D.1 Noise Distributions in Deep Q-L.erniing
Figure 6 presents graphs corresponding to Figure Ic for all environments considered in this paper.
Continuous control results were generated from DoubleGum with default pessimism (c (c= -0.1).
D.2 Adjusting The Pessimism Factor
Figure 7 presents graphs corresponding to Figure 2 for all continuous control environments considered in this paper.
E Further Experimental Details
E.1 Noise Distribution Discrepancy with Extreme Q-Learning
In Appendix D.2 of Page 19, Garg et al. (2023) fitted a Gumbel distribution to the TD errors on three continuous control environments. The Gumbel distribution was a good fit in two of the three environments they benchmarked on. We could not reproduce this result and attribute the discrepancy to experimental differences.
Garg et al. (2023) logged their batch of 256 TD errors once every 5,000 steps during training for 100,000 timesteps, producing = 4000 samples which were aggregated. They also computed boot- strapped targets with online parameters. In contrast, we sample 10,000 TD errors with bootstrapped targets computed from target parameters at single timestep instance, and we do not aggregate samples across timesteps.
E.2 Continuous Control Benchmarks and Evaluation
As mentioned in Section 6.2, the evaluation metric in continuous control was the normalized IQM with 95% stratified bootstrap confidence intervals from Agarwal et al. (2021). Returns were normalized by a minimum value computed from the mean of 100 rollouts sampled from a uniform policy and the maximum possible return from the environment. When the maximum value was not specified, we used the maximum value of any single rollout attained by any of the basclines.
We benchmarked on four continuous control suites: DeepMind Control (Tassa et al., 2018; Tun- yasuvunakool et al., 2020), MuJoCo (Todorov al., 2012; Brockman al., 2016), MelaWorld (Yu et al., 2020), and Box2D (Brockman et al., 2016). These environments were selected to be as extensive as possible. DeepMind Control and MetaWorld were chosen because of their diversity of tasks, while the MuJoCo and Box2D environments are popular benchmarks within the common interface of OpenAl Gym (Brockman et al., 2016), now Gymnasium (Farama Foundation, 2023). No citation exists for Gymnasium as of writing this paper, and we link to wttp//gttttttttge//g/// their GitHub repository https::///// //gi thub. cowm//arraaaFoundaa...... www.aaooonnnnnnnnnniiiiiiiiiiiai imm as suggested in https:/ // /gi thub. com/Farama CaatamaaFFoouut Foundati in htt.aaii///yaaammmmiissssssss umt.ssues/11111
23

---

CariPole-vi
Acnhhttvv
MountainCar-v0
2,5
2.4
2.0
NLL
1.6
1.5
15
0.8
1.0
0.0
-1.5
-0.8
0.0
0.5
0.0
100
0.5
1.0
Timesteps (in millions)
(a) Discrete Control
acrohotssiiguup
reacher-hard
finger-tum tum hand
hopper-hop
fish-swim
chectah run
7.7
2:2
27
277
2.4
2.4
2.0
2.8
2.1
1.85
1.8
18
1.6
1.5
1.5
3.4
NL.
00
0.5
0.5
1.0
1.0
0.0
0.5
1.0
0.0
0.5
15
quadruped-ron
swimmnee swimmerls
humanoid-rrn
dog-ran
1.80
1.80
1.76
1.65
1.05
1.50
1.5
1.35
1.36
1.44
1.4
0.0
0.5
1.0
0.0
0.5
10
0.0
1.0
0.0
6.5
1.0
1.0
Timesteps (in millions)
(b) DeepMind Control
Hopper-vi
HallChectab-ed
Walker2d-v4
Humanoid-v
2.8
1.68
2.
2.2
NLL
2.0
188
1.6
1.6
1.4
0.
Timesteps (in millions)
(c) MuJoCo
button- burton-press-v2
door-open-y v/
drawer drawer-close-22
drawer-apen-v2
2
insert side-v2
pick pick place-v2
1.2
1.8
1.4
1.6
2.4
2.5
1.5
2.0
1.5
2.3
15-
1.50
2.0
1755
1:24
2.1
2.4
1.00
1.2
pushav2
reach-v2
window--pen--v
windon-closo--22
Paskethall--2
dial- -ur---2
24
2.50
2.25
NLJ
2.0
1.8
2.25
2.00
2.00-
1.75
1.6
1.5
1:35
1.4
1.50
1.25
0.5
swocp-intoo-22
hammer-v2
asscmbly-22
2.2
2.0
2.0
1.8
1.6
1.74
1.50
1.25
1.2
0.5
1.3
Timesteps (in millions)
(d) Meta World
3
Bipedal Waleea/lWlleeettddoeeers
3.5
3.2
2.5
Homo-Normal
NLL
Hetero-Logistic (expected)
2.0
2.0
Moment-Min Matched Hetero-Normal
1.6
0.0
0.5
1.0
0.0
0.5
Timesteps millions)
(e) Box2D
Figure 6: Negative Log-Likelihoods (NLLs) of the noise in Deep Q-Learning under different distributions throughout training (lower better). Mean calculated per-task + standard deviation. The legend for all graphs is in Figure 6e.
24

---

acrobot-swingun
readber-aard
Bigerr-turn hard
hoppe-hhp
fish-swim
chectal--un
1500
75
100
1200
75
40
30
45-
60
60
90-
19
Magnitude
walker-run
quadruped-run
swimmer--see immeril
humanoid-run
dog-run
100
100
50
240
100
40-
45
120
70-
20
30
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Timesteps (in millions)
(a) DeepMind Control
Hopper-y4
HallCheetahb-4
Ant-v4
Humanoid-v
Magnitude
1500
1200
5300
4000
1600
750
2400
900
900
1600
800
250
800
1.0
0.5
0.0
1.0
0..
1.0
0.5
Timesteps millions)
(b) MuJoCo
button-press-22
dour-open-v2
drower-cloee-22
drawer-open-v2
peg insert-siie--22
pick-place-22
1600
1000
1300
1000
1200
900
300
500
750
750
400
600
500
250
250
240
250
pusb-v2
reach-v2
window-open-t
window-close-22
basketball -x2
dial -2
Magnitude
1.200
0423
1000
800
2000
1250
1000
1250
1000
300
600
200
600
400
800
400
500
750
250
500
750
1.0
0.0
sweep-into-v2
hammer-v2
assembly-v2
1200
1000
000
750
600
500
300
250
0.0
0.5
1.0
0.0
0.5
1.0
0.0
1.0
Timesteps millions)
(c) MetaWorld World
BipeduJWalkee-v3
-- 0.5
Magnitude
300
0.1 (Defanlt)
150
100
< 0.0
0.1
300
< 0.5
0.5
0.0
0.5
1.0
Timesteps millions)
(d) Box2D
Figure 7: The effect of changing pessimism factor on the target Q-value in continuous control. IQM calculated pas- standard deviation. The legend for graphs is Figure 7d.
25

---

DeepMind Control (DMC) was designed to benchmark continuous control, over a broad range of agent morphologies. We selected agent morphologies that could be trained from states with broad range of action spaces from 1 (acrobot) to 38 (dog). We did not benchmark on humanoid, CMU as this environment was not intended to solved with RI. from scratch, unlike the other baselines. The hardest task was selected from each of the agent morphologies. Properties of the 11 DMC tasks are presented in Table la.
Meta World was designed to have a diverse range of tasks to evaluate the generalization ability of learned policies. Each environment within MetaWorld is therefore made up of multiple tasks, all with the same underlying structure of an MDP but with different numerical values of their parameters. We follow the method of Seyde et al. (2022) to benchmark on a single Meta World task by first selecting an environment and then randomly selecting a set of numerical parameters. Each new instantiation of a MetaWorld task would result in a different set of hyperparameters. As such, we expect the error bars in the aggregate statistics of Meta World to be substantially larger than the other environments. We benchmark on tasks formed from the union of the MI.1, MT10, and MI.10 train tasks that a policy in MetaWorld would be trained on, well as the five environments benchmarked in Seyde et al. (2022). Properties of the 15 MetaWorld tasks are presented Table Ic.
MuJoCo was evaluated on the same tasks as SAC (Haarnoja et al., 2018b). These tasks were all iooonotion-bsedd Properties of the 5 MuJoCo tasks are presented in Table 1b.
Box2D was evaluated on all continuous control tasks from states. Properties of the 2 Box2D tasks are presented in ld.
E.3
Discrete Control Baselines
Discrete control algorithms were implemented as deseribed in Section 3.4. Hyperparameters used in discrete control algorithms are detailed in Tables 2 and 3. We provide explanations for these design choices as follows.
The original DQN algorithm in Mnih et al. (2015) was designed for pixel inputs. We modified DQN to use state inputs by using an architecture described in Section 3.4 we used in continuous control that was popular for use with state inputs. Conversely to the continuous control architecture, we found removing GroupNorm (Wu and He, 2018) was crucial to getting DQN to work. Similarly to the continuous control architecture, we found that changing the initialization and target network updating drastically improved performance. We also used the MSE and Adam (Kingma and Ba, 2014) optimizers as Ceron and Castro (2021) showed that this yielded improved performance over the Huber Loss (Huber, 1992) and RMSProp (Hinton al., 2012) of the original DQN. Our implementation of DQN solves classic discrete control tasks that the CleanRI. (Huang et al., 2022b) reproduction of the original DQN paper at battps://docts
DQN:
.leanrl.dev//e.. aaalevrrllllle allorralgrrttti thms thms/dqn/
#experiment-ress results.- could not solve.
Dueling Double DQN (Dueling DDQN) was a baseline modified from Hessel et al. (2018) designed to be as compatible with DoubleGum as possible. Rainbow evaluated six innovations to DQN: Double DQN (Van Hasselt al., 2016), Dueling DQN (Wang et al., 2016), noisy networks (Fortunato et al., 2017), n-step returns, C51 distributional RL (Bellemare et al., 2017), and prioritized replay (Schaul al., 2016). We only used the first two of these six innovations in DoubleGum. We did not find n-step returns effective in discrete domains we considered, nor prioritized replay. Distributional RL was incompatible with DoubleGum, while Schwarzer al. (2023) did not find noisy networks advantageous.
DoubleDQN was implemented following Van Hasselt et al. (2016) by computing bootstrapped targets of Quew (s, max. QNEe (s, Q)). Dueling DQN was implemented following Wang et al. (2016), with the advantage and value heads having two layers with a hidden layer of size 256 and ReLU activations. The stability of Dueling DQN was greatly improved by setting the biases of both dueling heads O.
DoubleGum was implemented as Dueling DDQN with an additional variance head described in Section 3.4.
26

---

Table 1: Properties of Continuous Control Environments
(a) DeepMind Control
Environment
Task
Action Dimension
Maximum Return
Minimum Return
swingup reacher hard hard hopper hop
acrobot
1000
3,252
2
1000
8.547
finger-turn
1000
67.78
4
1000
0.07236
fish swim
1000
70.99
cheetah
6
1000
3.647
walker run
1000
22.96
quadruped
12
1000
108.2
swimmer
swimmer15
14
1000
157
humanoid run
21
1000
0.877
dog run
38
1000
4.883
(b) MuloCo
Task
Action Dimension
Maximum Return
Minimum Return
Hopper-v4
3572
18.52
HalfCheetah-v4
6
11960
-283.4
Walker2d-v4
5737
2.753
Ant-v4
8
6683
-60.06
Humanoid-v4
6829
122.5
MetaWorld
Task
Action Dimension
Maximum Return
Minimum Return
button-press-v2
4
10000
187.5
door-open-v2
4
10000
277.1
draver-close-v2
4
10000
842.5
drawer-open-v2
4
10000
631.8
peg-insert-side-v22
4
10000
8,083
pick-place-v2
4
10000
5,449
push-v2
4
10000
30.62
reach-v2
4
10000
776.1
window-open-v2
4
10000
230.3
windor-close-v2
4
10000
306.7
basketball-v2
4
10000
10.2
dial-turn-v2
4
10000
125.6
sweep-into-v2
4
10000
63.41
hammer-v2
4
10000
395.1
assembly-vt
4
10000
226.3
(d) Box2D
Task
Action Dimension
Maximum Return
Minimum Return
BipedalWalker-v3
4
300
-99.97
BipddalWallerrraaddore--33
4
300
-107.9
27

---

Table 2: Shared Hyperparameters of Benchmarked Algorithms
Hyperparameter
Value
Evaluation Episodes
10
Evaluation Frequency
Maximum Timesteps 100
Discount Factor y
0.99
n-Step Returns
step
Replay Ratio
Replay Buffer Size
1,000,000
Maximum Timesteps
1, 1,000,000
Table 3: Hyperparameters for Discrete Control
Hyperparameter
Value
Starting Timesteps
2,000
Maximum Timesteps
100,000
Exploration
Policy Churn Adam
Optimizer
Learning rate
3e-4
Number of groups in network GroupNorm
Network structure
Linear (256), ReLU, Linear (256), ReLU
Table 4: Hyperparameters for Continuous Control
Hyperparameter
Value
Starting Timesteps
10,000
Maximum Timesteps
1. 1,000,000
Exploration Noise
0.2
Policy Noise in Critic Loss
0.1
Policy Noise in Actor Loss
0.1
Actor optimizer
Adam
Actor learning rate
3e-4
Critic optimizer
Adam
Critic learning rate
3e-4
Number of groups in Actor GroupNorm
16
Number groups in Critic GroupNorm
16
Critic target networks EMA 77
5e-3
Actor target networks EMA
Linear (256), GroupNorm, ReLU Linear (256), GroupNorm, ReLU
Critic structure
Linear (256), GroupNorm, ReLU
Actor structure
Linear (256), GroupNorm, ReLU
Table 5: Pessimism Hyperparameters Continuous Control
Pessimism Hyperparameter
Algorithm
Default
DeepMind Control
MuJoCo
MetaWorld
Box2D
DoubleGum (ours)
-0.1 Twin Twin Twin (8 5)
0.1
-0.5
0.1
0..1
DDPG/TD3
Single Single Single (3)
Twin
Single
Twin
SAC
Twin Single (5)
Single
Twin
XQL
Twin (2)
Twin (5)
QR-DDPG
Single
Single
Twin
Single
Twin
FinerTID3
3
28

---

E.4 Continuous Control Baselines
Continuous control algorithms were implemented as described in Section 3.4. Hyperparameters used in continuous control algorithms are detailed in Tables 2 and 4. Pessimism hyperparameters are presented in Table and were found following results in Appendix F.2.
As mentioned, all implementations used networks with two hidden layers of width 256, with orthogo- nal initialization (Saxe et al., 2013) and GroupNorm (Wu and He, 2018). Following Kostrikov (2021), target network parameters were updated with an EMA of 5e - 3 in the critic and 0 in the actor. All these design choices differ from their original implementations but improved aggregate performance. We provide explanations for these design choices as follows.
DDPG introduced in Lillicrap et al. (2015) and Fujimoto al. (2018) updated the design choices of DDPG to empirically improve its performance. In addition to the existing changes, our implementation uses the noise clipping scheme in the actor specified by Laskin et al. (2021).
TD3 was implemented with three changes from Fujimoto et al. (2018). First, we update the actor once per critic update- ie using delay of 1. This is such that the only hyperparameter change between our DDPG and TD3 is the use of Twin Networks. Secondly, we update the actor to maximize the mean of two critics rather than a single critic, design choice we found empirically reduced variance between training runs. Thirdly, we do not compute the EMA of actor-network parameters. Removing this EMA improves sample efficiency but at the cost of higher variance.
FDn (our introduced baseline) was implemented with the same hyperparameters as TD3 but with an ensemble of 5 critic networks. We chose to use 5 networks because we tuned the pessimism factor hyperparameter of DoubleGum over 5 values. The 5 critics in FinerTD3 enable five values of pessimism to be used. Pessimism of FinerTDDs is adjusted in the bootstrapped targets. The 5 critic values are sorted by decreasing positivity, and the 2th smallest value is used as the target critic value in the bootstrapped targets.
SAC was implemented with hyperparameters from Kostrikov (2021), which we found improved performance. Kostrikov (2021) differs from Haarnoja et al. (2018b) in two additional ways from those mentioned. The standard deviation in the actor was clipped to [-10, 2), and the target entropy was the action dimension divided by 2 instead of just the action dimension.
XQL Garg et al. (2023) presents off-policy algorithms: X-TD3 and X-SAC. We use X-TD3 be consistent with the DDPG fixed-variance actor of DoubleGum and refer to it throughout as XQL.. XQL. tunes two hyperparameters per task: the of twin networks/not and scalar hyperparameter We swept over the same 3-values as Garg et al. (2023): 1, 2, 5 without Twin Critics and 3, 4, 10 and 20 with Twin Critics, was tuned in the same way as pessimism we found a default B value and 33 tuned per-suite, B values are presented in Table 5 and were found following results in Appendix F.2
MoG-DDPG is formed by combining a Mixture-of Gaussians (MoG) critic with DDPG. The MoG critic was introduced in Appendix A of Barth-Maron et al. (2018) and improved by Shahriari et al. (2022). The latter paper combines the MoG critic with DDPG with distributed training, but we remove the distributed training component because we do not use in DoubleGum.
-aauuss-f-GGaussinns
QR-DDPG (our introduced baseline) combines the quantile regression method of Dabney et al. (2018b) with DDPG actor. Although Ma et al. (2020); Wurman et al. (2022) and Teng et al. (2022) have combined quantile regression with SAC, we combine with DDPG because DoubleGum is built on top of DDPG. Like Dabney et al. (2018b), we use 201 quantiles, but these are initialized with orthogonal initialization and are optimized with the MSE, rather than the Huber loss. QR was developed for discrete control and uses the Huber loss with the RMSProp optimizer popular in discrete control methods. We found better performance with the MSE and Adam optimizer, perhaps confirming the result of Ceron and Castro (2021) in distributional RL for continuous control.
DoubleGum was implemented as DDPG with a variance head described in Section 3.4.
F.5 Compute Requirements
A single training run for discrete control may take up to 3 to 5 minutes on laptop with an Intel Core 19 CPU, NVIDIA 1050 GPU and 31.0 GiB of RAM. On the same system, a single training run for continuous control takes 1- hours.
29

---

Table 6: Discrete Control Numerical Results
Score at 100K timesteps (IQM over 12 seeds)
Task
DoubleGum (ours)
DQN
DuelingDDQN
CartPole-vi
500 113.4
475 4: 105.5
+:
496.9 + 89.1
Acrobot-vi
-62.78 + 1.775
-73.52 5.191
-64.12 + 17.15
MountainCar-vo
-98.17 + 2.45
-99.37 + 5.914
-98.75 30.73
The overwhelming majority of our experiments were run on private infrastructure. This cluster had a mixture of Intel Broadwell, Skylake, Cascade Lake, AMD Rome, AMD Milan CPUs, and NVIDIA P100s, V100s, and A100s GPUs. Benchmarking continuous control took roughly ten times longer than benchmarking discrete control. Multi-threaded experiments for continuous control running twelve seeds in parallel took 5 8 hours. 8 algorithms (DoubleGum, DDPG, TD3, MoG-Crities, SAC, XQL., QR-DDPG, FinerTDD3 were benchmarked over 33 continuous control environments, and there were further runs for hyperparameter sweeps (4 for DoubleGum, 1 for SAC, 6 for XQL, 1 for QR-DDPG and 4 for FinerTD3), yielding 24 runs total. These algorithms were run least 10 times for development and hyperparameter tuning. This yields a lower bound of 8 x 33 x 24 x 10 63360 hours (7.23 years) of computation.
Assuming that all experiments were run on Tesla V100-SXM2-168 (TDP of 250W), the cluster was had a carbon efficiency of 0.0006 kgCOzeg/k (that of the surrounding power grid) and that there were 63360 hours of cumulative computation, the total emissions were 9.51 kgCOzeq. equivalent driving 36km in average Estimations conducted using the Machinel.earning Impact calculator presented in Lacoste et al. (2019).
F
Further Results
F.1 Adjusting the Pessimism of DoubleGum
Figure shows that sample efficiency is sensitive to the pessimism factor e adjusting pessimism per suite greatly impacts sample efficiency. The best performing was = -0.1, 1, and was thus set as the default pessimism factor value.
Figure shows that the performance of DoubleGum may be improved when the degree of pessimism changed per suite. This graph used to determine what pessimism factor use in each suite, whose values are reported in Table 5.
F.2 Adjusting the Pessimism of Baseline Algorithms
This section presents graphs used determine which pessimism values to use for baseline algorithms. All final values are reported in Table 5.
Figure 10 shows that sample efficiency sensitive to the use of pessimism determined by the use of Twin Networks/not. In aggregate, each method was improved by using Twin Networks. Twin networks were therefore set as the default pessimism option for all baseline algorithms apart from QR-DDPG, because Twin Networks was not used with quantile regression in (Dabney et al., 2018b). Figure 11 was used determine whether use Twin Networks/not on a per suite basis.
Similarly, Figures 14 and were respectively used to determine pessimism hyperparameters for FinerTD3. In these graphs, numbers refer to the ,th smallest value returned by the ensemble of target critics. Finally, Figures 12 and 13 were respectively used to determine pessimism and B hyperparameters for XQI..
F.3 Discrete Control
Table 6 presents results for discrete control at 100K timesteps.
30

---

=
c= -0.1 (Default)
IQM
c 0.1
0.5
Timestops (in millions)
Figure 8: Adjusting the pessimism factor in DoubleGum, IQM normalized score over 33 tasks in suites with 95% stratified bootstrap Cls.
DMC
MuJoCo
MetaWorld
Box2D
0.60
088
0.34
IQM
0.45
0.6
0.18
0.12
0.4
0.2
0.2
0.00
010
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0,4
066
0.5
1.0
0.0
0.2
0.4
0.6
0.5
1.0
0.2
04
0.5
0.8
100
Timesteps (in millions)
0.5
c= -0.1 (Default)
c=
0.1
c= 0.5
Figure 9: Adjusting pessimism in DoubleGum, per-suite IQM with 95% stratified bootstrap Cls.
DDPG
1QM
SAC
TWin-SAC
XQL
Twin-XQL
0.08
QR QR-DDPG
Twin-QR-DDB
0.00
02
04
10
Timesteps millions)
Figure 10: Adjusting pessimism of baseline algorithms with the use of Twin Networks/not, IQM normalized score over 33 tasks in 4 suites with 95% stratified bootstrap Cls. Methods that default to use Twin Networks are dashed.
DMC
MuJoCo
MetaWorld
Box2D
0.50
1.0
0.74
1QM
0.45
0.6
0.18
0.12
0.35
0.2
0.06
0.2
0.04
0.00
0.0
0.2
0.4
0.5
1.0
0.2
0.4
0.5
1.0
0.2
0.4
03
1.0
0.0
0.2
0.4
0.5
03
1.0
Timesteps (in millions)
DDPG
SAC
XQL
QR-DDPG Twin-QR-DDR
TD3
TWin-SAC
Twin-
Figure 11: Adjusting pessimism of baseline algorithms with the use of Twin Networks/not, per-suite IQM normalized score with 95% stratified bootstrap Cls. Methods that default to use Twin Networks are dashed.
Untumed
0.30-
8
0.24
IQM
=
B ==
B 8=5
0.6
20
0.00
02
Timesteps (in millions)
Figure 12: Adjusting pessimism of XQL, IQM normalized score over 33 tasks in 4 suites with 95% stratified bootstrap CIs. Methods that use Twin Networks are dashed.
31

---

DMC
MuJoCo
MetaWorld
Box2D
0.60
0.25
1.0
0.15
1QM
0.12
0.15
0.30
0.08
0.10
0.4
0.04
0.68
0.2
0.00
0.00
0.00
00
0.3
0.4
0.6
0.5
1.0
000
6.2
044
0.6
0.5
1.0
0.0
0.2
0.4
0.6
088
10
0.0
0.2
0.4
0.5
0.8
100
Timestops (in millions)
β
B
B
B 20
Figure 13: Adjusting pessimism of XQL, per-suite IQM normalized score with 95% stratified bootstrap Cls. Methods that use Twin Networks are dashed.
0.40
Pessimism 0
0.32
Pessimism (Default)
1QM
Pessimism 2
0.16
Pessimism 3
Pessimism 4
0.00
0.0
0.2
0.4
0.6
0.8
10
Timesteps (in millions)
Figure 14: Adjusting pessimism of FinerTD3, IQM normalized score over 33 tasks in 4 suites with 95% stratified bootstrap Cls.
DMC
MuJaCo
MetaWorld
Box2ID
0.00
0.20
IQM
0.45
0.6-
0.15
0.5
0.10
0.05
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
022
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Timesteps (in millions)
Pessimism
Pessimism 2
Pessimism
Pessimism 1 (Default)
Pessimism 3
Figure 15: Adjusting pessimism of FinerTD3, per-suite IQM normalized score with 95% stratified bootstrap Cls.
DMC
MuloCo
MetaWorld
Box2D
0.50
IQM
0.45
0.6
0.5
0.4
0.00
0.0
0.2
0.4
0.8
100
0.2
0.4
0.8
0.2
0.4
0.8
0.2
0.4
0.6
0.8
Timesteps (in millions)
DoubleGum, (Ours)
DDPG
Twin-SAC
QR-DDPG FinerTDa
MoG-DDPG
TD3
Twin-XQL
Figure 16: Continuous control with default parameters, per-suite IQM normalized score with 95% stratified bootstrap Cls. Methods that default use Twin Networks are dashed.
32

---

DMC
MuJoCo
MetaWorld
Box2D
0.60
0.5
0.30
1.4
0.24
0.5
1QM
0.13-
044
0106-
0.2
0.01
0.0
Timesteps (in millions)
DoubleGum, best (Ours)
SAC' (best w/wo Twin)
QR-DDPG (best w/wo Twin)
MoG-DDPG (untuned)
XQL (best of B w/wo Twin)
FinerTD3 (best pessimism)
best DDPG/TD3
Figure 17: Continuous control with the best pessimism hyperparameters tuned per suite, per-suite IQM normalized score with 95% stratified bootstrap Cls.
F.4 Continuous Control with Default Pessimism
Figures 16 and 18 respectively present aggregate per-suite and per-task results of DoubleGum benchmarked against baseline algorithms with default pessimism values.
Table 7 presents the performance of all algorithms with default pessimism at million timesteps. This graph has six subsections. The first four subsections present per-task results from DeepMind Control, MuJoCo, MetaWorld, and Box2D, respectively, corresponding to results from 18. The next subsection presents per-suite aggregate results, corresponding to Figure 16, while the last subsection presents aggregate results over all tasks and suites, corresponding to Figure 1. In aggregate results, only the IQM is reported.
E,5
Continuous Control with Pessimism adjusted Per-Suite
Figures 17 and 19 respectively present aggregate per-suite and per-task results of DoubleGum benchmarked against baseline algorithms with the best pessimism values adjusted per-suite.
Table presents the performance all algorithms with default pessimism million timesteps. This graph has six subsections. The first four subsections present per-task results from DeepMind Control, MuJoCo, MeraW and Box2D, respectively, corresponding to results from 19. The next subsection presents per-suite aggregate results, corresponding to Figure 17, while the last subsection presents aggregate results over all tasks and suites, corresponding to Figure In aggregate results, only the IQM is reported.
33

---

8.114 15.96
972.4 23.15
974.1 24,9
116 51,71
675 + 118.8
741.7 43.59
731.4 + 109.2
739.8 + 117.1
612.5 + 82.18
1.452 47.32
234.4 29.06
2183 890.6
10420 + 328.8
4276 1 357.1
6048 + 464.4
5668 +: 339.5
835 +: 1190
1059
4808 +: 1324
2048 + 1187
2146 + 1752
6.942 732.7
729.7 : 1571
2735 + 1113
2786 + 1488
4291 + 556
524,4 702.6
1258 1034
1496 1 2042
2866 + 1886
1528 + 921.1
322.1 1: 6.48
53.3 4: 40.58
Finer
0.5352
0.8305
0. 0.1867
0.7679
0.39
4083
DDP
88.26
14.86
924.3 - 73.62
304.3 #9 69.23
695.2 + 54.08
785.9 63.43
743.9 +: 29.61
772.4 1 80.69
499.8 = 64.12
117.8 17.68
155.5 22.55
1641 + 554.7
9148 1476
2205 1 890.8
2245 + 761.4
1814 : 465.6
1424 + 898.3
3818 1258
4178 + 1788
2820 + 1463
339.8 + 1211
451.2 847.2
74.32 847.7
3068 + 1231
2294 + 1.51
4022 + 1191
1197
2157 1465
2203 + 2026
3620 + 1311
2376 1032
247.4 4: 72.5
-84.9 4: 17.95
0.5787
0.4079
0.2017
0.3764
0.3255
356.7
972.9
1972
38.84 91.41
975. 1 4.506
967 1 59.48
135.9 - 65.21
632.7 1 99.64
761.3 H 56.24
705.5 4 114.6
743.9 1 100.6
433.5 1 148.5
119 62.17
229.5 + 20.56
94.89 + 122.7
9694 + 1571
152.7 1 79.47
3652 + 1685
160 2 4 32.99
1128 886.1
4750 + 1560
2256 1 1819
317.8 + 579.6
16.98 + 378
415.2 + 971.5
2552 + 1066
3916 + 1023
4574 + 866.3
1126 1670
1446 1 1020
3650 1 1591
2812 2 1869
439.6 6 + 1429
331.9 4 6.717
114.1 65.9
Score at IM timesteps (IQM standard deviation where appropriate)
Twin-XX XQL
0.5003
0.1423
0.1881
0.8659
0,2802
Table 7: Continuous control with default pessimism hypeparamnetess score after IM timesteps.
3956
0.007008 + 0.03587
Twin-SAC
6.568 1: 661664
976. 27.21
951.5 + 63.21
344.9 9 + 251.5
708.5 +: 44.35
538.6 311.1
677 + 181.7
225.4 44 + 130.5
0.8364 + 0.234
5 5.226 + 0.7772
942 + 135.3
7171 1000
2894 1 1109
5908 + 811.3
5286 + 703.2
1366 + 1133
4232 +. 454.2
3033 2206
2617 + 1642
7.378 2.552
5,553 2,761
25.99 76.78
3390 + 1243
491.3 + 1326
4372 +: 704.6
9.246 197.2
2192 1593
530.2 + 1712
487.5 = 945
947.2 = 983.3
321.1 + 21.67
-32 19 +: 56.1
0.293
0.6304
0.09127
0.6658
0.226
7.315 1 26.53
975.9 + 31.13
969.7 = 28.51
123,4 + 69.37
517.9 + 111.2
745 +: 52.78
696 +: 19.59.53
677.2 + 169.6
477.8 + 139.2
39.51 39.51 1 62.78
209.2 + 25.22
2589 + 939.2
10020 + 1390
3868 1: 593.1
5645 5 914
5241 = 302.4
635.5 4 1 566
3784 + 809.1
4743 1718
2951 + 1511
432 + 1191
7.501 478.7
42.5 930.5
2092 + 1304
3452 + 11.50
4202 + 1145
627.9 648.4
1228 1206
1977 + 1907
1340 + 1099
2085 = 801.6
321.2 4: 2.343
18.27 45.01
TD3
0.4588
0,7716
0.149
0.7204
0.3358
334.7 1 96.92
975.7 + 26.78
909.1 - 50.68
305.7 - 80.12
710.2 + 94.54
804.5 + 64.31
755.6 + 28.29
736 +: 87.58
531.5 + 104.1
119.6 + 25.09
177.4 + 12.17
1348 + 740.2
9552 + 1557
1466 1 470.3
976.9 + 302.6
2023 + 604,9
1093 + 1016
2691 + 1415
3880 +: 1916
2510 + 1467
402.4 + 1855
509.3 + 672.7
777.9 928.9
2992 + 1466
3029 + 1476
4352 +: 342.3
809. 1. 901.4
2385 1486
3007 + 1677
2664 + 1477
2265 + 861.8
218.6 + 115.9
30.12
DDPG
0.5823
0.3077
0.1904
0.319
0.3026
-71.08
MoG-DDPG
364,9 1 71.75
976.9 + 4.985
935.4 :: + 77.4
313.3 = 99.74
758.1 + 62.68
844.9 +: 53.54
778.7 18.77:22 21.86
818.2 17..2776.3 76.37
623,4 + 81.98
96.97 + 12,58
138.8 - 20.58
1211 + 319.2
9398 +: 1044
2196 1 1127
4644 + 966.2
1715 : 783.4
919.6 1: 1066
4288 +: 1431
4706 1: 1578
4074 + 1421
431.4 1 1417
212.5 + 355
310.9 1005
2889 + 909.7
2501 + 1764
4520 214.5
602.4 1106
1552 1: 1250
1417 1 1448
3500 + 1344
1941 # 1241
8.892
152.6 4: 52.39 39
0.6326
0.4663
0.193
0.8911
0.3709
325,4
DoubleGum
330.2 1 79. 44
979.4 #: 54.28
931.8 + 98.91
306.7 7 + 101
675.7 +: 58.5
883,1 + 22.51
783.6 +: 26.97
835 + 65.11
608.9 188992 + 137.8
142.5 12.6
187. - 13.14
1399 668
10710 + 608.3
4148 1 1439
6046 = 582.5
5645 904.1
1436 + 1241
3671 +. 1606
4839 +: 1726
2762 + 1212
1226 + 1769
12.49 + 110.5
191.1 1282
1746 +11111111 1127
2668 + 1515
4404 + 712
780.9 1238
1214 1 952.6
4237 + 1348
4308 + 1215
2606 660.4
315.2 +. 12.26
189 +: 86.95
0.6191
0.818
0.209
0.9391
0.428
BipedalWalkeertardddoo v3
DeepMind Control Aggregate
peg inseettettt side-v/2
finger turn, hard
swimmer swiisiimm wimmer15
MoraWorld Aggregate
acrobot swingup
Half Cheetahetah -4
button ppess-----e-- v22
drawer dramer-closed -close-v2
drawer open-pe--eee v2
window open--eeeeee v2
window www.-c-vze v22
MuJoCo Aggregate
Box2D Aggregate
reacher- hard
quadruped-re -un run
humanoid id-run ruu dog- run
pick-ple plac--ll
sweep into-v2
BipedalWaleer
hopper hhp fish-swim
cheetah- run
walker-ruu -run
Hopper -v4
Walker2d-vé
Humanoid-vy
cecon-presee
door door-open-Ve o--o--pee
basketball
iial-turn-v2
-2 v2
assembly-v2
All Aggregate
Suite/Task
Ant-v4
push-v2
reach-v2
hammer
34

---

serobot-swingup
finger-tum hand
hoppes--oo
1600
400
300
1000
300
750
300-
600
150
500
500
400
250
250
100-
200
200
IQM
0.0
0.5
1.0
000
0.5
0.0
1.0
0.0
0.5
1.0
000
0.5
0.0
0.5
1.0
walker-run
quadiuppl--un
swwimmet-swwmmmat 15
bumanoid--ng
dog-run
750
500
150
250
600
100
200
200
400
100
200
150
0.5
1.0
0.5
01.5
1.0
0.5
Timesteps millions)
(a) DeepMind Control
Hopper-v4
HalCChetta..vd vd
Walker2d.vd
Ant-v4
Humanoid-vd
40030
10000
4500
6000
1QM
4500
4500
1600
5000
3000-
800-
1500
1900
0.0
0.5
1.0
0.0
1.0
0.0
0.5
1.0
0.0
0.0
0.5
Timesteps (in millions)
(b) MuJoCo
button-press-v2
door-open-v2
drawer-clree-22
drawer-open--2
agg-inert-side-v22
pick-place-22
6000
60000
4500
1600
4000
3000
2000
1500
600-
1500
-400
800
push x2
reach-v2
window-open open-v2
window window-close-v/
busketball-v2
dial-tur--22
2400
5000
4000
6000
4500
3000
4500
1QM
2000
3000
1500
1000
1500
1000
1000
16000
010
0.5
1.0
0.0
0.5
1.0
000
0.5
1.0
0.0
0.5
1.0
0.0
0.5
0.0
0.5
swoep-into-v2
hammee-v2
assembly-x2
6000
4000
4500
4500
3000
3000
1500-
1500
1500
1000
Timesteps millions)
(c) MetaWorld World
BilkdalWalker-33
Bipeda/aalkeeHHaddonee
240
DoubleGum, (Ours)
1QM
150
Twin-XQI.
150-
DDPG
QR-DDPG
T133
Finer
Timesteps (in millions)
(d) Box2D
Figure 18: Continuous control with default pessimism hyperparameters, per-task IQM standard deviation. Methods that default to use Twin Networks are dashed. The legend for all graphs is in Figure 18d.
35

---

Finer TD3
283.3 89.79
976.3 4: 27.78
956.4 + 74,46
519.5 - 47.02
761.6 1: 41.15
699 +: 84.17
777 + 12.7
724.4 + 92.74
560.2 1 106.4 66.4
85.08 55.08 + 6,825
87.51 + 32.47
890.6
10420 328.8
4276 357.1
6048 + 464.4
5668 339.5
1086 + 758.2
4268 953.8
3741 +. 2037
1608 + 1333
1092 + 1699
501.4 963.9
256.8 + 453.4
2402 922.9
2964 + 1365
4272 +: 260
1868 959.1
1101 1 1255
2844 + 1934
4151 + 1115
2067 # 1507
6.48
53.3 + 40.58
0.6121
0.8305
0.2042
0.7679
0.4239
2183
322.
Table 8: Continuous control with the best pessimism hyperparaneiers adjusted per suite after 1M timesteps
Score at IM timesteps (IQM standard deviation where appropriate) with best pessinissm
QR-DDPG
225.7 1 131.1
13.21
952.6 H 65.92
346.8 846.8% = 105
708. 8 + 38.73
791.1 + 69.23
759.8 4 24.46
775 54.76
552.7 1 140.2
122.4 1 9.644
145.3 + 28.37
94.89 + 122.7
9694 1 1571
152.7 1 79.47
3652 H 1685
160.2 4 32.99
1506 4: 1047
3512 4: 1387
4588 4. 2042
2972 1 1382
100.8 + 1462
26.68 + 47.73
76.38 + 95.72
2583 + 1345
1672 + 1813
4528 + 232.3
99.83 1 745.6
1690 1 1408
1008 1 1782
1892 + 1387
164 # 401.5
331.9 4. 6,717
114.1 65.9
0.5787
0.8061
0.2017
0.601
0.3924
973
356.7 1: 88.26 26
972.9 + 14.86
924.3 - 73.62
304.3 + 69.23
695.2 3. 54.08
785.9 +: 63
743.9 +: 29.61
772.4 + 80.69
499.8 . + 64.12
117.8 + 17.68
155.5 + 22.55
2398 + 946.4
9855 +: 1323
4027 1 962
6185 + 168.3
5452 : 243.1
1424 + 898.3
1258
4178 1788
2820 + 1463
339.8 + 1211
451.2 + 847.2
74.32 847.7
3068 1231
2294 + 1551
4022 + 1191
1972 1. 1197
2157 1465
2203 + 2026
3620 + 1311
2376 + 1032
18.27
-31.74 1899
0.5918
0.1423
0.1204
0.8659
0.2786
XQL
311.3
3818
0,007651 + 56.18
0,9647 1 0.1496
226.6 134.5
976.2 56.58
972.4 + 25.53
616.4 + 22644
764.5 + 74.55
732 199.4
774 44 62,96
437.8 + 192.5
22.63 56.75
942 135.3
7171 + 1000
2894 1 1109
5908 = 811.3
5286 = 703.2
645.5 + 728.3
3114 + 1166
1339
1710 + 1514
9.693 + 232.9
4.821 1.873
90.81 745
2974 + 1393
1957 + 1832
4000 + 1045
9,932 11.24
1472 1178
247.9 + 1718
1180 + 1650
218.1 + 492.3
321.1 4: 21.67
-32 56.1
0.4762
0.6304
0.08515
0.6658
0.2719
SAC
4749
DDPG/TD3
334.7 96.92
975.7 + 26.78
909.1 - 50.68
305.7 = 80.12
710.2 94.54
804.5 64.31
755.6 +: 28.29
736 + 87.58
531.5 + 104.1
119.6 25.09
177. 1 12.17
939.2
10020 +: 1390
3868 1 593.1
5645 914
5241 + 302.4
1016
1415
3880 + 1916
2510 1 1467
402.4 : 1855
509.3 672.7
777.9 928.9
2992 + 1466
3029 + 1476
4352 + 342.3
901.4
1486
+ 1677
+ 1477
2265 + 861
+: 2.343
18.27 45.01
0.5823
0.7716
0.04
0.7204
0.3979
2589
1093
2691
809.1
2385
3007
2664
321.2
MoG-DDPG
71.75
976.9 1 4,985
935 : 77.4
313.3 - 99.74
758.1 + 62.68
844.9 :: 53.54
778.7 4: 21.86
818.2 + 76.37
623,4 + 81.98
96.97 12.58
138.8 + 20.58
319.2
9398 + 1044
1196 1 1127
4644 + 966.2
1715 783.4
919.6 + 1066
1431
4706 1. 1578
4074 1 1421
431.4 # 1417
212.5 + 355
310.9 1005
2889 + 909.7
2501 + 1764
4520 + 214.5
602.4 1106
1552 1250
1417 1 1448
3500 + 1344
1941 # 1241
325 125.44 4: 8.892
152.6 52.39
0.6326
0.4663
0.193
0.8911
0.3709
364.9
4288
1211
DoubleGum
330 . 1: 79.75 .4
979,4 54.28
931 + 98.91
306.7 + 101
675.7 + 58.5
883.1 +: 22.51
783.6 26.97
835.1 + 65.11
608.9 + 137.8
142.5 + 12.6
187.1 + 13,14
3290 829.1
9874 + 997.4
4871 1 403.7
5681 + 416.8
5565 + 160.7
1436 +: 1241
3671 +: 1606
1726
2762 + 1212
1226 + 1769
12.49 110.5
191.1 + 1282
1746 + 1127
2668 + 1515
4404 + 712
780.9 1238
1214 1. 952.6
4237 + 1348
4308 + 1215
2606 + 660,4
315.2 +: 12.26
189 + 86.95
0.6191
0.8367
0.2505
0.9391
0.4707
4839
BipedaalWalkeertarrdtooor
finger- turn, hard
swimmer- sw wirimmer15
side-v2
MetaWorld Aggregate
acrobot- swingup
HalfCheetahe v4 Walker2d-v.
draver- traver-close ccose-v22
drawer opw--ppen---e open-v2 v/2
peg- insert-e---e www.innneidee....eeeeeee.eeeeeet
windou open-epenn---e v/2
window close-v2
MuloCo Aggregate
Box2D Aggregate
reacher- reacher-hard hard
quadruped- wwed-ruu run
humanoid run deg run
button preps-resse
dial- tuun-v2 v2
sweep- sweep-into- into-v2
BipedalWalker-
DMC Aggregate
hopper hop fish rish-swim
cheetah- run
walker-rue run
Humanoid-- v4
door- door- open-v2
pick-place-te
basketball
All Aggregate
Hopper- v4
hamber-Fy v2
assembly-gg
Suite/Task
push-v2
reach-v2
Ant Ant-v4 v4
36

---

acrobott swingup
rencher-hard
finger-tu Jaand
hopper--hp
fish-swim
cheetab-nun
500
1600
1030
430
500
750
750
600
303
500
400
100
0
IQM
walker-run
quadruppd--un
swimmece
bumanoidd-ng
dog-ron
750
In0
100
600
150
450
100
200
0
50
Timesteps (in millions)
(a) DeepMind Control
Hopper-e4
HalfCheetah.vd
Walker2d-v4
Ant-v4
Humamoid-yd
(000
4000
10000
1QM
3000
7500
4500
7000
5000
3000
3000
10000
2500
1500
1.0
1.0
0.0
0.5
Timesteps (in millions)
(b) MuloCo
button-presss-V
door-open-v2
drawer-eloee-
opaner--pen--22
peg-insert-side-22
2400
pick-place-v2
2400
4500
3000
1000
3000
800
1500
1500
1500
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
push-r2
reach-v2
window- open +22
close-v2
basketball--2
dial-
2400
50000
0000
6000
4000
4500
1600
4500
4500
1QM
300
30000
M000
3000
1500
1000
-000
1000
11041
1000
sweep-into-v2
hammer-v2
assombly-v22
6000
4000
4500
1000
3000
2000
1500
1500
1000
1500
1000
0.0
0.5
1.0
0.0
1.0
Timesteps (in millions)
(c) Meta
Bipedalaaleer-e Bipedal Ripect/Walssetlaroooeess
DoubleCum, best (Ours)
450
MoGiDDDPG (untuned)
best of DDPG/TD3
1QM
150
SAC (best
QR-DDPG
-150
50
XQL
0.0
0.5
1.0
0.0
0.5
1.0
FinerTD3 (best pessimism)
Timesteps (in millions)
(d) Box2D
Figure 19: Continuous control with the best pessimism hyperparameters adjusted per suite, per-task IQM = standard deviation. The legend for all graphs is in Figure 19d.
37


Reviews:
[{"review_id": "MGVOwjYp3A", "reviewer": "Reviewer_dXQf", "summary": "Instead of modeling the TD error with a homoscedastic normal distribution, this paper tries to utilize two heteroscedastic Gumbel distributions for more complex and accurate error modeling. Based on this assumption, the authors presents a modified Q-learning algorithm, DoubleGum for solving discrete and continuous control tasks. In particular, DoubleGum can also achieve the effect of pessimism so as to avoid the overestimation. Empirical results show more stable training and competitive performance across classic discrete and continuous tasks.", "questions": "1. What's the motivation of using an additional Gumbel noise for the new function approximator $Q^{new}$ in Sec.3.1? Because this modeling does not seem to involve the Extreme Value Theorem, why to model the error using the same Gumbel distribution? \n\n2. About the DoubleGum for the continuous control, it's said that, for ease of implementation, the pessimism factor $c$ is setted as 0, if this, it doesn't look much different than the common off-policy methods;\n\n3. Sec.3.2 tries to connect the DoubleGum with the pessimistic value estimation, but the experimental results only contain the performance comparison for different choices of $c$, It is better to compare the magnitude of the value function directly;\n\n4. Motiviated by a similar idea, extreme Q-learning also model the TD-error as Gumbel distribution, so it is better to make a detailed comparison between the both algorithms, including methodological differences and experimental performance.\n\n5. About Fig.1a and Fig.1b, there is little difference for fittings by Logistic distribution or Normal distribution. I am not sure that it is a good evidence to justify the proposed error model.", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "It's a novel perspective to model the TD-error following the Gumbel distribution.", "weaknesses": "1. Double error modeling in section 3.1 seems a little redundant and not intuitive enough；\n2. Though linking the DoubleGum for continuous control to the pessimistic value estimation, the empirical evidence is lacking.\n3. Experimental results don't show outstanding performance improvement across multiple benchmarks, and lack of comparison with similar algorithms, e.g. extreme Q-learning[1]. In addition, It is shown that the Gumbel error model helps reduce the overestimation issue, which is quite natural considering its close connection with SAC. So why not compare its behavior with SAC in more detail?\n\n[1] Garg, Divyansh, et al. \"Extreme Q-Learning: MaxEnt RL without Entropy.\" The Eleventh International Conference on Learning Representations. 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your review!\n\nWe are very happy that you found our modeling of the TD-error novel.\n> Experimental results don't show outstanding performance improvements across multiple benchmarks\n\nWe have uploaded Figures 14 and 15 in our attached 1-page `.pdf`, which show aggregate learning curves across all 33 continuous control tasks. In these graphs, DoubleGum outperforms all benchmark algorithms. More details are given in point 1 of our global rebuttal.\n\n> Lack of comparison with similar algorithms [extreme Q-learning (XQL), SAC].\n\nWe have added the SAC and XQL benchmarks in Updated Figures 3 and 5 as well as Figures 14 and 15 in In our attached 1-page `.pdf`. In brief, DoubleGum outperforms SAC and XQL in aggregate. More details of how SAC was benchmarked are given in point 2 of our global rebuttal.\n\nOur XQL benchmark was based on TD3 as it marginally outperformed SAC (Last line of Appendix D.6). Pessimism of XQL was varied by the use of Twin Networks/not, respectively yielding Twin-XQL and XQL (equivalent to what the authors name X-TD3 and X-TD3-DQ). The value of $\\beta$ in XQL varies per task, but we vary them per suite, to be consistent with our other algorithms. We sweep over $\\beta$s of 3, 4, 10, 20 for Twin-XQL and 1, 2, 5 for XQL, consistent with Appendix D.6 in XQL. Over all continuous control tasks, the best $\\beta$ values were 20 and 5 for Twin-XQL and XQL respectively. These values were used in Updated Figure 3 and Figure 14 (graphs with default hyperparameters). For each suite in DeepMind Control, MuJoCo, MetaWorld, and Box2D, the best $\\beta$ values were respectively 3, 20, 20, and 4 for Twin-XQL and 1, 5, 5, and 5 for XQL. These values were used in Updated Figure 4 and Figure 15 (graphs where pessimism is tuned per suite).\n\nDoubleGum outperforms XQL with both default hyperparameters (Figure 14) and tuned pessimism per suite (Figure 15). Over each suite, DoubleGum outperforms XQL in all suites apart from Box2D, both with default hyperparameters (Updated Figure 3) and tuned pessimism per suite (Updated Figure 5).\n\n> What's the motivation of using an additional Gumbel noise for the new function approximator $Q^\\text{new}$?\n\nWe introduced $Q_\\theta^\\text{new}$ to write a Bellman Equation with one heteroscedastic Logistic noise source (Equation 19), instead of a Bellman Equation written with $Q_\\theta$ with two heteroscedastic Gumbel noise sources (Equation 15). The additional Gumbel noise source shifts $Q_\\theta$ into $Q_\\theta^\\text{new}$ using the property of the log-sum-exp (Appendix B.2). We realize that a simpler way of presenting this is to absorb the Gumbel noise source, simplifying Section 3.1 by removing Equations 16-18.\n\n> it's said that, for ease of implementation, the pessimism factor is setted as 0, if this, it doesn't look much different than the common off-policy methods\n\nWe have now changed the default pessimism factor to $c = -0.1$ as this gives us empirically better per-suite performance (Updated Figure 3) and aggregate performance with fixed hyperparameters (Figure 14). We would like to point out that functionally, our algorithm uses variance networks unlike the common off-policy methods, which is not common in Q-Learning.\n\n> Empirical evidence is lacking [for pessimistic value estimation ...] compare the magnitude of the value function directly)\n\nThis is a great suggestion, and we have added Figure 17 in our 1-page `.pdf'`. This graph shows how the magnitude of the target value function averaged over 256 training samples varies during training. The line and error bars show the mean $\\pm$ standard deviation of the averaged target value function. Four graphs are presented, showing results of one task from each suite. In each graph throughout training, the magnitude of the target value function is smaller for a smaller $c$, although there is some overlap between the error bars.\n\n> Make a detailed comparison between [our algorithm DoubleGum and XQL], including methodological differences and experimental performance\n\nWe would be happy to include a comparison of XQL and DoubleGum. We have detailed the experimental comparison above. Methodologically, in brief, XQL models TD-errors using one homoscedastic Gumbel, whereas DoubleGum models the same errors with two heteroscedastic Gumbels. This leads to a different loss function of the critic: XQL uses Gumbel regression with a loss derived from the MLE of a Gumbel, whereas our loss function is derived from moment matching the heteroscedastic Logistic (derived by combining the two heteroscedastic Gumbels) with a heteroscedastic Gaussian. While DoubleGum learns the heteroscedastic spread parameter continually throughout training, the spread parameter in XQL is a hyperparameter defined at the beginning of training and fixed throughout.\n\n> About Fig.1a and Fig.1b, there is little difference for fittings by Logistic distribution or Normal distribution.\n\nWe expect to see little difference in fittings between a *hetero*scedastic Normal and the *hetero*scedastic Logistic. However, we also expect to see a vast difference in fit between a *hetero*scedastic Logistic and the *homo*scedastic Normal. We believe that did not make the distinction between hetero and homo Normals is clear enough in Figure 1. Therefore, we have updated Figure 1 in our attached 1-page `.pdf`. More details are in point 3 of our global rebuttal.\n\nUpdated Fig.1a fits homoscedastic distributions and Updated Fig.1b fits heteroscedastic distributions. Visually, the heteroscedastic distributions fit the underlying histogram far better than the homoscedastic distributions. Also, Updated Fig.1c matches our expectation and shows little difference between hetero-Logistic and hetero-Normal, but a vast difference with the homo-Normal.\n\nWe hope that our response clarifies the motivation for the mathematics in Section 3.1, and presents compelling evidence for DoubleGum's pessimistic value estimation."}, {"review_id": "O02gSBe1Nq", "reviewer": "Reviewer_WFvy", "summary": "Typically, TD learning assumes that the TD error follows a normal distribution with a fixed variance (induced by the Bellman squared loss). The authors argue that this assumption is too coarse in practice since the maximization of noisy Q-values across actions (when backing up the value in TD learning) is usually not Gaussian. To fix this, the authors discuss the proper limiting distribution of the Q-function when accounting for the Q-function maximization, and provides a practical temporal-difference backup algorithm that can accurately capture the distribution in the discrete control case and approximately capture it in the continuous control case. On a range of simulated robotic and control tasks, the proposed method is able to achieve better performance compared with existing approaches that naively backup the noisy target (without considering the interaction between the noise and the action maximization).", "questions": "- L170: $\\sigma(s)$ => $\\sigma(s, a)$?\n- I wonder how much of the problem in the assumption (that TD error is assumed to be Gaussian) can already be addressed by using distributional RL. \n- It is very interesting to see that the method has a parameter that can adjust for the degree of pessimism more smoothly than the twin networks in TD3. There are also methods like RedQ that can adjust this with more fine-grain controls which has not been considered by the authors (though admittedly RedQ would be more expensive as it requires more critic ensemble elements).", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "To the best of my knowledge, this is the first paper that uses Gumbel distributions to model TD errors. The main strength of the paper is in the theoretical analysis around this TD error model and a theoretically justified, novel algorithm that can perform approximated TD backup with this new TD error model. The writing is clear and very easy to follow.", "weaknesses": "The main weakness of the paper is the empirical evaluation. Overall, I am not convinced that 1) assuming that TD errors follow a homoscedastic normal distribution is problematic on the domains of tasks being tested, 2) the proposed method is able to improve upon existing approaches on these tasks. \n\n\n- Section 4: I am not sure I am convinced that the Logistic model fits the empirical data better than the Gaussian model (from Figure 1). It actually makes me wonder how much is actually needed in practice to accurately account for the Gumbel noise. When do you expect the gap between these two models to be bigger? \n- L196-197: How is the error bar computed? It seems a bit problematic to claim that the normal distribution is a suitable approximation given that their error bars overlap since it could be just due to the distribution variance. \n- Just from Figure 3 it is hard to tell whether DoubleGum is better than MoG-DDPG. DoubleGum is slightly better on MuJoCo but slightly worse on both DMC and Box2D.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your review!\n\nWe are very happy you highlighted our strong theoretical analysis of TD-errors, our justification for our resultant algorithm (DoubleGum), and that you found the writing clear and very easy to follow.\n\n> [not convinced that] the proposed method is able to improve upon existing approaches\n\nWe have uploaded Figures 14 and 15 in our attached 1-page `.pdf`, which show aggregate learning curves across all 33 continuous control tasks. In these graphs, DoubleGum outperforms all benchmark algorithms. More details are given in point 1 of our global rebuttal.\n\n> not sure I am convinced that the Logistic model fits the empirical data better than the Gaussian model (from Figure 1)\n\nWe expect little difference in fit between the *hetero*scedastic Normal and *hetero*scedastic Logistic. However, we also expect a vast difference in fit between a *hetero*scedastic Logistic and the *homo*scedastic Normal. We have updated Figure 1 to show this, and it is presented most prominently in Figure 1c. Updated Figure 1a fits homoscedastic distributions and Updated Figure 1b fits heteroscedastic distributions. Visually, heteroscedastic distributions fit far better than homoscedastic ones. More details are in point 3 of our global rebuttal.\n\n> How much is actually needed in practice to accurately account for the Gumbel noise\n\nDue to Updated Figure 1c, we believe we can do this with the heteroscedastic Normal.\n\n> When do you expect the gap between these two models to be bigger?\n\nIn Updated Figure 1c, the gap between the homoscedastic Normal and the heteroscedastic distributions increases during training. However, the gap between both heteroscedastic distributions remains close throughout.\n\n> L196-197: How is the error bar computed?\n\nAs mentioned in lines 185-6, \"The line and error bars in Figure 1c reflect the empirical mean and standard deviation of the fitted NLLs computed over 12 training runs\". The same procedure is used in Updated Figure 1.\n\n> Is seems a bit problematic to claim that the normal distribution is a suitable approximation given that their error bars overlap?\n\nDue to Updated Figure 1c, we claim that the heteroscedastic Normal is a suitable approximation, but the homoscedastic Normal is not. \n\n> Just from Figure 3 it is hard to tell whether DoubleGum is better than MoG-DDPG.\n\nFigure 14 in our 1-page `.pdf` is an aggregate version of Figure 3. This new graph shows that in aggregate, DoubleGum outperforms MoG-DDPG. More details are presented in Point 1 of our global rebuttal.\n\n> L170: $\\sigma(s)$ => $\\sigma(s, a)$?\n\nThis is indeed a typo! \n\n> how much of the problem in the assumption (that TD error is assumed to be Gaussian) can already be addressed by using distributional RL\n\nWe added a new baseline: QR-DDPG, which combines Quantile Regression [1] with DDPG. Following the original implementation, we learn 201 discrete quantiles. Pessimism of QR-DDPG is adjusted by deciding whether to use Twin Critics/not. Twin-QR-DDPG trains an ensemble of two Quantile Critics, and selects the minimum of two quantile estimates to compute the bootstrapped estimates. Results are reported in our 1-page `.pdf`.\n\nDoubleGum outperforms QR-DDPG with default hyperparameters (Figure 14) and tuned pessimism per-suite (Figure 15). With default hyperparameters, DoubleGum outperforms QR-DDPG, with a bit of overlap in DeepMind Control and MetaWorld (Updated Figure 3), and comfortably outperforms QR-DDPG with tuned pessimism per suite (Updated Figure 5). In aggregate, MoG-DDPG outperformed QR-DDPG (Figure 14). Note that MoG-DDPG is a form of distributional RL and as mentioned in lines 236-7, DoubleGum may be considered a special case of MoG-DDPG.\n\nOur theory showed that TD-errors follow a heteroscedastic Logistic, and we believe that modeling this distribution should be sufficient for distributional RL. We hypothesize that more complex distributions considered by QR and MoG might overfit, but this is a question beyond the scope of our work.\n\nFinally, there are empirically larger gains in performance by adjusting pessimism (ie choosing DDPG or TD3) vs choosing to use distributional RL/not (QR-DDPG vs DDPG) (Figure 13, attached `.pdf`). While it is important to model the distribution of TD-errors, another important component is adjusting pessimism, and DoubleGum presents a computationally efficient way of doing both.\n\n[1] Dabney, W., Rowland, M., Bellemare, M. G., and Munos, R. Distributional reinforcement learning with quantile regression.\n\n> methods like RedQ that can adjust this [pessimism] with more fine-grain controls which has not been considered by the authors\n\nAs REDQ involves an increased replay ratio, we did not have the time during the rebuttal period to benchmark it. Instead, we introduce a new algorithm which we name FinerTD3. FinerTD3 trains an ensemble of critics and selects the $i$th smallest value within the ensemble as the bootstrapped target per sample. Note TD3 is a special case with an ensemble of 2 and selecting the smallest value. We used an ensemble size of 5. Over all continuous control tasks, we found that it was best to select the second-smallest value in the ensemble. These values were used in Updated Figure 3 and Figure 14 (graphs with default hyperparameters). Per each suite in DeepMind Control, MuJoCo, MetaWorld and Box2D, the best smallest values to select were respectively 5 (largest), 2 (second smallest), 4 (fourth smallest), and 2(second smallest). These values were used in Updated Figure 4 and Figure 15 (graphs where pessimism is tuned per suite).\n\nIn aggregate (Figures 14 and 15), FinerTD3 performs marginally poorer than DoubleGum. We believe this is because FinerTD3 adjusts pessimism finer than other baselines, but still not as fine as the continuous scalar in DoubleGum.\n\nWe hope that our response better motivates the problem of modeling TD-errors as homoscedastic normal distribution and better showcases the empirical improvements of DoubleGum."}, {"review_id": "KDiYIi2kP4", "reviewer": "Reviewer_uPm8", "summary": "The paper proposes a combination of Gumbel noise with Q-learning. The idea is based on the high level observation that regular L-2 loss based Q-learning can be understood as maximum likelihood estimation with Gaussian noise. The paper derives the practical algorithm and shows some improvements over baselines.", "questions": "#### === ** Definition of Q-function and random variables ** ===\n\nThe definition of the equality in distribution look a bit confusing to me in Eqn 9, \n\n$$Q_\\theta(s,a) =_d Q*(s,a) - g_\\theta(s,a)$$\n\nHere $g_\\theta(s,a)$ is a random variable whose distribution depends on $(s,a)$. As a result, the RHS is a random variable with a fixed probability law (once $\\theta$ is fixed). However, the Q-function on the LHS is a single scalar at $(s,a)$. In general, how should we expect such an equality to hold? If we parameterize LHS by a neural network, it outputs a single scalar and the equality cannot hold in general.\n\nRelated to this, the equality of Eqn 10 seems to be an equality in scalar, i.e. both LHS and RHS are scalar quantities. However, $g_\\theta(s,a)$ on the LHS is a random variable, and is in general not a scalar? The definition of $y*$ in Eqn 14, on the other hand, confirms that Eqn 10 should have been an equality in scalar since the RHS $y*(s,a)$ is a scalar.\n\n#### === **Parameterization of Q-function** ===\n\nFollowing the above discussion, in line 127 there is a use of distributional equality again, where LHS is $Q_\\theta^\\text{new}$. For this equality to hold, we do have to parameterize the Q-function output as a distribution at $(s,a)$ instead of a single scalar? A further clarification on this would be very helpful.\n\n#### === **Convergence guarantee** ===\n\nThe algorithm proposed is very linked to max-entropy RL, the convergence property of which has been established. The L-2 based Q-learning algorithm is also convergent under suitable conditions. I wonder if there is a similar guarantee here, and if yes would be good to have in the main paper.\n\n#### === **Empirical results** ===\n\nGumbel DDPG seems to improve over DDPG but not over TD3. I guess a main reason might be a TD3 uses double Q-learning, which is orthogonal to what's proposed here. Can we combine Gumbel trick with TD3 and show improvements over TD3? DDPG is in general a much more inferior baseline than TD3 due to the over-estimation bias, it'd be nice to showcase improvements over TD3 too.\n\nFrom Fig 4 it seems that the performance is sensitive to the hyper-parameter $c$. How is the choice of $c=0$ selected in other experiments in comparison to alternative baselines? Is $c=0$ the best-performing hyper-parameter in general?\n\n", "limitations": "Discussed above.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper takes a relatively novel perspective on interpreting L-2 loss based Q-learning algorithm as maximum likelihood estimation, and derives an empirically novel algorithm based on the Gumbel noise in place of Gaussian noise. The resulting algorithm looks easy to implement and showcases some practical improvements.", "weaknesses": "Since the algorithm is heuristically motivated, the paper seems to lack a solid theoretical grounding as to whether the newly proposed algorithm would converge or not (not formally stated or proved in the paper). The empirical improvements are interesting but can use more results to further demonstrate the promise of this new approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your review!\n\nWe are very happy that you found both theoretical and empirical novelty within our paper, and that our resulting DoubleGum algorithm was practical and easy to implement.\n\n> The definition of the equality in distribution look a bit confusing to me in Eqn 9\n\nWe have realized we should use an equality in samples $=$ whenever we've used a distributional equality $\\overset{d}{=}$. Equation 9 constructs a new random variable $Q_\\theta(s, a) \\sim Q(s, \\cdot)$ by defining a data-generating process that subtracts a random variable $g_{\\theta, a}(s) \\sim g(s)$ from the optimal $Q^\\star(s, a)$ given an $\\theta$ and $a$ index. As Equation 9 concerns samples and not random variables, it is not appropriate to use a distributional equality.\n\n$Q_\\theta$s is a random variable because it varies throughout training. In supervised learning, it is standard to model the output of a neural network as a random variable -- in regression the neural network estimate is a random variable that differs from the optimal value by Normal noise. We work through the mathematics of this example in Appendix A.1. There should again be a sample equality in the equation between lines 472-3.\n\n> equality of Eqn 10 seems to be an equality in scalar i.e. both LHS and RHS are scalar quantities.\n\nThe LHS of Eqn 10 involves samples from two random variables whose sum is constructed to be a constant, which is the RHS.\n\n> we do have to parameterize the Q-function output as a distribution at $(s, a)$ instead of a single scalar?\n\n$Q_{\\theta}^\\text{new}$ is a scalar. In our anonymous codebase (link in appendix lines 547-8), line 36 of `policies_cont/networks/gaussian_critic.py` shows this.\n\n> TD3 uses double Q-learning, which is orthogonal to what's proposed here\n\nWe do not think DoubleGum is orthogonal to TD3 as both algorithms induce pessimism. The pessimism of DoubleGum is adjusted by changing the pessimism factor $c$. TD3 computes a pessimistic estimate of the bootstrapped target by taking a sample-wise minimum from the ensemble. Pessimism of DDPG can be varied by using Twin Critics/not.\n\n> it'd be nice to showcase improvements over TD3 too\n\nOur implementations of DDPG with Twin Networks and TD3 are equivalent, and we have thus changed  \"DDPG (best w/wo Twin Networks)\" to \"Best of DDPG/TD3\" in our 1-page `.pdf` whenever necessary. Thus, in aggregate, DoubleGum with default hyperparameters outperforms all benchmark algorithms, including TD3 (Figure 14). Also, the pessimism factor hyperparameter $c$ may be adjusted to outperform TD3 both in aggregate across suites (Figure 15) and per suite (Updated Figure 5). More details are presented in Point 1 of our global rebuttal.\n\n> it seems that the performance is sensitive to the hyper-parameter $c$\n\nWe intend $c$ in DoubleGum to be similar to how the learning rate is used in stochastic gradient descent optimizers, so sensitivity is preferred. We cover this in Point 4 of our global rebuttal.\n\n> How is $c$ selected? Is $c=0$ the best-performing hyper-parameter in general?\n\nIn our attached 1-page `.pdf`, we have created Figure 16, which is Figure 4 aggregated into one graph. We have now changed the default pessimism factor to $c = -0.1$, as it is the value with best aggregate performance.\n\n> lack a solid theoretical grounding as to whether the newly proposed algorithm would converge or not (not formally stated or proved in the paper)\n\nTo the best of our knowledge, there are two types of convergence analysis in Q-Learning: 1) operator-theoretic analysis over tabular Q-functions, and 2) training dynamics of neural network parameters. We believe the second is more appropriate for DoubleGum, because our theory addresses issues in using neural networks (and not tables) for Q-learning. DoubleGum uses variance networks, but we are unaware of any literature analyzing their training dynamics. We thus believe convergence guarantees are beyond the scope of this rebuttal (and our paper)\n\nNevertheless, we are happy to provide an intuition about convergence we are happy to include in our paper. First, consider the following a numerically equivalent loss function to that of DoubleGum in Equation 27: $\\sigma_{\\bar{\\theta}} \\log \\sigma_\\theta + \\frac{\\epsilon_\\theta^2}{\\sigma_\\theta}$. For each value of $\\epsilon_\\theta^2$, our resultant function is convex wrt the numerical value of $\\sigma_\\theta$. If $\\sigma_\\theta$ is large, the loss is dominated by $\\sigma_{\\bar{\\theta}} \\log \\sigma_\\theta$ and gradient descent will minimize $\\sigma_\\theta$. If $\\sigma_\\theta$ is small, the loss is dominated by $\\epsilon_\\theta^2$, and we can now rely on convergence guarantees of l2 Q-learning.\n\nThe guarantee we use is DR3 [2], which states convergence if $\\phi_\\theta(s, a)^\\top \\phi_\\theta(s, a) - \\gamma \\phi_\\theta(s, a)^\\top \\phi_\\theta(s, a) \\geq 0$, where $\\phi(s, a)$ is the penultimate layer of the network. (This was also found by [3]). Following work [4] satisfied this condition by normalizing the layer before the value head. To do this, we used GroupNorm with 16 groups as it had empirically stronger performance. In 100 training runs on three of the hardest environments (humanoid-run, Humanoid-v4, and BipedalWalkerHardcore-v3), all seeds resulted in positive learning curves. We would be happy to include this ablation in our appendix.\n\n[1] Seitzer, M. et al. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks.\n\n[2] Aviral Kumar et al. DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization. \n\n[3] Wang, Z. T. and Ueda, M. A convergent and efficient deep Q network algorithm.\n\n[4] Kumar, A. et al. Offline q-learning on diverse multi-task data both scales and generalizes.\n\nTo reiterate, for the reasons outlined above, we believe that theoretical guarantees for the convergence of DoubleGum are outside the scope of our work.\nNevertheless, we hope the additional results we have presented showcase the promise of our new approach."}, {"review_id": "I2gaetFfFl", "reviewer": "Reviewer_je5E", "summary": "The paper provides empirical evidence that it is inaccurate to assume the temporal difference error follows a homoscedastic normal distribution. The paper proposes to use Gumbel distribution to make a replacement. The optimal action value is considered as the learned value estimation plus a noise sampled from a Gumbel distribution. The Bellman Equation has changed accordingly. Based on this idea, a new algorithm for both discrete and continuous control tasks is derived, called DoubleGum. The paper empirically tested DoubleGum in various tasks, including several relatively complex environments.", "questions": "I would appreciate it if the authors could explain the abovementioned concerns.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper provides clear empirical evidence to explain that the temporal difference error is better fitted to a Guambel distribution rather than a normal distribution. \n\n- The paper has a relatively complete related work list. The related work section considered other analyses of the noise in Q-learning and works proposing similar methods.\n\n- DoubleGum empirically works on both discrete and continuous control tasks. To support this, in the experiment section, the paper empirically tested it on multiple environments, including both simple and complex tasks.\n\n- Detailed hyperparameters and computational resources needed are reported, improving the reproducibility. \n\n", "weaknesses": "- The main concern comes from the gap between the theory and the practical algorithm. The whole idea is based on one assumption, which is, estimating the temporal difference error with a Gumbel distribution is better than Normal distribution. The Bellman equation is also modified according to the same intuition. However, when deriving the practical algorithm, the paper points out that the Q-function cannot be learned by Gumbel or logistic regression because there lacks a sufficient statistical estimator. So, DoubleGum instead used a generalized method of moments to match the logistic with a normal distribution.\n\n- Although the empirical results did suggest DoubleGum learned faster than the baseline (DDPG), it is not clear to me if the advantage does come from the Gumbel distribution, as the practical algorithm is not using the Gumbel distribution as previous sections discuss. \n\n- The other concern is mainly about the baseline choice. SAC is another algorithm that empirically works well for online learning. It estimates the action value with considering the entropy. The setting that uses a softmax instead of max is consistent with a part of the proposed method in the paper. It might be worth checking SAC’s performance in the tasks listed in paper, and comparing it with DoubleGum.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your review!\n\nWe are delighted to read your positive feedback on our experimental methodology and reproducibility, which we care deeply about.\n\n> The whole idea based on one assumption, estimating the temporal difference error with a Gumbel distribution is better than Normal distribution.\n\nWe would like to point out that we estimate TD errors with *two* heteroscedastic Gumbel distributions (Equation 15), which we rewrite into a heteroscedastic Logistic (Equation 19).\n\n> DoubleGum instead used a generalized method of moments to match the logistic with a normal distribution.\n\nWe would like to point out that treating the heteroscedastic Logistic distribution (Equation 19) as a *homo*scedastic Normal (Equation 5) is problematic, but generalized moment matching with a *hetero*scedastic Normal is not.  We realize that we did not make the distinction between hetero/homo clear enough in our paper, and will update the text if accepted.\n\nThe generalized method of moments estimator has been widely used in the statistics and econometrics literature.  We were specifically inspired to use the generalized method of moments from Equations 42-45 of [1], which estimates the location and spread parameters of a Gumbel distribution in this manner.  The same method is presented in [2] to demonstrate sampling from a Gumbel distribution in the NumPy package.\n\n[1] Jowitt, P. W. (1979). The extreme-value type-1 distribution and the principle of maximum entropy.\n\n[2] Code sample at the bottom of the documentation for numpy.random.gumbel\n\n> not clear to me if the advantage does come from [modelling] the Gumbel distribution\n\nThe advantage comes from modeling the Q-values heteroscedastically, which is motivated by our theory that reveals that the TD errors are heteroscedastic, not homoscedastic.  Learning a sample-dependent variance is not typically done in Q-learning, and not in our baselines, and therefore in our experiments with default hyperparameters (Updated Figure 3, Figure 14), is the source of our empirical advantage.\n\n> checking SAC’s performance in the tasks listed in paper, and comparing it with DoubleGum\n    \nWe now benchmark against SAC. and show that in Updated Figures 3 and 5, as well as Figures 14 and 15 in our attached 1-page `.pdf`, DoubleGum is empirically stronger.  More details are given in point 2 of our global rebuttal.\n\nWe hope that our response helps bridge the gap between our mathematical analysis and the resulting algorithm, and shows stronger empirical evidence for the effectiveness of DoubleGum."}, {"review_id": "whE1Xcyzx5", "reviewer": "Reviewer_Mh9L", "summary": "The authors propose a novel off-policy algorithm DoubleGum based on their new noise model which is claimed to be more stable for training and reaches higher performance for various discrete and continuous environments. ", "questions": "1.Curious why SAC is not included in the experiments. To my knowledge, it is a very strong off-policy algorithm. It is stable and outperforms DDPG and TD3 in a bunch of continuous tasks. \n2.Can we claim DoubleGum with adjusting pessimism the new state-of-the art?\n3.Will there be a way to measure how much more stable is DoubleGum(with fixed pessimism) across different tasks?Since in fixed pessimism, as shown in fig3, the performance seems to be sacrificed. \n", "limitations": "As authors mentioned in the paper, the proposed algorithm seems to be very sensitive to hyper-parameters, which is not very applicable if we need to tune pessimism per task.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and well organized. The authors proposed a novel noise model and off-policy DRL algorithm and analyse their strength through a set of extensive experiments. Some experimental results are interesting and strong. They also provide nice math supporting the paper.\n\n", "weaknesses": "Figures are a bit confusing. For example, it's a bit hard to see just from the figures that how stable is the new algorithm comparing to existing state-of-the-art. Also, I saw TD3, DDPG MoG-DDPG in figure3, then it is just DDPG (best w/wo Twin Networks) in figure 5. Makes me a bit confused and not sure whether DoubleGum with adjusting pessimism outperforms all the other algorithms or just DDPG?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your review!\n\nWe are happy to read that you found our algorithm novel and that our paper was well-written and organized.\n\n> TD3, DDPG MoG-DDPG in figure3, then it is just DDPG (best w/wo Twin Networks) in figure 5\n\nFigures 3 and 5 show two different ways of evaluating DoubleGum.  They correspond to two different methods in which we benchmark DoubleGum, namely 1) keeping hyperparameters constant (to their default values) across all tasks and suites 2) varying hyperparameters between suites, but constant within all tasks per suite.  We would like to point out that we do not vary hyperparameters across tasks belonging to the same suite, and thus we do not tune pessimism per task.  We realize that we did not make this clear in our original text, and, if accepted, would be happy to change it.\n\nThe hyperparameter that we tune is pessimism.  For DoubleGum, this corresponds to changing pessimism factor $c$.  For DDPG, this corresponds to whether we use Twin Networks or not, resulting in our caption of \"DDPG (best w/wo Twin Networks)\".  As our implementations of DDPG with Twin Networks and TD3 are equivalent, we have updated the aforementioned caption to \"Best of DDPG/TD3\" in our 1-page `.pdf` whenever necessary.\n\n> whether DoubleGum with adjusting pessimism outperforms all the other algorithms or just DDPG?\n\nWe have uploaded Figures 14 and 15 in our attached 1-page `.pdf`, which show aggregate learning curves across all 33 continuous control tasks.  In these graphs, DoubleGum outperforms all benchmark algorithms, including the newly benchmarked SAC and XQL.  More details are given in point 1 of our global rebuttal.\n\n> Curious why SAC is not included in the experiments\n\nWe now benchmark against SAC and show that in Updated Figures 3 and 5, as well as Figures 14 and 15 in our attached 1-page `.pdf`, DoubleGum is empirically stronger.  More details are presented in Point 2 of our global rebuttal.\n\n> Can we claim DoubleGum with adjusting pessimism the new state-of-the art?\n\nWe are hesitant to claim that DoubleGum is the new SOTA, due to the wide recent algorithmic innovations such as [1, 2], and the fact that most recent papers do not benchmark on as many tasks as we do, making comparisons difficult.  However, we would like to claim that DoubleGum is the best-performing core algorithm over all continuous control tasks in aggregate (Figures 14, 15), and would thus like to recommend that DoubleGum form the new core algorithm for many tasks, just as SAC was the core algorithm for [1], or TD3 was for [2].\n\n[1] P. D’Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, and A. Courville. Sample-\nefficient reinforcement learning by breaking the replay ratio barrier\n\n[2] S Fujimoto, WD Chang, EJ Smith, SS Gu, D Precup, D Meger. For SALE: State-Action Representation Learning for Deep Reinforcement Learning\n\n> a way to measure how much more stable is DoubleGum(with fixed pessimism) across different tasks?\n\nIn our 1-page `.pdf`, Figures 14 and 15 show learning curves aggregated across all 33 continuous control tasks we benchmark on.  These learning curves show how consistently the algorithms perform across all tasks, and thus how stable the algorithms are between tasks.  DoubleGum achieves a higher learning curve than the baselines indicating that it is more stable across tasks than the baseline algorithms.\n\n>  the proposed algorithm seems to be very sensitive to hyper-parameters \n\nWe would like to point out that while we have shown that DoubleGum is sensitive to the pessimism factor $c$, we do not think DoubleGum is any more sensitive to any other hyperparameters than our baselines are.\n\n> [DoubleGum] not very applicable if we need to tune pessimism per task.\n\nWe would like to reiterate that in our evaluations, we do not tune per task and only per suite.  If accepted, we will update our text to make this clearer.  As we intend $c$ in DoubleGum to be similar to how the learning rate is used in stochastic gradient descent optimizers, we evaluate in two different ways: untuned with default parameters, and tuned per suite.  We go into more detail in Point 4 of our global rebuttal.\n\n> in fixed pessimism, as shown in fig3, the performance seems to be sacrificed\n\nWe would expect the default parameter to have poorer performance than if we tuned it specifically for performance per task/suite.  Nevertheless, the aggregate learning curve for DoubleGum with fixed default pessimism still outperforms all baseline methods (Figure 14)."}]

Author rebuttal:
Many thanks to all reviewers for their comments and feedback!

We are delighted that all five reviewers mentioned the strong experimental analysis of our DoubleGum algorithm and that four reviewers highlighted the novelty of our theoretical analysis (dXQf, WFvy, uPm8, Mh9L).  In addition, we are also very happy that four reviewers marked the presentation as 3 (dXQf, WFvy, je5E, Mh9L), with two explicitly mentioning the clarity of writing (WFvy, Mh9L).

There were four common issues raised by our reviewers.

**1. Three reviewers (dXQF, WFvy, Mh9L) were concerned that we did not clearly show empirical improvements of our algorithm (DoubleGum) over baselines.**  We have better presented DoubleGum's empirical improvements over all baselines in Figures 14 and 15 in our 1-page `.pdf`.  These graphs aggregate all learning curves over all tasks in all suites into one graph.  They show that in aggregate, DoubleGum outperforms all baselines when all methods use default hyperparameters (Figure 14) and also when all methods' pessimism hyperparameters are tuned per suite (Figure 15).

**2. Three reviewers (dXQf, je5E, and Mh9L) suggested benchmarking the algorithm SAC.**  We have benchmarked SAC and added its aggregate learning curves to Updated Figures 3 and 5 as well as Figures 14 and 15 in our 1-page `.pdf`.  Pessimism of SAC is tuned by varying the use of Twin Networks/not, just as we do with TD3/DDPG in Updated Figure 3, as specified in line 250.  In aggregate, DoubleGum outperforms SAC with both default hyperparameters (Figure 14) and tuned pessimism per suite (Figure 15).  Finally, over each suite, DoubleGum outperforms SAC in all suites, both with default hyperparameters (Updated Figure 3) and tuned pessimism per suite (Updated Figure 5).  If accepted, we will upload individual learning curves from each task in the appendix.

**3. Two reviewers (dXQf, WFvy) did not find Figure 1 convincing.**  Figure 1 aims to present empirical evidence showing that the heteroscedastic Normal is a far better approximation of the heteroscedastic Logistic than a homoscedastic Normal.  We have updated Figure 1 in our 1-page `.pdf` to better present this empirical evidence.  No data has been changed between the original and updated Figure 1s.  Instead, we have changed the labels to show which distributions are hetero and homoscedastic, removed distributions in the graph that we do not discuss in the main paper, and added a homoscedastic Normal NLL curve to subgraph 'c' such that the goodness of fit for all three aforementioned distributions may be compared.

**4. Two reviewers (uPm8, Mh9L) queried the sensitivity and stability of our algorithm with respect to our pessimism hyperparameter $c$.**  We intend $c$ in DoubleGum to be used in a similar way to the learning rate in gradient descent optimizers.  We thus intend DoubleGum to be sensitive to $c$ such that performance may be improved by tuning $c$, but we have also presented a default value of $c=-0.1$ (updated from $c=0$ in our paper) that showcases good aggregate performance across all tasks (Figure 14).  The default value is inspired by optimizers with a default learning rate with good aggregate performance to naively try (eg Adam has a lr of 3e-4).

Individual issues raised by the reviewers include suggesting more algorithms to be benchmarked (XQL, distributional RL, an algorithm with finer pessimism control), empirical evidence for the effectiveness of our pessimism factor hyperparameter, a discussion of theoretical convergence guarantees for our algorithm DoubleGum, and clarifications of our mathematics.  We address all individual issues in individual rebuttals.

Meta-review: As the generally positive reviews indicate, this paper presents concepts and results that could have broad community interest. There seems to be a difference in viewpoints that kept one reviewer critical.
Decision: Accept (spotlight)