Title: Double Gumbel Q-Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: deep reinforcement learning

Q-Learning

TD-Learning with function approximation

extreme value theory

maximum-likelihood estimation

moment-matching
Abstract: We show that Deep Neural Networks introduce two heteroscedastic Gumbel noise sources into Q-Learning.  To account for these noise sources, we propose Double Gumbel Q-Learning, a Deep Q-Learning algorithm applicable for both discrete and continuous control.  In discrete control, we derive a closed-form expression for the loss function of our algorithm.  In continuous control, this loss function is intractable and we therefore derive an approximation with a hyperparameter whose value regulates pessimism in Q-Learning.  We present a default value for our pessimism hyperparameter that enables DoubleGum to outperform DDPG, TD3, SAC, XQL, quantile regression, and Mixture-of-Gaussian Critics in aggregate over 33 tasks from DeepMind Control, MuJoCo, MetaWorld, and Box2D and show that tuning this hyperparameter may further improve sample efficiency.

Double Gumbel Q-Learning

Abstract

We show that Deep Neural Networks introduce two heteroscedastic Gumbel noise sources into Q-Learning. To account for these noise sources, we propose Double Gumbel Q-Learning, a Deep Q-Learning algorithm applicable for both discrete and continuous control. In discrete control, we derive a closed-form expression for the loss function of our algorithm. In continuous control, this loss function is intractable and we therefore derive an approximation with a hyperparameter whose value regulates pessimism in Q-Learning. We present a default value for our pessimism hyperparameter that enables DoubleGum to outperform DDPG, TD3, SAC, XQL, quantile regression, and Mixture-of-Gaussian Critics in aggregate over 33 tasks from DeepMind Control, MuJoCo, MetaWorld, and Box2D and show that tuning this hyperparameter may further improve sample efficiency.

1 Introduction

Reinforcement Learning (RL) algorithms learn optimally rewarding behaviors (Sutton and Barto, 1998, 2018). Recent RL algorithms have attained superhuman performance in Atari (Mnih et al., 2015), Go (Silver et al., 2016), Chess (Schrittwieser et al., 2020), StarCraft (Vinyals et al., 2019), simulation racing (Wurman et al., 2022), and have also been used to control stratospheric balloons (Bellemare et al., 2020) and magnetic fields in Tokamak nuclear fusion reactors (Degrave et al., 2022). All these aforementioned agents use a Q -function (Watkins and Dayan, 1992) to measure the respective quality of their behaviors.

Many algorithms (Lillicrap et al., 2015; Mnih et al., 2016, 2015; Schulman et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018b) learn a Q -function by minimizing the Mean-Squared Bellman Error (MSBE). The functional form of the MSBE was motivated by an analogy to the popular Mean-Squared Error (MSE) in supervised learning regression problems, but there is little theoretical justification for its use in RL (Bradtke and Barto, 1996; Ernst et al., 2005; Riedmiller, 2005).

Training a deep neural network to minimize the MSBE is empirically unstable (Irpan, 2018; Henderson et al., 2018; Van Hasselt et al., 2016) because deep neural networks induce overestimation bias in the Q -function (Thrun and Schwartz, 1993; Fujimoto et al., 2018). A popular method to reduce overestimation 'pessimistically' selects less positive Q -values from an ensemble of Q -functions, effectively returning low-quantile estimates (Fujimoto et al., 2018; Kuznetsov et al., 2020; Ball et al., 2023). However, an ensemble is computationally expensive and quantiles only provide discretegrained control over the degree of pessimism. Curiously, pessimism has mainly been applied to 'continuous control' problems that require continuous-valued actions and not discrete control.

This paper analyzes noise in Deep Q -Learning. We introduce a loss function for Deep Q -Learning and a scalar hyperparameter that adjusts pessimism without an ensemble. These two innovations form Double Gumbel Q -Learning (DoubleGum) , 1 an off-policy Deep Q -Learning algorithm applicable to both discrete and continuous control. Our paper is structured as follows:

Section 2.1 shows that the MSBE implicitly uses a noise model of the Bellman Optimality Equation with additive homoscedastic noise. Section 4.1 shows that this noise model is empirically too coarse to fully capture the distribution of noise in Deep Q -Learning.

Section 3 derives a noise model of the Soft Bellman Equation with additive heteroscedastic logistic noise derived from two heteroscedastic Gumbel noise sources. Section 4.1 shows that our noise model is a better empirical fit to the noise distribution in Deep Q -Learning.

Section 3.3 shows that when the soft value function in the Soft Bellman Equation is intractable in continuous control, its approximation leads to methods that adjust pessimism. Section 3.1 presents a method to adjust pessimism with a single scalar hyperparameter, and its effect is empirically verified in Section 4.2.

Section 5 discusses related work, notably presenting DoubleGum as a special case of Distributional RL and a generalization of existing Maximum Entropy (MaxEnt) RL algorithms.

Section 6 benchmarks DoubleGum, showing marginal improvements over DQN baselines on 3 classic discrete control tasks and improved aggregate performance over DDPG, SAC, TD3, XQL and two Distributional RL algorithms over 33 continuous control tasks.

2 Mathematical Background

Reinforcement Learning agents learn to make maximally rewarding decisions in an environment (Sutton and Barto, 1998, 2018). In Markov Decision Process (MDP) environments, a Q -function specifies the expected amount of reward an agent obtains (Watkins and Dayan, 1992) and is also known as a state-action value function. MDPs define a transition probability p ( s ′ | s, a ) and reward r ( s, a, s ′ ) for every pair of states s, s ′ ∈ S and action a ∈ A (Bellman, 1957; Howard, 1960). Decisions are made by a policy π ( a | s ) at every discrete timestep and the Q -function of π at s , a is

where the discount factor γ ∈ ]0 , 1[ ensures finite Q for infinite time-horizon n (Samuelson, 1937). The expectation in the Q -function is computed over a trajectory sequence of s and a induced by the coupling of the policy and the transition function.

In every MDP, an optimal policy π ⋆ makes maximally rewarding decisions following s → arg max a Q ⋆ ( s, a ) (Watkins and Dayan, 1992). Q ⋆ is the optimal value function specified as the unique fixed point of the Bellman Optimality Equation of the MDP (Bellman, 1957)

In Deep Q -Learning, Q ⋆ is approximated by a neural network Q θ with parameters θ . Following Munos and Szepesvári (2008), θ is updated by a sequence of optimization problems whose i th stage iteration is given by

where the 'stop-gradient' operator ( ¯ · ) denotes evaluating but not optimizing an objective with respect to its argument and y is dubbed the bootstrapped target. In this work, we restrict our environments to deterministic MDPs, so the resultant expectation over s ′ is computed over one sample.

Typically, the loss function L( Q θ , y ¯ θ i ) = E s,a ( Q θ ( s, a ) -y ¯ θ i ( s, a ) ) 2 is the Mean-Squared Bellman Error (MSBE). At every i , L is minimized with stochastic gradient descent by a fixed number of gradient descent steps and not optimized to convergence (Mnih et al., 2015; Lillicrap et al., 2015).

2.1 Deriving the Mean-Squared Bellman Error

Thrun and Schwartz (1993) model Q θ (s, a) as a random variable whose value varies with θ, s and a . They set up a generative process Q θ ( s, a ) = Q ⋆ ( s, a ) -ϵ θ,s,a showing that the function approximator Q θ is a noisy function of the implicit true value Q ⋆ , but they do not make any assumptions about ϵ . 2

We derive the MSBE with two additional assumptions: ϵ θ,s,a iid ∼ N (0 , σ 2 ) , where N is a homoscedastic normal due to the Central Limit Theorem (CLT) 3 and secondly Q ⋆ ( s, a ) ≈ y ¯ θ ( s, a ) , where θ are the parameters at an arbitrary optimization stage. Incorporating these two assumptions into the noise model yields y ¯ θ ( s, a ) = Q θ ( s, a ) + ϵ θ,s,a , uncovering the implicit assumption that TemporalDifference (TD) errors y ¯ θ -Q θ follow a homoscedastic normal. Maximum Likelihood Estimation (MLE) of θ on the resultant noise model yields the MSBE after abstracting away the constants

2.2 The Limiting Distribution in Bootstrapped Targets

Both assumptions used to derive the MSBE are theoretically weak. First, the CLT states that iid samples of an underlying distribution tend towards a homoscedastic normal but do not account for the form of the underlying distribution. Secondly, there is no analytic justification for Q ⋆ ( s, a ) ≈ y ¯ θ ( s, a ) . Substituting the Thrun and Schwartz (1993) model in the RHS of the Bellman Optimality Equation yields

̸

When the max over a finite number of iid samples with unbounded support is taken, the Extreme Value Theorem (EVT) gives the limiting distribution of the resultant random variable as a Gumbel distribution (Fisher and Tippett, 1928; Gnedenko, 1943), defined here as g θ,s ′ ∼ G ( α, β ) . Expressions relating α and β to the parameters of the underlying iid random variables are rarely analytically expressible (Kimball, 1946; Jowitt, 1979) and there is no guarantee Q ⋆ ( s, a ) ≈ y ¯ θ ( s, a ) .

3 Deriving a Deep Q-Learning Algorithm from First Principles

We find an analytic expression that relates the limiting Gumbel distribution to parameters of the underlying noise distribution from function approximation. To do so, we assume that noise in the Thrun and Schwartz (1993) model is a heteroscedastic Gumbel noise with state-dependent spread

Appendix A.1 restates the finding of McFadden et al. (1973) and Rust (1994), which results in the following expression with the soft value function V soft θ

Substituting Equations 2 and 3 into 1 yields a new noise model of the Soft Bellman Optimality Equation with two function approximators and two heteroscedastic Gumbel noise sources

We now develop this noise model into a loss function used in our algorithm, DoubleGum.

3.1 Merging Two Distributions

We replace Q θ with a new function approximator Q new θ ( s, a ) = Q θ ( s, a ) + g θ ( s ) , resulting in a new soft value function on the RHS of Equation 4

and a LHS with a heteroscedastic logistic random variable obtained in Appendix A.3 from the difference of two Gumbel random variables with equation spreads. Note that this operation is only permissible in MDPs without terminating states. Equation 4 simplifies into

the Soft Bellman Equation with heteroscedastic logistic noise. The bootstrapped targets in this equation is the RHS, which we will denote by y soft ¯ θ ( s, a ) .

3.2 Parameter Estimation with Moment Matching

Equation 5 does not allow MLE of θ because the heteroscedastic logistic is not a member of the exponential family and lacks a sufficient statistical estimator. We learn θ with moment matching by MLE of a heteroscedastic normal y soft ¯ θ ( s, a ) ∼ N ( Q new θ ( s, a ) , σ θ ( s )) given by

and recovering the logistic location and spread as Q new θ ( s, a ) and β θ ( s ) = σ θ ( s ) √ 3 π .

3.3 Computing the Bootstrapped Targets

y soft θ is calculated with one sample from s ′ and a ′ where appropriate. In discrete control, A is finite and the expression V soft, new θ is a sum. The resultant bootstrapped target in discrete control is

In continuous control, V soft, new θ is intractable because A contains infinitesimally small quantities. Appendix A.4, adapted from Equation 19 in Haarnoja et al. (2017), variationally approximates the integral with respect to a policy neural network π ϕ ( a | s ) with parameters ϕ , producing

Denoting C [ π ϕ || p θ ] as a constant π √ 3 c yields bootstrapped targets of

We treat c as a hyperparameter named the pessimism factor defined before training and held constant throughout. Appendix F.1 empirically finds that a default value of c = -0 . 1 and details how c may be adjusted. Appendix B.3 shows that the pessimism factor performs the same role as pessimistic ensembles but with more computational efficiency and fine-grained control.

3.4 Algorithm

Algorithm 1 presents pseudocode for DoubleGum, an off-policy Deep Q -Learning algorithm that learns θ in both discrete and continuous control. DoubleGum is named after the two Gumbels used in the derivation of its noise model, Equation 4. We outline some implementation details that empirically improved the stability of training.

Target Networks : As is standard, y discrete and y continuous are computed with target networks, whose parameters we denote by ψ . We used target networks with exponential moving weight updating. This

Algorithm 1: DoubleGum: An Off-Policy Deep Q -Learning Algorithm

is commonly used in continuous control (Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018b), and has been recently been shown to improve discrete control (D'Oro et al., 2022).

Q-Functions with Variance Networks : We learn σ with variance networks (Nix and Weigend, 1994; Kendall and Gal, 2017). In discrete control, σ θ ( s ) is a variance head added to the Q -network, and is approximated as σ θ ( s, a ) in continuous control. The variance head is a single linear layer whose input is the penultimate layer of the main Q -network and output is a single unit with a SoftPlus activation function (Dugas et al., 2000) that ensures its positivity. Seitzer et al. (2022) improves stability during training by multiplying the entire loss by σ ¯ θ , the numerical value of the variance network, yielding a loss function for θ which for continuous control is

Note that in the above equation σ ¯ θ ( s, a ) would be σ ¯ θ ( s ) in discrete control. We do not have any convergence guarantees for this loss function, and Appendix C discusses this issue in more detail.

Policy : In discrete control, actions are taken by max a Q new θ ( s, a ) as standard. Exploration was performed by policy churn (Schaul et al., 2022). In continuous control, the policy π ϕ is a separate network. We use a DDPG fixed-variance actor because Yarats et al. (2021) showed it trained more stably than a SAC actor with learned variance. Following Laskin et al. (2021), the actor's outputs were injected with zero-mean Normal noise with a standard deviation of 0.2 truncated to 0.3.

Network Architecture : All networks had two hidden layers of size 256, ReLU activations (Glorot et al., 2011), orthogonal initialization (Saxe et al., 2013) with a gain of √ 2 for all layers apart from the last layer of the policy and variance head, which had gains of 1 . This initialization had been shown to be empirically advantageous in policy-gradient and Q -Learning methods in Huang et al. (2022a) and Kostrikov (2021), respectively. Ball et al. (2023) improves stability in continuous control using LayerNorm (Ba et al., 2016). We find that the similar method of GroupNorm (Wu and He, 2018) with 16 groups without a shift and scale (Xu et al., 2019) worked better, but only in continuous control. All parameters were optimized by Adam (Kingma and Ba, 2014) with default hyperparameters.

-

-

-

4

-

2

0

2

4

TD Error

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Frequency Density

a

-

1

0

1

2

3

4

Standardized TD Error

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Frequency Density

b

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

-

0

.

8

0

.

0

0

.

8

1

.

6

2

.

4

NLL

c

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

0

.

050

0

.

025

0

.

000

0

.

025

0

.

050

Empirical (Histogram)

Homo-Normal

Hetero-Logistic (expected)

Homo-Gumbel (Garg et al., 2023)

Moment-Matched Hetero-Normal

0

20

40

60

80

100

0

300

600

900

1200

1500

DMC: acrobot-swingup

0

20

40

60

80

100

0

300

600

900

1200

1500

MuJoCo: Hopper-v4

0

20

40

60

80

100

0

150

300

450

Box2d: BipedalWalker-v3

0

20

40

60

80

100

0

400

800

1200

1600

MetaWorld: button-press-v2

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

Magnitude

c=-0.5

c=-0.1

c=0.0

c=0.1

c=0.5

4 Empirical Evidence for Theoretical Assumptions

4.1 Noise Distributions in Deep Q-Learning

Figure 1 shows the evolution of noise during training DoubleGum in the classic control task of CartPole-v1 . Histograms in Figures 1(a) and 1(b) were generated from 10 4 samples from the replay buffer after 1 million training timesteps. Figure 1(a) computes TD-errors by y ψ ( s, a, r, s ′ ) -Q new θ ( s, a ) , while Figure 1(b) computes standardized TD-errors by dividing the previous equation by σ θ ( s ) or σ θ ( s, a ) where appropriate. The normal, logistic, and Gumbel distributions were fitted by moment matching, and homo(scedastic) and hetero(scedastic) distributions were respectively fitted to unstandardized and standardized data. Figure 1(c) shows the mean and standard deviation over 12 Negative Log-Likelihoods (NLLs), each computed from a different training run with a different (randomly chosen) initial seed. Every NLL was computed over 10 4 samples from the replay buffer every 1000 timesteps of training.

Figure 1(a) shows that a homoscedastic normal coarsely fits the empirical distribution, forming a good estimate of the mean but not the spread. Our result contradicts Garg et al. (2023) that fitted a Gumbel to the empirical distribution. We discuss this discrepancy in Appendix E.1. Figure 1(b) shows that a heteroscedastic Logistic captures both the mean and spread of the TD-errors, validating Equations 5 and its derivation. Finally, Figure 1(c) shows that a moment-matched heteroscedastic normal used in DoubleGum is a suitable approximation to the heteroscedastic logistic throughout training, validating Equation 6. Appendix D.1 shows that the trend in Figure 1(c) holds for other discrete control environments as well as continuous control.

4.2 Adjusting The Pessimism Factor

Figure 2 plots a 12-sample IQM and standard deviation over 1 256 ∑ 256 i =1 Q new ψ ( s i , a i ) , the average magnitude of the target Q -value used in bootstrapped targets. Figure 2 and Appendix D.2 show that the average magnitude increases as the pessimism factor c increases, validating its effectiveness.

5 Related Work

5.1 Theoretical Analyses of the Noise in Q-Learning

Logistic Q-Learning Bas-Serrano et al. (2021) presents a similar noise model to DoubleGum but with a homoscedastic logistic instead of a heteroscedastic logistic in Equation 5. Their distribution is derived from the linear-programming perspective of value-function learning described in Section 6.9 of Puterman (2014)) and Peters et al. (2010). While DoubleGum learns a Q -function off-policy, Logistic Q -Learning uses on-policy rollouts to compute the Q -values of the current policy. We do not benchmark against Logistic Q -Learning because their method is on-policy and only uses linear function approximation.

Extreme Q-Learning (XQL) Garg et al. (2023) presents a noise model for Deep Q -Learning with one homosecdastic Gumbel noise source, as opposed to the two heteroscedastic Gumbels in Equation 4. Parameters are learned by the LINear-EXponential (LINEX) loss Varian (1975) formed from the log-likelihood of a Gumbel distribution. XQL is presented in more detail in Appendix B.4.

Gumbel Noise in Deep Q-Learning : Thrun and Schwartz (1993) argued that the max -operator in bootstrapped targets transformed zero-mean noise from function approximation into statistically biased noise. The authors did not recognize that the resultant noise distribution was Gumbel, and this was realized by Lee and Powell (2012). Unlike DoubleGum, these two works did not make assumptions about the distribution of function approximator noise but instead focused on mitigating the overestimation bias of bootstrapped targets. In economics, McFadden et al. (1973) and Rust (1994) assume the presence of Gumbel noise from noisy reward observations (unlike DoubleGum, which assumes that noise comes from function approximation) and derives the soft value function we present in Appendix A.1 for static and dynamic discrete choice models. XQL brings the Rust-McFadden et al. model to deep reinforcement learning to tackle continuous control.

The soft value function was introduced to model stochastic policies (Rummery and Niranjan, 1994; Fox et al., 2015; Haarnoja et al., 2017, 2018b). The most prominent algorithm that uses the soft value function is Soft Actor-Critic (SAC) (Haarnoja et al., 2018a,b), which Appendix B.2 shows is a special case of DoubleGum when Q new θ is learned by homoscedastic instead of heteroscedastic normal regression and the spread is a tuned scalar parameter instead of a learned state-dependent standard deviation. Appendix B.2 also shows that Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al., 2015) is a simpler special case of SAC that has recently been shown to outperform and train more stably than SAC (Yarats et al., 2021).

5.2 Empirically Similar Methods to DoubleGum

Distributional RL models the bootstrapped targets and Q -function as distributions. In these methods, a Q -function is learned by minimizing the divergence between itself and a target distribution. The most similar distributional RL method to DoubleGum is Mixture-of-Gaussian (MoG) Critics , introduced in Appendix A of Barth-Maron et al. (2018). DoubleGum is a special case of MoGCritics with only one Gaussian (such as in Morimura et al. (2012)) and mean samples of the target distribution. Curiously, Shahriari et al. (2022) shows that sample-efficiency of training improves when bootstrapped target sampled are increasingly near the mean but did not try mean sampling. Nevertheless, Shahriari et al. (2022) show that MoG-Critics outperforms a baseline with the C51 distributional head (Bellemare et al., 2017) popular in discrete control, obtaining state-of-the-art results in DeepMind Control. In discrete control with distributional RL, C51 has been superseded by Quantile Regression (QR) methods (Dabney et al., 2018b,a; Yang et al., 2019) that predict quantile estimates of a distribution. Ma et al. (2020); Wurman et al. (2022) and Teng et al. (2022) apply QR to continuous control.

Adjusting pessimism greatly improves the sample efficiency of RL. Fujimoto et al. (2018) showed empirical evidence of overestimation bias in continuous control and mitigated it with pessimistic estimates computed by Twin Networks . However, Twin Networks may sometimes harm sample efficiency because the optimal degree of pessimism varies across environments. To address this, the degree of pessimism is adjusted during training (Lan et al., 2020; Wang et al., 2021; Karimpanal et al., 2021; Moskovitz et al., 2021; Kuznetsov et al., 2020, 2022; Ball et al., 2023), often with the help of an ensemble. In contrast, DoubleGum uses one Q -network and one scalar hyperparameter fixed throughout training.

0

20

40

60

80

100

0

150

300

450

600

CartPole-v1

0

20

40

60

80

100

-

600

-

450

-

300

-

150

0

Acrobot-v1

0

20

40

60

80

100

-

210

-

180

-

150

-

120

-

90

-

60

MountainCar-v0

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in thousands)

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

IQM of Return

DoubleGum (Ours)

DQN

Dueling DDQN

Many other RL methods add or subtract the learned standard deviation to bootstrapped targets . Risk-Aware RL subtracts a learned standard deviation to learn a low-variance policy (La and Ghavamzadeh, 2013; Tamar and Mannor, 2013; Tamar et al., 2016). Upper-Confidence Bounded (UCB) methods add a learned standard deviation to explore high-variance regions (Lee et al., 2021; Teng et al., 2022). These methods use a combination of ensembles and variance networks, but it is also possible to derive a Bellman equation for the variance following Dearden et al. (1998). The current state-of-the-art method in RL with variance estimation is Mai et al. (2022), which uses both variance networks and ensembles. The use of ensembles is motivated by the need to capture model uncertainty - differences between Q -functions with different parameters. We believe that ensembles are unnecessary because model uncertainty will be expressed in bootstrapped targets from two different timesteps as parameters change through learning and that all variation in bootstrapped targets will henceforth be captured by variance networks.

6 Results

6.1 Discrete Control

We benchmarked DoubleGum on classic discrete control tasks against two baselines from the DQN family of algorithms (Figure 3). All algorithms were implemented following Section 3.4. Appendix E.3 discusses the baseline algorithms in more detail.

Performance was evaluated by the InterQuartile Mean (IQM) over 12 runs, each one with a different randomly initialized seed. Agarwal et al. (2021) showed that the IQM was a robust performance metric in RL. 12 was chosen because it was the smallest multiple of four (so a IQM could be computed) greater than the 10 seeds recommended by Henderson et al. (2018). In each run, the policy was evaluated by taking the mean of 10 rollouts every 1000 timesteps, following Fujimoto et al. (2018).

Figure 3 shows that DoubleGum sometimes obtains better sample efficiency than baselines, but not significantly more to necessitate further discrete control experiments. In the remainder of this work, we focus on continuous control, where we found DoubleGum to be more effective.

6.2 Continuous Control

We present two modes of evaluating DoubleGum because our algorithm has a pessimism factor hyperparameter we choose to tune per suite. We, therefore, benchmark all continuous control algorithms with default pessimism (Figure 4) and without pessimism-tuning per-suite (Figure 5). Table 5 presents default and per-suite pessimism values.

We compare against seven baselines. The first five are popular algorithms in the continuous control literature: DDPG (Lillicrap et al., 2015), TD3 (Fujimoto et al., 2018), SAC (Haarnoja et al., 2018b), MoG-Critics (Shahriari et al., 2022) and XQL (Garg et al., 2023). Here, DDPG, TD3, and SAC represent MaxEnt RL, MoG-Critics represent Distributional RL, and XQL is the algorithm with the most similar noise model to ours. We introduce two further baselines: QR-DDPG, a stronger Distributional RL baseline that combines quantile regression (Dabney et al., 2018b) with DDPG, and FinerTD3, TD3 with an ensemble of five networks as opposed to a network of two ensembles

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Untuned

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

IQM

DoubleGum,

c

=

-

0

.

1

(Ours)

MoG-DDPG

DDPG

TD3

Twin-SAC

Twin-XQL

QR-DDPG

FinerTD3

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Tuned per suite

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

IQM

DoubleGum, best

c

, (Ours)

MoG-DDPG (untuned)

Best of DDPG/TD3

SAC (best w/wo Twin)

XQL (best of

b

w/wo Twin)

QR-DDPG (best w/wo Twin)

FinerTD3 (best pessimism)

in the original TD3 that enables finer control over the degree of pessimism with an ensemble of five networks. All algorithms were implemented following design decisions outlined in Section 3.4. Appendix E.4 discusses the algorithms in more detail.

The pessimism of DoubleGum was tuned by changing its pessimism factor. The pessimism of baseline algorithms was tuned by manually choosing whether to use Twin Networks (Fujimoto et al., 2018) or not. Note that we refer to DDPG with Twin Networks as TD3. The pessimism of FinerTD3 was tuned by selecting which quantile estimate to use from an ensemble of five networks. The pessimism of MoG-Critics could not be tuned because its critic does not support paired sampling between ensemble members. Appendices F.1 and F.2 respectively detail how the pessimisms of DoubleGum and baseline algorithms were adjusted.

We benchmarked DoubleGum on 33 tasks over 4 continuous control suites comprising respectively of 11 DeepMind Control (DMC) tasks (Tassa et al., 2018; Tunyasuvunakool et al., 2020), 5 MuJoCo tasks (Todorov et al., 2012; Brockman et al., 2016), 15 MetaWorld tasks (Yu et al., 2020) and 2 Box2D tasks (Brockman et al., 2016). We follow Agarwal et al. (2021) and evaluate performance with the normalized IQM with 95% stratified bootstrap confidence intervals aggregated over all 33 tasks. 12 runs from each task was collected by a similar method to that described in Section 6.1. Further details of the tasks and their aggregate metric are detailed in Appendix E.2.

Figure 4 shows that DoubleGum outperformed all baselines in aggregate over 33 tasks when all algorithms used their default pessimism settings. Figure 5 shows that DoubleGum outperformed all baselines in aggregate over 33 tasks when the pessimism of all algorithms is adjusted per suite. Comparing the figures shows that adjusting the pessimism of DoubleGum per suite also attained a higher aggregate score than DoubleGum with default pessimism.

7 Discussion

This paper studied the noise distribution in Deep Q -Learning from first principles. We derived a noise model for Deep Q -Learning that used two heteroscedastic Gumbel distributions. Converting our noise model into an algorithm yielded DoubleGum, an off-policy Deep Q -Learning algorithm applied to both discrete and continuous control.

In discrete control, our algorithm attained competitive performance to the baselines. Despite having an numerically exact loss function in discrete control, DoubleGum was very sensitive to hyperparameters.

Practically, using Dueling DQN (Wang et al., 2016) to learn a Q -function was crucial to getting DoubleGum to work. Appendix D.1 shows that our noise model fits the underlying noise distribution of Deep Q -Learning and we therefore suspect that instability in discrete control might be due to the training dynamics of deep learning and not our theory.

In continuous control, we introduced a pessimism factor hyperparameter to approximate our otherwise intractable noise model. We provided a default value for the pessimism factor that outperformed popular Q -Learning baselines in aggregate over 33 tasks. Tuning this hyperparameter yielded even greater empirical gains. Our method of tuning pessimism was more computationally efficient and finer-grained than popular methods that tuned a quantile estimate from an ensemble.

In continuous control, DoubleGum outperformed all baselines in aggregate. We hypothesize that DoubleGum outperformed MaxEnt RL baselines because DoubleGum is a more expressive generalization of SAC, which is itself more expressive than DDPG as shown in Appendix B.2. Our theory showed that TD-errors follow a heteroscedastic Logistic, and we believe that modeling this distribution should be sufficient for distributional RL. We hypothesize that more complex distributions considered by the Distributional RL methods QR and MoG overfit to the replay buffer and might not generalize well to online rollouts. FinerTD3 performs marginally poorer than DoubleGum, even when pessimism was adjusted per-suite. We believe this is because FinerTD3 adjusts pessimism finer than other baselines, but still not as fine as the continuous scalar in DoubleGum. Finally, we hypothesize that DoubleGum outperformed XQL because our noise model better fits the underlying noise distribution in Deep Q -Learning, as shown in Appendix D.1.

This paper shows that better empirical performance in Deep RL may be attained through a better understanding of theory. To summarize, we hope that our work encourages the community to increase focus on reducing the gap between theory and practice to create reinforcement learning algorithms that train stably across a wide variety of environments.

7.1 Limitations

Theoretically, our work lacks a convergence guarantee for DoubleGum. This is exceptionally challenging because to the best of our knowledge there are currently no convergence guarantees for heteroscedastic regression. Appendix C discusses convergence in more detail.

Experimentally, there are many areas left open for future work. For speed in proof of concept experiments, we only swept over five values for the pessimism factor hyperparameter and only tuned per-suite. We anticipate that a more thorough search of the pessimism factor combined with a per-task selection will improve our results even further. An obvious follow-up would be an algorithm that automatically adjusted the pessimism factor during training. Additionally, we only focused on environments with state observations to not deal with representation learning from visual inputs. Another obvious next step would be to train DoubleGum on visual inputs or POMDPs.

References

Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems , 34:2930429320.

Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450 .

Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. (2023). Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02948 .

Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., Dhruva, T., Muldal, A., Heess, N., and Lillicrap, T. (2018). Distributed distributional deterministic policy gradients. In International Conference on Learning Representations .

Bas-Serrano, J., Curi, S., Krause, A., and Neu, G. (2021). Logistic q-learning. In International Conference on Artificial Intelligence and Statistics , pages 3610-3618. PMLR.

Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., and Wang, Z. (2020). Autonomous navigation of stratospheric balloons using reinforcement learning. Nature , 588(7836):7782.

Bellemare, M. G., Dabney, W., and Munos, R. (2017). A distributional perspective on reinforcement learning. In International conference on machine learning , pages 449-458. PMLR.

Bellman, R. (1957). A markovian decision process. Journal of mathematics and mechanics , pages 679-684.

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs.

Bradtke, S. J. and Barto, A. G. (1996). Linear least-squares algorithms for temporal difference learning. Machine learning , 22(1-3):33-57.

Bridle, J. (1989). Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. Advances in neural information processing systems , 2.

Bridle, J. S. (1990). Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing , pages 227-236. Springer.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym.

Ceron, J. S. O. and Castro, P. S. (2021). Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning , pages 1373-1383. PMLR.

Chen, X., Wang, C., Zhou, Z., and Ross, K. W. (2021). Randomized ensembled double q-learning: Learning fast without a model. In International Conference on Learning Representations .

Dabney, W., Ostrovski, G., Silver, D., and Munos, R. (2018a). Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning , pages 1096-1105. PMLR.

Dabney, W., Rowland, M., Bellemare, M., and Munos, R. (2018b). Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32.

Dearden, R., Friedman, N., and Russell, S. (1998). Bayesian q-learning. Aaai/iaai , 1998:761-768.

Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., et al. (2022). Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602(7897):414-419.

D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Bellemare, M. G., and Courville, A. (2022). Sampleefficient reinforcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022 .

Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., and Garcia, R. (2000). Incorporating second-order functional knowledge for better option pricing. Advances in neural information processing systems , 13.

Ernst, D., Geurts, P., and Wehenkel, L. (2005). Tree-based batch mode reinforcement learning. Journal of Machine Learning Research , 6.

Farama Foundation (2023). Farama-Foundation/Gymnasium. original-date: 2022-09-08T01:58:05Z.

Fisher, R. A. and Tippett, L. H. C. (1928). Limiting forms of the frequency distribution of the largest or smallest member of a sample. In Mathematical proceedings of the Cambridge philosophical society , volume 24, pages 180-190. Cambridge University Press.

Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., et al. (2017). Noisy networks for exploration. arXiv preprint arXiv:1706.10295 .

Fox, R., Pakman, A., and Tishby, N. (2015). Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1512.08562 .

Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning , pages 1587-1596. PMLR.

Garg, D., Hejna, J., Geist, M., and Ermon, S. (2023). Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328 .

Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 315-323. JMLR Workshop and Conference Proceedings.

Gnedenko, B. (1943). Sur la distribution limite du terme maximum d'une serie aleatoire. Annals of mathematics , pages 423-453.

Gumbel, E. J. (1935). Les valeurs extrêmes des distributions statistiques. In Annales de l'institut Henri Poincaré , volume 5, pages 115-158.

Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. (2017). Reinforcement learning with deep energy-based policies. In International conference on machine learning , pages 1352-1361. PMLR.

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018a). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR.

Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018b). Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 .

Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. (2018). Deep reinforcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32.

Hennigan, T., Cai, T., Norman, T., and Babuschkin, I. (2020). Haiku: Sonnet for JAX.

Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. (2018). Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence .

Hinton, G., Srivastava, N., and Swersky, K. (2012). Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on , 14(8):2.

Hoffman, M. W., Shahriari, B., Aslanides, J., Barth-Maron, G., Momchev, N., Sinopalnikov, D., Sta´ nczyk, P., Ramos, S., Raichuk, A., Vincent, D., Hussenot, L., Dadashi, R., Dulac-Arnold, G., Orsini, M., Jacq, A., Ferret, J., Vieillard, N., Ghasemipour, S. K. S., Girgin, S., Pietquin, O., Behbahani, F., Norman, T., Abdolmaleki, A., Cassirer, A., Yang, F., Baumli, K., Henderson, S., Friesen, A., Haroun, R., Novikov, A., Colmenarejo, S. G., Cabi, S., Gulcehre, C., Paine, T. L., Srinivasan, S., Cowie, A., Wang, Z., Piot, B., and de Freitas, N. (2020). Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979 .

Howard, R. A. (1960). Dynamic programming and markov processes. John Wiley.

Huang, S., Dossa, R. F. J., Raffin, A., Kanervisto, A., and Wang, W. (2022a). The 37 implementation details of proximal policy optimization. In ICLR Blog Track . https://iclr-blog-track.github.io/2022/03/25/ppoimplementation-details/.

Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Araújo, J. G. (2022b). Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research , 23(274):1-18.

Huber, P. J. (1992). Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution , pages 492-518. Springer.

Irpan, A. (2018). Deep reinforcement learning doesn't work yet. https://www.alexirpan.com/2018/02/ 14/rl-hard.html .

Jowitt, P. (1979). The extreme-value type-1 distribution and the principle of maximum entropy. Journal of Hydrology , 42(1-2):23-38.

Karimpanal, T. G., Le, H., Abdolshah, M., Rana, S., Gupta, S., Tran, T., and Venkatesh, S. (2021). Balanced q-learning: Combining the influence of optimistic and pessimistic targets. arXiv preprint arXiv:2111.02787 .

Kendall, A. and Gal, Y. (2017). What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems , 30.

Kimball, B. F. (1946). Sufficient statistical estimation functions for the parameters of the distribution of maximum values. The Annals of Mathematical Statistics , 17(3):299-309.

Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .

Kostrikov, I. (2021). JAXRL: Implementations of Reinforcement Learning algorithms in JAX.

Kuznetsov, A., Grishin, A., Tsypin, A., Ashukha, A., Kadurin, A., and Vetrov, D. (2022). Automating control of overestimation bias for reinforcement learning.

Kuznetsov, A., Shvechikov, P., Grishin, A., and Vetrov, D. (2020). Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning , pages 5556-5566. PMLR.

La, P. and Ghavamzadeh, M. (2013). Actor-critic algorithms for risk-sensitive mdps. Advances in neural information processing systems , 26.

Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. (2019). Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 .

Lan, Q., Pan, Y., Fyshe, A., and White, M. (2020). Maxmin q-learning: Controlling the estimation bias of q-learning. arXiv preprint arXiv:2002.06487 .

Laskin, M., Yarats, D., Liu, H., Lee, K., Zhan, A., Lu, K., Cang, C., Pinto, L., and Abbeel, P. (2021). Urlb: Unsupervised reinforcement learning benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) .

Lee, D. and Powell, W. (2012). An intelligent battery controller using bias-corrected q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 26, pages 316-322.

Lee, K., Laskin, M., Srinivas, A., and Abbeel, P. (2021). Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning , pages 6131-6141. PMLR.

Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 .

Lin, L.-J. (1992). Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning , 8(3):293-321.

Ma, X., Xia, L., Zhou, Z., Yang, J., and Zhao, Q. (2020). Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004.14547 .

Mai, V., Mani, K., and Paull, L. (2022). Sample efficient deep reinforcement learning via uncertainty estimation. arXiv preprint arXiv:2201.01666 .

McFadden, D. et al. (1973). Conditional logit analysis of qualitative choice behavior . Institute of Urban and Regional Development, University of California.

Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR.

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. nature , 518(7540):529-533.

Morimura, T., Sugiyama, M., Kashima, H., Hachiya, H., and Tanaka, T. (2012). Parametric return density estimation for reinforcement learning. arXiv preprint arXiv:1203.3497 .

Moskovitz, T., Parker-Holder, J., Pacchiano, A., Arbel, M., and Jordan, M. (2021). Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems , 34:1284912863.

Munos, R. and Szepesvári, C. (2008). Finite-time bounds for fitted value iteration. J. Mach. Learn. Res. , 9:815-857.

Nix, D. A. and Weigend, A. S. (1994). Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 ieee international conference on neural networks (ICNN'94) , volume 1, pages 55-60. IEEE.

Peters, J., Mulling, K., and Altun, Y. (2010). Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 24, pages 1607-1612.

Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons.

Riedmiller, M. (2005). Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method. In European conference on machine learning , pages 317-328. Springer.

Rummery, G. A. and Niranjan, M. (1994). On-line Q-learning using connectionist systems , volume 37. University of Cambridge, Department of Engineering Cambridge, UK.

Rust, J. (1994). Structural estimation of markov decision processes. Handbook of econometrics , 4:3081-3143.

Samuelson, P. A. (1937). A note on measurement of utility. The review of economic studies , 4(2):155-161.

Saxe, A. M., McClelland, J. L., and Ganguli, S. (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 .

Schaul, T., Barreto, A., Quan, J., and Ostrovski, G. (2022). The phenomenon of policy churn. In Advances in Neural Information Processing Systems .

Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized experience replay. In Bengio, Y. and LeCun, Y., editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings .

Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020). Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609.

Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015). Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR.

Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S. (2023). Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning , pages 30365-30380. PMLR.

Seitzer, M., Tavakoli, A., Antic, D., and Martius, G. (2022). On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. arXiv preprint arXiv:2203.09168 .

Seyde, T., Werner, P., Schwarting, W., Gilitschenski, I., Riedmiller, M., Rus, D., and Wulfmeier, M. (2022). Solving continuous control via q-learning. arXiv preprint arXiv:2210.12566 .

Shahriari, B., Abdolmaleki, A., Byravan, A., Friesen, A., Liu, S., Springenberg, J. T., Heess, N., Hoffman, M., and Riedmiller, M. (2022). Revisiting gaussian mixture critic in off-policy reinforcement learning: a sample-based approach. arXiv preprint arXiv:2204.10256 .

Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489.

Sutton, R. S. and Barto, A. G. (1998). Reinforcement learning: An introduction . MIT press.

Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction . The MIT Press, second edition.

Tamar, A., Di Castro, D., and Mannor, S. (2016). Learning the variance of the reward-to-go. The Journal of Machine Learning Research , 17(1):361-396.

Tamar, A. and Mannor, S. (2013). Variance adjusted actor critic algorithms. arXiv preprint arXiv:1310.3697 .

Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018). Deepmind control suite. arXiv preprint arXiv:1801.00690 .

Teng, M., van de Panne, M., and Wood, F. (2022). Exploration with multi-sample target values for distributional reinforcement learning. arXiv preprint arXiv:2202.02693 .

Thrun, S. and Schwartz, A. (1993). Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum , volume 6, pages 1-9.

Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE.

Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., and Tassa, Y. (2020). dm_control: Software and tasks for continuous control. Software Impacts , 6:100022.

Van Hasselt, H., Guez, A., and Silver, D. (2016). Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence , volume 30.

Varian, H. R. (1975). A bayesian approach to real estate assessment. Studies in Bayesian econometric and statistics in Honor of Leonard J. Savage , pages 195-208.

Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019). Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354.

Wang, H., Lin, S., and Zhang, J. (2021). Adaptive ensemble q-learning: Minimizing estimation bias via error feedback. Advances in Neural Information Processing Systems , 34:24778-24790.

Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. (2016). Dueling network architectures for deep reinforcement learning. In International conference on machine learning , pages 1995-2003. PMLR.

Watkins, C. J. and Dayan, P. (1992). Q-learning. Machine learning , 8:279-292.

Wu, Y. (2021). td3-jax. https://github.com/yifan12wu/td3-jax .

Wu, Y. and He, K. (2018). Group normalization. In Proceedings of the European conference on computer vision (ECCV) , pages 3-19.

Wurman, P. R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T. J., Capobianco, R., Devlic, A., Eckert, F., Fuchs, F., et al. (2022). Outracing champion gran turismo drivers with deep reinforcement learning. Nature , 602(7896):223-228.

Xu, J., Sun, X., Zhang, Z., Zhao, G., and Lin, J. (2019). Understanding and improving layer normalization. Advances in Neural Information Processing Systems , 32.

Yamada, H. (2019). cpprb.

Yang, D., Zhao, L., Lin, Z., Qin, T., Bian, J., and Liu, T.-Y . (2019). Fully parameterized quantile function for distributional reinforcement learning. Advances in neural information processing systems , 32.

Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. (2021). Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645 .

Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. (2020). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning , pages 1094-1100. PMLR.

Zhang, Y., Lin, J., Li, F., Adler, Y., Rasul, K., Schneider, A., and Nevmyvaka, Y. (2023). Risk bounds on aleatoric uncertainty recovery. In International Conference on Artificial Intelligence and Statistics , pages 6015-6036. PMLR.

Appendix

Table of Contents

A Proofs

A.1 The SoftMax Location of Gumbel Distribution

We refer to the log -sumexp operator as the SoftMax operator. This is not the same-named operator in Bridle (1989, 1990), which we suggest should be (re-)named SoftArgMax.

Theorem 1.

where G is a Gumbel distribution. A Gumbel random variable g ∼ G ( α, β ) specified by location α ∈ R and spread β > 0 has PDF p ( g ) = 1 β exp( -z -exp( -z )) with z = ( g -α ) /β and CDF P ( g ) = 1 β exp( -exp( -z )) (Gumbel, 1935).

Proof. First note

Then, denoting y = max i [ α i + g i ]

When applied to discrete Q -Learning, we produce

where g θ,a ( · ) , g θ ( · ) ∼ G (0 , β θ ( · )) , for all a ∈ A .

We assume that the same result holds for the continuous case if |A| < ∞ , an assumption first used in Lemma 1, Appendix B.1, Page 11 of Haarnoja et al. (2018a) to ensure boundedness. Here, we require the output of the max -operator to be bounded, which cannot be the case when the number of its arguments is ∞ .

A.2 Shifting the Value Function

Theorem 2.

Proof.

A.3 The Difference between Two Gumbel Random Variables is a Logistic Theorem 3.

where L is a logistic distribution. A logistic random variable l ∼ L ( α, β ) with location α and spread β has PDF exp ( -( l -α ) /β ) β (1+exp( -( l -α ) /β )) 2 and CDF 1 1+exp( -( l -α ) /β ) .

Proof. First, construct the convolution based on the joint PDF

Rewriting x 2 as x yields

which is the CDF of L (0 , β ) .

A.4 Soft Q-Learning Identity

Theorem 4. For an arbitrary p ( x )

Proof.

When applied to Q -Learning, the following identity produces

B Further Theory and Derivations

B.1 Actor Loss

The actor losses used in DoubleGum, SAC, and DDPG are all derived from the same principle. For a given s , the actor loss function should minimizes the following (reverse) KL-Divergence, previously presented in Equation 7.

This simplifies as

which is then estimated by Monte-Carlo samples from π ϕ as

SAC (Haarnoja et al., 2018a,b) has a policy with learned variance and state-independent β . DDPG (Lillicrap et al., 2015) has a fixed-variance policy which removes the second term in Equation 15 as it is constant with respect to the maximization. DoubleGum has a state-dependent β ( s ) , but uses the same actor loss as DDPG because DoubleGum uses a DDPG fixed-variance policy.

B.2 Maximum-Entropy Reinforcement Learning

SACv1 (Haarnoja et al., 2018b) is a special case of DoubleGum and DDPG (Lillicrap et al., 2015) is a special case of SAC. All three continuous control algorithms have an actor and critic loss derived from the same principle. Section B.1 shows this for the actor losses of DoubleGum, SAC, and DDPG. We now relate the critic losses to each other, starting from the most general case, DoubleGum. In continuous control, DoubleGum uses the following noise model, formed from substituting Equation 7 into Equation 5:

Here, l θ,a ( · ) ∼ L (0 , β θ ( · )) is a logistic distribution and p θ ( a | s ) ∝ exp Q new θ ( s,a ) β θ ( s ) . The DoubleGum critic loss is derived from this noise model by approximating the RHS with Equation 8 and learning θ with moment matching in Section 3.2.

The SAC noise model is derived from Equation 16 in three ways. First, SAC approximates l θ,a ( s ) ∼ L (0 , β θ ( · )) as n θ,a ∼ N (0 , σ ) , motivated by the fact that both distributions have the same mean/mode. Secondly, SAC approximates the DoubleGum state-dependent logistic spread β θ ( s ) as temperature parameter β learned not as a part of the critic but by itself with Lagrangian dual gradient descent. Thirdly, SAC breaks down C [ π ϕ || p θ ] = H [ π ϕ ] + D KL [ π ϕ || p θ ] before assuming that the KLDivergence is negligible, given that it is minimized by the actor loss. These three approximations yield the SAC noise model as

MLE of θ wrt the above noise model yields the MSBE critic loss.

DDPG is a special case of SAC that assumes β → 0 , removing the last term in Equation 17. lim β → 0 p θ ( a | s ) becomes deterministic, so π ϕ may be modelled by a deterministic policy.

B.3 Interpreting the Cross-Entropy as a Pessimism Factor

In continuous control, Fujimoto et al. (2018) introduced Twin Networks, a method that improved sample-efficiency with pessimistic bootstrapped targets computed by returning a sample-wise minimum from an ensemble of two Q -functions. Follow-up work selects a quantile estimate from an ensemble (Kuznetsov et al., 2020; Chen et al., 2021; Ball et al., 2023), which we demonstrate is equivalent to estimating V soft, new θ,β .

Suppose there is an ensemble of n networks where the i th network follows Q θ i ( s, a ) = Q θ ( s, a ) + z i ( s, a ) . Here, Q θ is an 'ideal' function approximator never instantiated nor computed and z is an arbitrary noise source. When n is sufficiently large,

A Gumbel random variable g ∼ G ( α, β ) has E [ g ] = α + γ e β , where γ e is the Euler-Mascheroni constant, so for a deterministic environment the bootstrapped targets become

recovering Equation 8, the DoubleGum continuous control targets, up to an additive term γα ( s ′ ) , while -γγ e β ( s ′ ) recovers the spread γcσ θ ( s ′ ) up to a negative scaling factor, indicating that the default c should be negative. Moskovitz et al. (2021) and Ball et al. (2023) showed that the appropriate ensemble size and selected quantile changes the overestimation bias, so appropriate values would ensure α ( s ′ ) = 0 .

B.4 Comparison between DoubleGum and XQL

We present an explanation of Extreme Q -Learning (XQL) as presented in Appendix C.1 of Garg et al. (2023). XQL can be derived from Soft Bellman Equation backups given by

and β is a fixed hyperparameter. Computing the log-sum-exp of V soft is intractable in continuous control, as the sum over a ′ becomes an integral in continuous control tasks.

Garg et al. (2023) present a method of estimating its value using Gumbel regression. Given a (potentially infinite) set of scalars x ∈ X , Gumbel regression provides a method to estimate the numerical value of log-sum-exp β ( x ) = β log ∑ X exp x/β . Gumbel regression assumes x ∼ G ( α, β ) , where G is a homoscedastic Gumbel distribution, and β is a fixed (hyper)parameter. α estimated by MLE tends towards log-sum-exp β ( x ) . MLE is performed by numerically maximizing the log -likelihood of a Gumbel distribution, which recovers the LINear-EXponential (LINEX) loss function introduced by Varian (1975).

Garg et al. (2023) incorporate Gumbel regression into deep Q -Learning in two ways, which they name X-SAC and X-TD3. X-SAC combines Gumbel regression to estimate the soft value function used in SACv0 (Haarnoja et al., 2018a). The soft value function V soft ρ ( s ) is a neural network whose parameters ρ are learned by Gumbel regression from Q ψ ( s, a ) ∼ G ( V ρ ( s ) , β ) , where ψ are target network parameters. A neural network Q θ with parameters θ may then be learned by the MSE between itself and E p ( s ′ | s,a ) [ r ( s, a, s ′ ) + γV soft ρ ( s )] . X-SAC is vastly different from DoubleGum, because our algorithm does not estimate the soft value function with a separate neural network.

Gumbel regression is directly used to learn the Q -values in X-TD3. First, the bootstrapped targets are thusly rewritten

In environments with deterministic environments, which comprise all environments considered by Garg et al. (2023) and our paper, Lemma C.1 of Garg et al. (2023) provides a method of learning the soft value function with Gumbel regression on y soft ( s, a ) ∼ G ( Q θ ( s, a ) , γβ ) . The Gumbel regression objective used in X-TD3 to learn θ is vastly different from the moment matching with the logistic distribution DoubleGum uses to learn θ .

To motivate their use of Gumbel regression, Garg et al. (2023) derived a noise model which they use to present empirical evidence of homoscedastic Gumbel noise. In contrast, we presented empirical evidence of heteroscedastic logistic noise formed from a noise model with two heteroscedastic Gumbel distributions.

C A Discussion on The Convergence of DoubleGum

To the best of our knowledge, there are two types of convergence analysis in Q -Learning: 1) operatortheoretic analysis over tabular Q -functions, and 2) training dynamics of neural network parameters. We believe the second is more appropriate for DoubleGum, because our theory addresses issues in using neural networks (and not tables) for Q -learning. Nevertheless, for completeness, we discuss convergence guarantees for the tabular setting and the function approximation setting. While we can guarantee convergence for the former setting, we have no guarantees for the second.

C.1 Tabular Q-Functions

Appendices B.1 and B.2 present DoubleGum as a MaxEnt RL algorithm. When Q -functions are tabular, Appendix A of Haarnoja et al. (2018a) shows that MaxEnt RL algorithms may be derived

Algorithm 2: DoubleGum Soft Policy Iteration

from soft policy iteration. We therefore present a convergence proof for DoubleGum with tabular Q -functions based on soft policy iteration.

DoubleGum treats the return as coming from a logistic distribution and learns its location and spread. In the tabular setting, two tables would need to be learned, Q ( s, a ) and β ( s ) . An algorithm to learn these tables in a finite MDP with soft policy iteration is presented in Algorithm 2. Policy evaluation is done by Lines 4-6 while Line 7 performs policy improvement.

Proof of convergence of Algorithm 2 is similar to the SAC proof of convergence in Appendix B of Haarnoja et al. (2018a). This should not be surprising, given that Appendix B.2 shows SAC as a special case of DoubleGum. We first show that policy evaluation converges and that a new policy found by policy improvement does not reduce the magnitude of the value function.

Lemma 5 (Soft Policy Evaluation) . Consider the Soft Policy Evaluation operator given by

lim i →∞ Q i converges to the soft Q -value.

Proof. Following Appendix A.4

the bootstrapped targets may be thusly rewritten

where r ′ ( s, a, s ′ ) = r ( s, a, s ′ ) + β ( s ′ ) C [ π || p ] .

The proof of Soft Policy Improvement should be identical to SAC, given that Appendix B.1 shows that DoubleGum and SAC use identical actor losses. As such, Lemma 5 can be used in place of Lemma 1 in Theorem 1 of Haarnoja et al. (2018a), thus showing convergence of DoubleGum in the tabular setting.

C.2 Deep Q-Functions

Parameters of the deep Q -function used by DoubleGum in Algorithm 1 are learned by a loss function equivalent to that of heteroscedastic normal regression. Convergence of DoubleGum in the function approximation setting would therefore rely on convergence of heteroscedastic normal regression.

Zhang et al. (2023) introduces PAC-bounds for heteroscedastic normal regression, but on the condition that the mean-estimate is close to the ground truth mean, as mentioned in Paragraph 1 of Section 4. This is empirically achieved by Seitzer et al. (2022), who analyze heteroscedastic normal regression and find that the mean-estimate frequently converges to an underfitting solution. This is because the Negative Log-Likelihood (NLL) of a normal distribution is minimized when the variance becomes large - in Equation 6, this term is denoted with σ 2 θ . As such, changes in Q new θ will not change the loss function much. To rectify this, Seitzer et al. (2022) multiplies the NLL of the normal with the numerical value of the standard deviation, reducing the dominance of σ θ on the loss function.

D Further Empirical Evidence for Theoretical Assumptions

D.1 Noise Distributions in Deep Q-Learning

Figure 6 presents graphs corresponding to Figure 1c for all environments considered in this paper. Continuous control results were generated from DoubleGum with default pessimism ( c = -0 . 1 ).

D.2 Adjusting The Pessimism Factor

Figure 7 presents graphs corresponding to Figure 2 for all continuous control environments considered in this paper.

E Further Experimental Details

E.1 Noise Distribution Discrepancy with Extreme Q-Learning

In Appendix D.2 of Page 19, Garg et al. (2023) fitted a Gumbel distribution to the TD errors on three continuous control environments. The Gumbel distribution was a good fit in two of the three environments they benchmarked on. We could not reproduce this result and attribute the discrepancy to experimental differences.

Garg et al. (2023) logged their batch of 256 TD errors once every 5,000 steps during training for 100,000 timesteps, producing ≈ 4000 samples which were aggregated. They also computed bootstrapped targets with online parameters. In contrast, we sample 10,000 TD errors with bootstrapped targets computed from target parameters at a single timestep instance, and we do not aggregate samples across timesteps.

E.2 Continuous Control Benchmarks and Evaluation

As mentioned in Section 6.2, the evaluation metric in continuous control was the normalized IQM with 95% stratified bootstrap confidence intervals from Agarwal et al. (2021). Returns were normalized by a minimum value computed from the mean of 100 rollouts sampled from a uniform policy and the maximum possible return from the environment. When the maximum value was not specified, we used the maximum value of any single rollout attained by any of the baselines.

We benchmarked on four continuous control suites: DeepMind Control (Tassa et al., 2018; Tunyasuvunakool et al., 2020), MuJoCo (Todorov et al., 2012; Brockman et al., 2016), MetaWorld (Yu et al., 2020), and Box2D (Brockman et al., 2016). These environments were selected to be as extensive as possible. DeepMind Control and MetaWorld were chosen because of their diversity of tasks, while the MuJoCo and Box2D environments are popular benchmarks within the common interface of OpenAI Gym (Brockman et al., 2016), now Gymnasium (Farama Foundation, 2023). No citation exists for Gymnasium as of writing this paper, and we link to their GitHub repository https://github.com/Farama-Foundation/Gymnasium as suggested in https://github.com/Farama-Foundation/Gymnasium/issues/82 .

0

.

0

0

.

5

1

.

0

-

0

.

8

0

.

0

0

.

8

1

.

6

2

.

4

CartPole-v1

0

.

0

0

.

5

1

.

0

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

Acrobot-v1

0

.

0

0

.

5

1

.

0

-

1

.

5

0

.

0

1

.

5

3

.

0

MountainCar-v0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

NLL

(a) Discrete Control

0

.

0

0

.

5

1

.

0

1

.

2

1

.

5

1

.

8

2

.

1

2

.

4

2

.

7

acrobot-swingup

0

.

0

0

.

5

1

.

0

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

reacher-hard

0

.

0

0

.

5

1

.

0

1

.

2

1

.

5

1

.

8

2

.

1

2

.

4

2

.

7

finger-turn hard

0

.

0

0

.

5

1

.

0

1

.

2

1

.

5

1

.

8

2

.

1

2

.

4

2

.

7

hopper-hop

0

.

0

0

.

5

1

.

0

1

.

3

1

.

4

1

.

5

1

.

6

1

.

7

fish-swim

0

.

0

0

.

5

1

.

0

1

.

4

1

.

6

1

.

8

2

.

0

cheetah-run

0

.

0

0

.

5

1

.

0

1

.

35

1

.

50

1

.

65

1

.

80

walker-run

0

.

0

0

.

5

1

.

0

1

.

3

1

.

4

1

.

5

1

.

6

1

.

7

1

.

8

quadruped-run

0

.

0

0

.

5

1

.

0

1

.

35

1

.

50

1

.

65

1

.

80

swimmer-swimmer15

0

.

0

0

.

5

1

.

0

1

.

36

1

.

44

1

.

52

1

.

60

1

.

68

1

.

76

humanoid-run

0

.

0

0

.

5

1

.

0

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

2

.

4

dog-run

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

NLL

(b) DeepMind Control

0

.

0

0

.

5

1

.

0

1

.

2

1

.

6

2

.

0

2

.

4

2

.

8

Hopper-v4

0

.

0

0

.

5

1

.

0

1

.

36

1

.

44

1

.

52

1

.

60

1

.

68

1

.

76

HalfCheetah-v4

0

.

0

0

.

5

1

.

0

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

Walker2d-v4

0

.

0

0

.

5

1

.

0

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

Ant-v4

0

.

0

0

.

5

1

.

0

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

Humanoid-v4

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

NLL

(c) MuJoCo

0

.

0

0

.

5

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

2

.

0

button-press-v2

0

.

0

0

.

5

1

.

0

1

.

2

1

.

6

2

.

0

2

.

4

door-open-v2

0

.

0

0

.

5

1

.

0

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

drawer-close-v2

0

.

0

0

.

5

1

.

0

1

.

2

1

.

5

1

.

8

2

.

1

2

.

4

drawer-open-v2

0

.

0

0

.

5

1

.

0

1

.

00

1

.

25

1

.

50

1

.

75

2

.

00

2

.

25

peg-insert-side-v2

0

.

0

0

.

5

1

.

0

1

.

2

1

.

5

1

.

8

2

.

1

2

.

4

2

.

7

pick-place-v2

0

.

0

0

.

5

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

push-v2

0

.

0

0

.

5

1

.

0

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

reach-v2

0

.

0

0

.

5

1

.

0

1

.

2

1

.

5

1

.

8

2

.

1

2

.

4

window-open-v2

0

.

0

0

.

5

1

.

0

1

.

2

1

.

5

1

.

8

2

.

1

2

.

4

window-close-v2

0

.

0

0

.

5

1

.

0

1

.

25

1

.

50

1

.

75

2

.

00

2

.

25

2

.

50

basketball-v2

0

.

0

0

.

5

1

.

0

1

.

25

1

.

50

1

.

75

2

.

00

2

.

25

dial-turn-v2

0

.

0

0

.

5

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

sweep-into-v2

0

.

0

0

.

5

1

.

0

1

.

25

1

.

50

1

.

75

2

.

00

2

.

25

hammer-v2

0

.

0

0

.

5

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

2

.

0

2

.

2

assembly-v2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

NLL

(d) MetaWorld

0

.

0

0

.

5

1

.

0

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

BipedalWalker-v3

0

.

0

0

.

5

1

.

0

1

.

2

1

.

6

2

.

0

2

.

4

2

.

8

3

.

2

BipedalWalkerHardcore-v3

Homo-Normal

Hetero-Logistic (expected)

Moment-Matched Hetero-Normal

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

NLL

(e) Box2D

0

.

0

0

.

5

1

.

0

0

300

600

900

1200

1500

acrobot-swingup

0

.

0

0

.

5

1

.

0

0

25

50

75

100

reacher-hard

0

.

0

0

.

5

1

.

0

0

25

50

75

100

finger-turn hard

0

.

0

0

.

5

1

.

0

0

10

20

30

40

50

hopper-hop

0

.

0

0

.

5

1

.

0

0

15

30

45

60

75

fish-swim

0

.

0

0

.

5

1

.

0

0

20

40

60

80

cheetah-run

0

.

0

0

.

5

1

.

0

0

20

40

60

80

walker-run

0

.

0

0

.

5

1

.

0

20

40

60

80

100

quadruped-run

0

.

0

0

.

5

1

.

0

30

45

60

75

swimmer-swimmer15

0

.

0

0

.

5

1

.

0

0

10

20

30

40

humanoid-run

0

.

0

0

.

5

1

.

0

0

60

120

180

240

300

dog-run

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Magnitude

(a) DeepMind Control

0

.

0

0

.

5

1

.

0

0

300

600

900

1200

1500

Hopper-v4

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

HalfCheetah-v4

0

.

0

0

.

5

1

.

0

0

800

1600

2400

3200

4000

Walker2d-v4

0

.

0

0

.

5

1

.

0

0

100

200

300

400

500

Ant-v4

0

.

0

0

.

5

1

.

0

400

800

1200

1600

Humanoid-v4

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Magnitude

(b) MuJoCo

0

.

0

0

.

5

1

.

0

0

400

800

1200

1600

button-press-v2

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

1250

door-open-v2

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

1250

drawer-close-v2

0

.

0

0

.

5

1

.

0

200

400

600

800

1000

drawer-open-v2

0

.

0

0

.

5

1

.

0

0

300

600

900

1200

peg-insert-side-v2

0

.

0

0

.

5

1

.

0

-

250

0

250

500

750

1000

pick-place-v2

0

.

0

0

.

5

1

.

0

0

300

600

900

1200

push-v2

0

.

0

0

.

5

1

.

0

200

400

600

800

1000

reach-v2

0

.

0

0

.

5

1

.

0

0

400

800

1200

1600

2000

window-open-v2

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

1250

window-close-v2

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

basketball-v2

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

1250

dial-turn-v2

0

.

0

0

.

5

1

.

0

0

300

600

900

1200

sweep-into-v2

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

hammer-v2

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

1250

assembly-v2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Magnitude

(c) MetaWorld

0

.

0

0

.

5

1

.

0

0

150

300

450

BipedalWalker-v3

0

.

0

0

.

5

1

.

0

-

300

0

300

600

900

BipedalWalkerHardcore-v3

c

=

-

0

.

5

c

=

-

0

.

1

(Default)

c

=

0

.

0

c

=

0

.

1

c

=

0

.

5

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Magnitude

(d) Box2D

DeepMind Control (DMC) was designed to benchmark continuous control, over a broad range of agent morphologies. We selected agent morphologies that could be trained from states with a broad range of action spaces from 1 ( acrobot ) to 38 ( dog ). We did not benchmark on humanoid_CMU as this environment was not intended to be solved with RL from scratch, unlike the other baselines. The hardest task was selected from each of the agent morphologies. Properties of the 11 DMC tasks are presented in Table 1a.

MetaWorld was designed to have a diverse range of tasks to evaluate the generalization ability of learned policies. Each environment within MetaWorld is therefore made up of multiple tasks, all with the same underlying structure of an MDP but with different numerical values of their parameters. We follow the method of Seyde et al. (2022) to benchmark on a single MetaWorld task by first selecting an environment and then randomly selecting a set of numerical parameters. Each new instantiation of a MetaWorld task would result in a different set of hyperparameters. As such, we expect the error bars in the aggregate statistics of MetaWorld to be substantially larger than the other environments. We benchmark on tasks formed from the union of the ML1, MT10, and ML10 train tasks that a policy in MetaWorld would be trained on, as well as the five environments benchmarked in Seyde et al. (2022). Properties of the 15 MetaWorld tasks are presented in Table 1c.

MuJoCo was evaluated on the same tasks as SAC (Haarnoja et al., 2018b). These tasks were all locomotion-based. Properties of the 5 MuJoCo tasks are presented in Table 1b.

Box2D was evaluated on all continuous control tasks from states. Properties of the 2 Box2D tasks are presented in 1d.

E.3 Discrete Control Baselines

Discrete control algorithms were implemented as described in Section 3.4. Hyperparameters used in discrete control algorithms are detailed in Tables 2 and 3. We provide explanations for these design choices as follows.

DQN : The original DQN algorithm in Mnih et al. (2015) was designed for pixel inputs. We modified DQN to use state inputs by using an architecture described in Section 3.4 we used in continuous control that was popular for use with state inputs. Conversely to the continuous control architecture, we found removing GroupNorm (Wu and He, 2018) was crucial to getting DQN to work. Similarly to the continuous control architecture, we found that changing the initialization and target network updating drastically improved performance. We also used the MSE and Adam (Kingma and Ba, 2014) optimizers as Ceron and Castro (2021) showed that this yielded improved performance over the Huber Loss (Huber, 1992) and RMSProp (Hinton et al., 2012) of the original DQN. Our implementation of DQN solves classic discrete control tasks that the CleanRL (Huang et al., 2022b) reproduction of the original DQN paper at https://docs.cleanrl.dev/rl-algorithms/dqn/ #experiment-results_1 could not solve.

Dueling Double DQN (Dueling DDQN) was a baseline modified from Hessel et al. (2018) designed to be as compatible with DoubleGum as possible. Rainbow evaluated six innovations to DQN: Double DQN (Van Hasselt et al., 2016), Dueling DQN (Wang et al., 2016), noisy networks (Fortunato et al., 2017), n -step returns, C51 distributional RL (Bellemare et al., 2017), and prioritized replay (Schaul et al., 2016). We only used the first two of these six innovations in DoubleGum. We did not find n -step returns effective in discrete domains we considered, nor prioritized replay. Distributional RL was incompatible with DoubleGum, while Schwarzer et al. (2023) did not find noisy networks advantageous.

DoubleDQN was implemented following Van Hasselt et al. (2016) by computing bootstrapped targets of Q new ψ ( s, max a Q new θ ( s, a )) . Dueling DQN was implemented following Wang et al. (2016), with the advantage and value heads having two layers with a hidden layer of size 256 and ReLU activations. The stability of Dueling DQN was greatly improved by setting the biases of both dueling heads to 0.

DoubleGum was implemented as Dueling DDQN with an additional variance head described in Section 3.4.

E.4 Continuous Control Baselines

Continuous control algorithms were implemented as described in Section 3.4. Hyperparameters used in continuous control algorithms are detailed in Tables 2 and 4. Pessimism hyperparameters are presented in Table 5 and were found following results in Appendix F.2.

As mentioned, all implementations used networks with two hidden layers of width 256, with orthogonal initialization (Saxe et al., 2013) and GroupNorm (Wu and He, 2018). Following Kostrikov (2021), target network parameters were updated with an EMA of 5 e -3 in the critic and 0 in the actor. All these design choices differ from their original implementations but improved aggregate performance. We provide explanations for these design choices as follows.

DDPG was introduced in Lillicrap et al. (2015) and Fujimoto et al. (2018) updated the design choices of DDPG to empirically improve its performance. In addition to the existing changes, our implementation uses the noise clipping scheme in the actor specified by Laskin et al. (2021).

TD3 was implemented with three changes from Fujimoto et al. (2018). First, we update the actor once per critic update - ie using a delay of 1. This is such that the only hyperparameter change between our DDPG and TD3 is the use of Twin Networks. Secondly, we update the actor to maximize the mean of two critics rather than a single critic, a design choice we found empirically reduced variance between training runs. Thirdly, we do not compute the EMA of actor-network parameters. Removing this EMA improves sample efficiency but at the cost of higher variance.

FinerTD3 (our introduced baseline) was implemented with the same hyperparameters as TD3 but with an ensemble of 5 critic networks. We chose to use 5 networks because we tuned the pessimism factor hyperparameter of DoubleGum over 5 values. The 5 critics in FinerTD3 enable five values of pessimism to be used. Pessimism of FinerTD3 is adjusted in the bootstrapped targets. The 5 critic values are sorted by decreasing positivity, and the i th smallest value is used as the target critic value in the bootstrapped targets.

SAC was implemented with hyperparameters from Kostrikov (2021), which we found improved performance. Kostrikov (2021) differs from Haarnoja et al. (2018b) in two additional ways from those mentioned. The standard deviation in the actor was clipped to [ -10 , 2] , and the target entropy was the action dimension divided by 2 instead of just the action dimension.

XQL Garg et al. (2023) presents two off-policy algorithms: X-TD3 and X-SAC. We use X-TD3 to be consistent with the DDPG fixed-variance actor of DoubleGum and refer to it throughout as XQL. XQL tunes two hyperparameters per task: the use of twin networks/not and scalar hyperparameter β . We swept over the same β -values as Garg et al. (2023): 1, 2, 5 without Twin Critics and 3, 4, 10 and 20 with Twin Critics. β was tuned in the same way as pessimism - we found a default β value and a β tuned per-suite. β values are presented in Table 5 and were found following results in Appendix F.2.

MoG-DDPG is formed by combining a Mixture-of-Gaussians (MoG) critic with DDPG. The MoG critic was introduced in Appendix A of Barth-Maron et al. (2018) and improved by Shahriari et al. (2022). The latter paper combines the MoG critic with DDPG with distributed training, but we remove the distributed training component because we do not use it in DoubleGum.

QR-DDPG (our introduced baseline) combines the quantile regression method of Dabney et al. (2018b) with a DDPG actor. Although Ma et al. (2020); Wurman et al. (2022) and Teng et al. (2022) have combined quantile regression with SAC, we combine it with DDPG because DoubleGum is built on top of DDPG. Like Dabney et al. (2018b), we use 201 quantiles, but these are initialized with orthogonal initialization and are optimized with the MSE, rather than the Huber loss. QR was developed for discrete control and uses the Huber loss with the RMSProp optimizer popular in discrete control methods. We found better performance with the MSE and Adam optimizer, perhaps confirming the result of Ceron and Castro (2021) in distributional RL for continuous control.

DoubleGum was implemented as DDPG with a variance head described in Section 3.4.

E.5 Compute Requirements

A single training run for discrete control may take up to 3 to 5 minutes on a laptop with an Intel Core i9 CPU, NVIDIA 1050 GPU and 31.0 GiB of RAM. On the same system, a single training run for continuous control takes 1 - 2 hours.

±

±

±

The overwhelming majority of our experiments were run on private infrastructure. This cluster had a mixture of Intel Broadwell, Skylake, Cascade Lake, AMD Rome, AMD Milan CPUs, and NVIDIA P100s, V100s, and A100s GPUs. Benchmarking continuous control took roughly ten times longer than benchmarking discrete control. Multi-threaded experiments for continuous control running twelve seeds in parallel took 5 - 8 hours. 8 algorithms (DoubleGum, DDPG, TD3, MoG-Critics, SAC, XQL, QR-DDPG, FinerTD3) were benchmarked over 33 continuous control environments, and there were further runs for hyperparameter sweeps (4 for DoubleGum, 1 for SAC, 6 for XQL, 1 for QR-DDPG and 4 for FinerTD3), yielding 24 runs in total. These algorithms were run at least 10 times for development and hyperparameter tuning. This yields a lower bound of 8 × 33 × 24 × 10 = 63360 hours (7.23 years) of computation.

Assuming that all experiments were run on Tesla V100-SXM2-16GB (TDP of 250W), the cluster it was run on had a carbon efficiency of 0.0006 kgCO 2 eq/kWh (that of the surrounding power grid) and that there were 63360 hours of cumulative computation, the total emissions were 9.51 kgCO 2 eq, equivalent to driving 36km in an average car. Estimations were conducted using the MachineLearning Impact calculator presented in Lacoste et al. (2019).

F Further Results

F.1 Adjusting the Pessimism of DoubleGum

Figure 8 shows that sample efficiency is sensitive to the pessimism factor c adjusting pessimism per suite greatly impacts sample efficiency. The best performing c was c = -0 . 1 , and was thus set as the default pessimism factor value.

Figure 9 shows that the performance of DoubleGum may be improved when the degree of pessimism is changed per suite. This graph was used to determine what pessimism factor to use in each suite, whose values are reported in Table 5.

F.2 Adjusting the Pessimism of Baseline Algorithms

This section presents graphs used to determine which pessimism values to use for baseline algorithms. All final values are reported in Table 5.

Figure 10 shows that sample efficiency is sensitive to the use of pessimism determined by the use of Twin Networks/not. In aggregate, each method was improved by using Twin Networks. Twin networks were therefore set as the default pessimism option for all baseline algorithms apart from QR-DDPG, because Twin Networks was not used with quantile regression in (Dabney et al., 2018b). Figure 11 was used to determine whether to use Twin Networks/not on a per suite basis.

Similarly, Figures 14 and 15 were respectively used to determine pessimism hyperparameters for FinerTD3. In these two graphs, numbers refer to the i th smallest value returned by the ensemble of target critics. Finally, Figures 12 and 13 were respectively used to determine pessimism and β hyperparameters for XQL.

F.3 Discrete Control

Table 6 presents results for discrete control at 100K timesteps.

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

IQM

c

=

-

0

.

5

c

=

-

0

.

1

(Default)

c

=

0

c

=

0

.

1

c

=

0

.

5

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

15

0

.

30

0

.

45

0

.

60

DMC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

MuJoCo

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

06

0

.

12

0

.

18

0

.

24

0

.

30

MetaWorld

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Box2D

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

IQM

c

=

0

.

5

c

=

0

.

1

(Default)

c

=

0

c

=

0

.

1

c

=

0

.

5

-

-

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

08

0

.

16

0

.

24

0

.

32

0

.

40

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

IQM

DDPG

SAC

QR-DDPG

TD3

Twin-SAC

Twin-XQL

Twin-QR-DDPG

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

15

0

.

30

0

.

45

0

.

60

DMC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

MuJoCo

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

06

0

.

12

0

.

18

0

.

24

MetaWorld

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Box2D

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

IQM

DDPG

TD3

SAC

Twin-SAC

Twin-XQL

QR-DDPG

Twin-QR-DDPG

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

06

0

.

12

0

.

18

0

.

24

0

.

30

Untuned

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

IQM

b

=

3

b

=

4

b

=

10

b

=

20

b

=

1

b

=

2

b

=

5

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

15

0

.

30

0

.

45

0

.

60

DMC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

04

0

.

08

0

.

12

0

.

16

MuJoCo

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

MetaWorld

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Box2D

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

IQM

b

=

3

b

=

4

b

=

10

b

=

20

b

=

1

b

=

2

b

=

5

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

08

0

.

16

0

.

24

0

.

32

0

.

40

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

IQM

Pessimism 0

Pessimism 1 (Default)

Pessimism 2

Pessimism 3

Pessimism 4

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

15

0

.

30

0

.

45

0

.

60

DMC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

MuJoCo

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

MetaWorld

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Box2D

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

IQM

Pessimism 0

Pessimism 1 (Default)

Pessimism 2

Pessimism 3

Pessimism 4

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

15

0

.

30

0

.

45

0

.

60

DMC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

MuJoCo

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

06

0

.

12

0

.

18

0

.

24

MetaWorld

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Box2D

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

050

-

0

.

025

0

.

000

0

.

025

0

.

050

IQM

DoubleGum,

c

=

-

0

.

1

(Ours)

MoG-DDPG

DDPG

TD3

Twin-SAC

Twin-XQL

QR-DDPG

FinerTD3

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

15

0

.

30

0

.

45

0

.

60

DMC

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

MuJoCo

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

00

0

.

06

0

.

12

0

.

18

0

.

24

0

.

30

MetaWorld

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Box2D

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

Timesteps (in millions)

-

0

.

05

0

.

00

0

.

05

IQM

DoubleGum, best

c

, (Ours)

MoG-DDPG (untuned)

best of DDPG/TD3

SAC (best w/wo Twin)

XQL (best of

b

w/wo Twin)

QR-DDPG (best w/wo Twin)

FinerTD3 (best pessimism)

F.4 Continuous Control with Default Pessimism

Figures 16 and 18 respectively present aggregate per-suite and per-task results of DoubleGum benchmarked against baseline algorithms with default pessimism values.

Table 7 presents the performance of all algorithms with default pessimism at 1 million timesteps. This graph has six subsections. The first four subsections present per-task results from DeepMind Control, MuJoCo, MetaWorld, and Box2D, respectively, corresponding to results from 18. The next subsection presents per-suite aggregate results, corresponding to Figure 16, while the last subsection presents aggregate results over all tasks and suites, corresponding to Figure 1. In aggregate results, only the IQM is reported.

F.5 Continuous Control with Pessimism adjusted Per-Suite

Figures 17 and 19 respectively present aggregate per-suite and per-task results of DoubleGum benchmarked against baseline algorithms with the best pessimism values adjusted per-suite.

Table 8 presents the performance of all algorithms with default pessimism at 1 million timesteps. This graph has six subsections. The first four subsections present per-task results from DeepMind Control, MuJoCo, MetaWorld, and Box2D, respectively, corresponding to results from 19. The next subsection presents per-suite aggregate results, corresponding to Figure 17, while the last subsection presents aggregate results over all tasks and suites, corresponding to Figure 1. In aggregate results, only the IQM is reported.

0

.

0

0

.

5

1

.

0

0

150

300

450

acrobot-swingup

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

reacher-hard

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

finger-turn hard

0

.

0

0

.

5

1

.

0

0

100

200

300

400

hopper-hop

0

.

0

0

.

5

1

.

0

0

200

400

600

800

fish-swim

0

.

0

0

.

5

1

.

0

0

200

400

600

800

1000

cheetah-run

0

.

0

0

.

5

1

.

0

0

200

400

600

800

walker-run

0

.

0

0

.

5

1

.

0

0

200

400

600

800

quadruped-run

0

.

0

0

.

5

1

.

0

0

150

300

450

600

750

swimmer-swimmer15

0

.

0

0

.

5

1

.

0

-

50

0

50

100

150

humanoid-run

0

.

0

0

.

5

1

.

0

0

50

100

150

200

250

dog-run

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

IQM

(a) DeepMind Control

0

.

0

0

.

5

1

.

0

0

800

1600

2400

3200

4000

Hopper-v4

0

.

0

0

.

5

1

.

0

0

2500

5000

7500

10000

HalfCheetah-v4

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

Walker2d-v4

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

7500

Ant-v4

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

Humanoid-v4

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

IQM

(b) MuJoCo

0

.

0

0

.

5

1

.

0

-

800

0

800

1600

2400

button-press-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

door-open-v2

0

.

0

0

.

5

1

.

0

-

2000

0

2000

4000

6000

drawer-close-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

drawer-open-v2

0

.

0

0

.

5

1

.

0

-

1500

0

1500

3000

4500

peg-insert-side-v2

0

.

0

0

.

5

1

.

0

-

1200

-

600

0

600

1200

1800

pick-place-v2

0

.

0

0

.

5

1

.

0

-

1600

-

800

0

800

1600

2400

push-v2

0

.

0

0

.

5

1

.

0

0

1000

2000

3000

4000

5000

reach-v2

0

.

0

0

.

5

1

.

0

-

1500

0

1500

3000

4500

window-open-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

window-close-v2

0

.

0

0

.

5

1

.

0

-

1000

0

1000

2000

3000

4000

basketball-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

dial-turn-v2

0

.

0

0

.

5

1

.

0

-

1500

0

1500

3000

4500

6000

sweep-into-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

hammer-v2

0

.

0

0

.

5

1

.

0

-

1000

0

1000

2000

3000

4000

assembly-v2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

IQM

(c) MetaWorld

0

.

0

0

.

5

1

.

0

-

150

0

150

300

450

BipedalWalker-v3

0

.

0

0

.

5

1

.

0

-

80

0

80

160

240

BipedalWalkerHardcore-v3

DoubleGum,

c

=

-

0

.

1

(Ours)

MoG-DDPG

DDPG

TD3

Twin-SAC

Twin-XQL

QR-DDPG

FinerTD3

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

IQM

(d) Box2D

0

.

0

0

.

5

1

.

0

0

100

200

300

400

500

acrobot-swingup

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

reacher-hard

0

.

0

0

.

5

1

.

0

0

250

500

750

1000

finger-turn hard

0

.

0

0

.

5

1

.

0

0

150

300

450

600

hopper-hop

0

.

0

0

.

5

1

.

0

0

200

400

600

800

fish-swim

0

.

0

0

.

5

1

.

0

0

200

400

600

800

1000

cheetah-run

0

.

0

0

.

5

1

.

0

0

200

400

600

800

walker-run

0

.

0

0

.

5

1

.

0

0

200

400

600

800

quadruped-run

0

.

0

0

.

5

1

.

0

0

150

300

450

600

750

swimmer-swimmer15

0

.

0

0

.

5

1

.

0

0

40

80

120

160

humanoid-run

0

.

0

0

.

5

1

.

0

-

50

0

50

100

150

200

dog-run

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

IQM

(a) DeepMind Control

0

.

0

0

.

5

1

.

0

0

1000

2000

3000

4000

Hopper-v4

0

.

0

0

.

5

1

.

0

0

2500

5000

7500

10000

HalfCheetah-v4

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

Walker2d-v4

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

Ant-v4

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

Humanoid-v4

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

IQM

(b) MuJoCo

0

.

0

0

.

5

1

.

0

0

800

1600

2400

button-press-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

door-open-v2

0

.

0

0

.

5

1

.

0

-

2000

0

2000

4000

6000

drawer-close-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

drawer-open-v2

0

.

0

0

.

5

1

.

0

-

1500

0

1500

3000

peg-insert-side-v2

0

.

0

0

.

5

1

.

0

-

800

0

800

1600

2400

pick-place-v2

0

.

0

0

.

5

1

.

0

-

1600

-

800

0

800

1600

2400

push-v2

0

.

0

0

.

5

1

.

0

0

1000

2000

3000

4000

5000

reach-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

window-open-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

window-close-v2

0

.

0

0

.

5

1

.

0

-

1000

0

1000

2000

3000

4000

basketball-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

dial-turn-v2

0

.

0

0

.

5

1

.

0

-

1500

0

1500

3000

4500

6000

sweep-into-v2

0

.

0

0

.

5

1

.

0

0

1500

3000

4500

6000

hammer-v2

0

.

0

0

.

5

1

.

0

-

1000

0

1000

2000

3000

4000

assembly-v2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

IQM

(c) MetaWorld

0

.

0

0

.

5

1

.

0

-

150

0

150

300

450

BipedalWalker-v3

0

.

0

0

.

5

1

.

0

-

80

0

80

160

240

BipedalWalkerHardcore-v3

DoubleGum, best

c

, (Ours)

MoG-DDPG (untuned)

best of DDPG/TD3

SAC (best w/wo Twin)

QR-DDPG (best w/wo Twin)

XQL (best of

b

w/wo Twin)

FinerTD3 (best pessimism)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Timesteps (in millions)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

IQM

(d) Box2D

Peer reviews:
comments: SUMMARY:
The authors propose a novel off-policy algorithm DoubleGum based on their new noise model which is claimed to be more stable for training and reaches higher performance for various discrete and continuous environments.

STRENGTHS:
The paper is well-written and well organized. The authors proposed a novel noise model and off-policy DRL algorithm and analyse their strength through a set of extensive experiments. Some experimental results are interesting and strong. They also provide nice math supporting the paper.

WEAKNESSES:
Figures are a bit confusing. For example, it's a bit hard to see just from the figures that how stable is the new algorithm comparing to existing state-of-the-art. Also, I saw TD3, DDPG MoG-DDPG in figure3, then it is just DDPG (best w/wo Twin Networks) in figure 5. Makes me a bit confused and not sure whether DoubleGum with adjusting pessimism outperforms all the other algorithms or just DDPG?

QUESTIONS:
1.Curious why SAC is not included in the experiments. To my knowledge, it is a very strong off-policy algorithm. It is stable and outperforms DDPG and TD3 in a bunch of continuous tasks. 
2.Can we claim DoubleGum with adjusting pessimism the new state-of-the art?
3.Will there be a way to measure how much more stable is DoubleGum(with fixed pessimism) across different tasks?Since in fixed pessimism, as shown in fig3, the performance seems to be sacrificed.

LIMITATIONS:
As authors mentioned in the paper, the proposed algorithm seems to be very sensitive to hyper-parameters, which is not very applicable if we need to tune pessimism per task.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides empirical evidence that it is inaccurate to assume the temporal difference error follows a homoscedastic normal distribution. The paper proposes to use Gumbel distribution to make a replacement. The optimal action value is considered as the learned value estimation plus a noise sampled from a Gumbel distribution. The Bellman Equation has changed accordingly. Based on this idea, a new algorithm for both discrete and continuous control tasks is derived, called DoubleGum. The paper empirically tested DoubleGum in various tasks, including several relatively complex environments.\n\nSTRENGTHS:\n- The paper provides clear empirical evidence to explain that the temporal difference error is better fitted to a Guambel distribution rather than a normal distribution. \n\n- The paper has a relatively complete related work list. The related work section considered other analyses of the noise in Q-learning and works proposing similar methods.\n\n- DoubleGum empirically works on both discrete and continuous control tasks. To support this, in the experiment section, the paper empirically tested it on multiple environments, including both simple and complex tasks.\n\n- Detailed hyperparameters and computational resources needed are reported, improving the reproducibility.\n\nWEAKNESSES:\n- The main concern comes from the gap between the theory and the practical algorithm. The whole idea is based on one assumption, which is, estimating the temporal difference error with a Gumbel distribution is better than Normal distribution. The Bellman equation is also modified according to the same intuition. However, when deriving the practical algorithm, the paper points out that the Q-function cannot be learned by Gumbel or logistic regression because there lacks a sufficient statistical estimator. So, DoubleGum instead used a generalized method of moments to match the logistic with a normal distribution.\n\n- Although the empirical results did suggest DoubleGum learned faster than the baseline (DDPG), it is not clear to me if the advantage does come from the Gumbel distribution, as the practical algorithm is not using the Gumbel distribution as previous sections discuss. \n\n- The other concern is mainly about the baseline choice. SAC is another algorithm that empirically works well for online learning. It estimates the action value with considering the entropy. The setting that uses a softmax instead of max is consistent with a part of the proposed method in the paper. It might be worth checking SAC’s performance in the tasks listed in paper, and comparing it with DoubleGum.\n\nQUESTIONS:\nI would appreciate it if the authors could explain the abovementioned concerns.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a combination of Gumbel noise with Q-learning. The idea is based on the high level observation that regular L-2 loss based Q-learning can be understood as maximum likelihood estimation with Gaussian noise. The paper derives the practical algorithm and shows some improvements over baselines.\n\nSTRENGTHS:\nThe paper takes a relatively novel perspective on interpreting L-2 loss based Q-learning algorithm as maximum likelihood estimation, and derives an empirically novel algorithm based on the Gumbel noise in place of Gaussian noise. The resulting algorithm looks easy to implement and showcases some practical improvements.\n\nWEAKNESSES:\nSince the algorithm is heuristically motivated, the paper seems to lack a solid theoretical grounding as to whether the newly proposed algorithm would converge or not (not formally stated or proved in the paper). The empirical improvements are interesting but can use more results to further demonstrate the promise of this new approach.\n\nQUESTIONS:\n#### === ** Definition of Q-function and random variables ** ===\n\nThe definition of the equality in distribution look a bit confusing to me in Eqn 9, \n\n$$Q_\\theta(s,a) =_d Q*(s,a) - g_\\theta(s,a)$$\n\nHere $g_\\theta(s,a)$ is a random variable whose distribution depends on $(s,a)$. As a result, the RHS is a random variable with a fixed probability law (once $\\theta$ is fixed). However, the Q-function on the LHS is a single scalar at $(s,a)$. In general, how should we expect such an equality to hold? If we parameterize LHS by a neural network, it outputs a single scalar and the equality cannot hold in general.\n\nRelated to this, the equality of Eqn 10 seems to be an equality in scalar, i.e. both LHS and RHS are scalar quantities. However, $g_\\theta(s,a)$ on the LHS is a random variable, and is in general not a scalar? The definition of $y*$ in Eqn 14, on the other hand, confirms that Eqn 10 should have been an equality in scalar since the RHS $y*(s,a)$ is a scalar.\n\n#### === **Parameterization of Q-function** ===\n\nFollowing the above discussion, in line 127 there is a use of distributional equality again, where LHS is $Q_\\theta^\\text{new}$. For this equality to hold, we do have to parameterize the Q-function output as a distribution at $(s,a)$ instead of a single scalar? A further clarification on this would be very helpful.\n\n#### === **Convergence guarantee** ===\n\nThe algorithm proposed is very linked to max-entropy RL, the convergence property of which has been established. The L-2 based Q-learning algorithm is also convergent under suitable conditions. I wonder if there is a similar guarantee here, and if yes would be good to have in the main paper.\n\n#### === **Empirical results** ===\n\nGumbel DDPG seems to improve over DDPG but not over TD3. I guess a main reason might be a TD3 uses double Q-learning, which is orthogonal to what's proposed here. Can we combine Gumbel trick with TD3 and show improvements over TD3? DDPG is in general a much more inferior baseline than TD3 due to the over-estimation bias, it'd be nice to showcase improvements over TD3 too.\n\nFrom Fig 4 it seems that the performance is sensitive to the hyper-parameter $c$. How is the choice of $c=0$ selected in other experiments in comparison to alternative baselines? Is $c=0$ the best-performing hyper-parameter in general?\n\nLIMITATIONS:\nDiscussed above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTypically, TD learning assumes that the TD error follows a normal distribution with a fixed variance (induced by the Bellman squared loss). The authors argue that this assumption is too coarse in practice since the maximization of noisy Q-values across actions (when backing up the value in TD learning) is usually not Gaussian. To fix this, the authors discuss the proper limiting distribution of the Q-function when accounting for the Q-function maximization, and provides a practical temporal-difference backup algorithm that can accurately capture the distribution in the discrete control case and approximately capture it in the continuous control case. On a range of simulated robotic and control tasks, the proposed method is able to achieve better performance compared with existing approaches that naively backup the noisy target (without considering the interaction between the noise and the action maximization).\n\nSTRENGTHS:\nTo the best of my knowledge, this is the first paper that uses Gumbel distributions to model TD errors. The main strength of the paper is in the theoretical analysis around this TD error model and a theoretically justified, novel algorithm that can perform approximated TD backup with this new TD error model. The writing is clear and very easy to follow.\n\nWEAKNESSES:\nThe main weakness of the paper is the empirical evaluation. Overall, I am not convinced that 1) assuming that TD errors follow a homoscedastic normal distribution is problematic on the domains of tasks being tested, 2) the proposed method is able to improve upon existing approaches on these tasks. \n\n\n- Section 4: I am not sure I am convinced that the Logistic model fits the empirical data better than the Gaussian model (from Figure 1). It actually makes me wonder how much is actually needed in practice to accurately account for the Gumbel noise. When do you expect the gap between these two models to be bigger? \n- L196-197: How is the error bar computed? It seems a bit problematic to claim that the normal distribution is a suitable approximation given that their error bars overlap since it could be just due to the distribution variance. \n- Just from Figure 3 it is hard to tell whether DoubleGum is better than MoG-DDPG. DoubleGum is slightly better on MuJoCo but slightly worse on both DMC and Box2D.\n\nQUESTIONS:\n- L170: $\\sigma(s)$ => $\\sigma(s, a)$?\n- I wonder how much of the problem in the assumption (that TD error is assumed to be Gaussian) can already be addressed by using distributional RL. \n- It is very interesting to see that the method has a parameter that can adjust for the degree of pessimism more smoothly than the twin networks in TD3. There are also methods like RedQ that can adjust this with more fine-grain controls which has not been considered by the authors (though admittedly RedQ would be more expensive as it requires more critic ensemble elements).\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInstead of modeling the TD error with a homoscedastic normal distribution, this paper tries to utilize two heteroscedastic Gumbel distributions for more complex and accurate error modeling. Based on this assumption, the authors presents a modified Q-learning algorithm, DoubleGum for solving discrete and continuous control tasks. In particular, DoubleGum can also achieve the effect of pessimism so as to avoid the overestimation. Empirical results show more stable training and competitive performance across classic discrete and continuous tasks.\n\nSTRENGTHS:\nIt's a novel perspective to model the TD-error following the Gumbel distribution.\n\nWEAKNESSES:\n1. Double error modeling in section 3.1 seems a little redundant and not intuitive enough；\n2. Though linking the DoubleGum for continuous control to the pessimistic value estimation, the empirical evidence is lacking.\n3. Experimental results don't show outstanding performance improvement across multiple benchmarks, and lack of comparison with similar algorithms, e.g. extreme Q-learning[1]. In addition, It is shown that the Gumbel error model helps reduce the overestimation issue, which is quite natural considering its close connection with SAC. So why not compare its behavior with SAC in more detail?\n\n[1] Garg, Divyansh, et al. \"Extreme Q-Learning: MaxEnt RL without Entropy.\" The Eleventh International Conference on Learning Representations. 2023.\n\nQUESTIONS:\n1. What's the motivation of using an additional Gumbel noise for the new function approximator $Q^{new}$ in Sec.3.1? Because this modeling does not seem to involve the Extreme Value Theorem, why to model the error using the same Gumbel distribution? \n\n2. About the DoubleGum for the continuous control, it's said that, for ease of implementation, the pessimism factor $c$ is setted as 0, if this, it doesn't look much different than the common off-policy methods;\n\n3. Sec.3.2 tries to connect the DoubleGum with the pessimistic value estimation, but the experimental results only contain the performance comparison for different choices of $c$, It is better to compare the magnitude of the value function directly;\n\n4. Motiviated by a similar idea, extreme Q-learning also model the TD-error as Gumbel distribution, so it is better to make a detailed comparison between the both algorithms, including methodological differences and experimental performance.\n\n5. About Fig.1a and Fig.1b, there is little difference for fittings by Logistic distribution or Normal distribution. I am not sure that it is a good evidence to justify the proposed error model.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}