SlowGuess/NeurIPS_2023_OCR huggingface dataset:
David Yu-Tung Hui
Mila, Université de Montréal
dythui2+drl@gmail.com
Aaron Courville
Pierre-Luc Bacon
We show that Deep Neural Networks introduce two heteroscedastic Gumbel noise sources into Q-Learning. To account for these noise sources, we propose Double Gumbel Q-Learning, a Deep Q-Learning algorithm applicable for both discrete and continuous control. In discrete control, we derive a closed-form expression for the loss function of our algorithm. In continuous control, this loss function is intractable and we therefore derive an approximation with a hyperparameter whose value regulates pessimism in Q-Learning. We present a default value for our pessimism hyperparameter that enables DoubleGum to outperform DDPG, TD3, SAC, XQL, quantile regression, and Mixture-of-Gaussian Critics in aggregate over 33 tasks from DeepMind Control, MuJoCo, MetaWorld, and Box2D and show that tuning this hyperparameter may further improve sample efficiency.
Reinforcement Learning (RL) algorithms learn optimally rewarding behaviors (Sutton and Barto, 1998, 2018). Recent RL algorithms have attained superhuman performance in Atari (Mnih et al., 2015), Go (Silver et al., 2016), Chess (Schrittwieser et al., 2020), StarCraft (Vinyals et al., 2019), simulation racing (Wurman et al., 2022), and have also been used to control stratospheric balloons (Bellemare et al., 2020) and magnetic fields in Tokamak nuclear fusion reactors (Degrave et al., 2022). All these aforementioned agents use a  $Q$ -function (Watkins and Dayan, 1992) to measure the respective quality of their behaviors.
Many algorithms (Lillicrap et al., 2015; Mnih et al., 2016, 2015; Schulman et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018b) learn a  $Q$ -function by minimizing the Mean-Squared Bellman Error (MSBE). The functional form of the MSBE was motivated by an analogy to the popular Mean-Squared Error (MSE) in supervised learning regression problems, but there is little theoretical justification for its use in RL (Bradtke and Barto, 1996; Ernst et al., 2005; Riedmiller, 2005).
Training a deep neural network to minimize the MSBE is empirically unstable (Irpan, 2018; Henderson et al., 2018; Van Hasselt et al., 2016) because deep neural networks induce overestimation bias in the  $Q$ -function (Thrun and Schwartz, 1993; Fujimoto et al., 2018). A popular method to reduce overestimation 'pessimistically' selects less positive  $Q$ -values from an ensemble of  $Q$ -functions, effectively returning low-quantile estimates (Fujimoto et al., 2018; Kuznetsov et al., 2020; Ball et al., 2023). However, an ensemble is computationally expensive and quantiles only provide discrete-grained control over the degree of pessimism. Curiously, pessimism has mainly been applied to 'continuous control' problems that require continuous-valued actions and not discrete control.
This paper analyzes noise in Deep  $Q$ -Learning. We introduce a loss function for Deep  $Q$ -Learning and a scalar hyperparameter that adjusts pessimism without an ensemble. These two innovations form Double Gumbel  $Q$ -Learning (DoubleGum), an off-policy Deep  $Q$ -Learning algorithm applicable to both discrete and continuous control. Our paper is structured as follows:
1. Section 2.1 shows that the MSBE implicitly uses a noise model of the Bellman Optimality Equation with additive homoscedastic noise. Section 4.1 shows that this noise model is empirically too coarse to fully capture the distribution of noise in Deep  $Q$ -Learning.
2. Section 3 derives a noise model of the Soft Bellman Equation with additive heteroscedastic logistic noise derived from two heteroscedastic Gumbel noise sources. Section 4.1 shows that our noise model is a better empirical fit to the noise distribution in Deep  $Q$ -Learning.
3. Section 3.3 shows that when the soft value function in the Soft Bellman Equation is intractable in continuous control, its approximation leads to methods that adjust pessimism. Section 3.1 presents a method to adjust pessimism with a single scalar hyperparameter, and its effect is empirically verified in Section 4.2.
4. Section 5 discusses related work, notably presenting DoubleGum as a special case of Distributional RL and a generalization of existing Maximum Entropy (MaxEnt) RL algorithms.
5. Section 6 benchmarks DoubleGum, showing marginal improvements over DQN baselines on 3 classic discrete control tasks and improved aggregate performance over DDPG, SAC, TD3, XQL and two Distributional RL algorithms over 33 continuous control tasks.
2 Mathematical Background
Reinforcement Learning agents learn to make maximally rewarding decisions in an environment (Sutton and Barto, 1998, 2018). In Markov Decision Process (MDP) environments, a  $Q$ -function specifies the expected amount of reward an agent obtains (Watkins and Dayan, 1992) and is also known as a state-action value function. MDPs define a transition probability  $p(s' \mid s, a)$  and reward  $r(s, a, s')$  for every pair of states  $s, s' \in S$  and action  $a \in \mathcal{A}$  (Bellman, 1957; Howard, 1960). Decisions are made by a policy  $\pi(a \mid s)$  at every discrete timestep and the  $Q$ -function of  $\pi$  at  $s, a$  is
$$
Q _ {\pi} (s, a) = \underset {\pi , p} {\mathbb {E}} \left[ \sum_ {j = i} ^ {n} \gamma^ {j} r _ {j} \mid s _ {i} = s, a _ {i} = s \right], \text {d e n o t i n g} r \left(s _ {j}, a _ {j}, s _ {j + 1}\right) \text {a s} r _ {j},
$$
where the discount factor  $\gamma \in ]0,1[$  ensures finite  $Q$  for infinite time-horizon  $n$  (Samuelson, 1937). The expectation in the  $Q$ -function is computed over a trajectory sequence of  $s$  and  $a$  induced by the coupling of the policy and the transition function.
In every MDP, an optimal policy  $\pi^{\star}$  makes maximally rewarding decisions following  $s\rightarrow$  arg  $\max_{a}Q^{\star}(s,a)$  (Watkins and Dayan, 1992).  $Q^{\star}$  is the optimal value function specified as the unique fixed point of the Bellman Optimality Equation of the MDP (Bellman, 1957)
$$
Q ^ {\star} (s, a) = \underset {p \left(s ^ {\prime} \mid s, a\right)} {\mathbb {E}} \left[ r + \gamma \max  _ {a ^ {\prime}} Q ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right) \right], \text {o v e r a l l} s \text {a n d} a, \text {d e n o t i n g} r (s, a, s ^ {\prime}) \text {a s} r. \tag {1}
$$
In Deep  $Q$ -Learning,  $Q^{\star}$  is approximated by a neural network  $Q_{\theta}$  with parameters  $\theta$ . Following Munos and Szepesvári (2008),  $\theta$  is updated by a sequence of optimization problems whose  $i^{\mathrm{th}}$  stage iteration is given by
$$
\theta_ {i + 1} = \arg \min _ {\theta} \mathrm {L} (Q _ {\theta}, y _ {\bar {\theta} _ {i}}), \mathrm {w h e r e} y _ {i} (s, a) = \underset {p (s ^ {\prime} | s, a)} {\mathbb {E}} \left[ r + \gamma \max _ {a ^ {\prime}} Q _ {\bar {\theta} _ {i}} (s ^ {\prime}, a ^ {\prime}) \right],
$$
where the 'stop-gradient' operator  $(\cdot)$  denotes evaluating but not optimizing an objective with respect to its argument and  $y$  is dubbed the bootstrapped target. In this work, we restrict our environments to deterministic MDPs, so the resultant expectation over  $s'$  is computed over one sample.
Typically, the loss function  $\mathrm{L}(Q_{\theta},y_{\bar{\theta}_i}) = \mathbb{E}_{s,a}\left(Q_{\theta}(s,a) - y_{\bar{\theta}_i}(s,a)\right)^2$  is the Mean-Squared Bellman Error (MSBE). At every  $i$ ,  $\mathrm{L}$  is minimized with stochastic gradient descent by a fixed number of gradient descent steps and not optimized to convergence (Mnih et al., 2015; Lillicrap et al., 2015).
$^{1}$ Code: https://github.com/dyth/doublegum
2.1 Deriving the Mean-Squared Bellman Error
Thrun and Schwartz (1993) model  $Q_{\theta}(s, a)$  as a random variable whose value varies with  $\theta, s$  and  $a$ . They set up a generative process  $Q_{\theta}(s, a) = Q^{\star}(s, a) - \epsilon_{\theta, s, a}$  showing that the function approximator  $Q_{\theta}$  is a noisy function of the implicit true value  $Q^{\star}$ , but they do not make any assumptions about  $\epsilon$ .<sup>2</sup>
We derive the MSBE with two additional assumptions:  $\epsilon_{\theta ,s,a}\stackrel {\mathrm{iid}}{\sim}\mathcal{N}(0,\sigma^2)$ , where  $\mathcal{N}$  is a homoscedastic normal due to the Central Limit Theorem (CLT)  $^{3}$  and secondly  $Q^{\star}(s,a)\approx y_{\bar{\theta}}(s,a)$ , where  $\theta$  are the parameters at an arbitrary optimization stage. Incorporating these two assumptions into the noise model yields  $y_{\bar{\theta}}(s,a) = Q_{\theta}(s,a) + \epsilon_{\theta ,s,a}$ , uncovering the implicit assumption that Temporal-Difference (TD) errors  $y_{\bar{\theta}} - Q_{\theta}$  follow a homoscedastic normal. Maximum Likelihood Estimation (MLE) of  $\theta$  on the resultant noise model yields the MSBE after abstracting away the constants
$$
\max _ {\theta} \underset {s, a} {\mathbb {E}} \log p (y _ {\bar {\theta}} (s, a)) = \min _ {\theta} \underset {s, a} {\mathbb {E}} \left[ \log \sigma \sqrt {2 \pi} + \frac {1}{\sigma^ {2}} \left(Q _ {\theta} (s, a) - y _ {\bar {\theta}} (s, a)\right) ^ {2} \right].
$$
2.2 The Limiting Distribution in Bootstrapped Targets
Both assumptions used to derive the MSBE are theoretically weak. First, the CLT states that iid samples of an underlying distribution tend towards a homoscedastic normal but do not account for the form of the underlying distribution. Secondly, there is no analytic justification for  $Q^{\star}(s,a) \approx y_{\bar{\theta}}(s,a)$ . Substituting the Thrun and Schwartz (1993) model in the RHS of the Bellman Optimality Equation yields
$$
Q ^ {\star} (s, a) = \underset {p (s ^ {\prime} | s, a)} {\mathbb {E}} \left[ r + \gamma \max _ {a ^ {\prime}} [ Q _ {\theta} (s ^ {\prime}, a ^ {\prime}) + \epsilon_ {\theta , s ^ {\prime}, a ^ {\prime}} ] \right] = \underset {p (s ^ {\prime} | s, a)} {\mathbb {E}} \left[ r + \gamma g _ {\theta , s ^ {\prime}} \right] \neq y _ {\bar {\theta}} (s, a).
$$
When the max over a finite number of iid samples with unbounded support is taken, the Extreme Value Theorem (EVT) gives the limiting distribution of the resultant random variable as a Gumbel distribution (Fisher and Tippett, 1928; Gnedenko, 1943), defined here as  $g_{\theta ,s^{\prime}} \sim \mathcal{G}(\alpha ,\beta)$ . Expressions relating  $\alpha$  and  $\beta$  to the parameters of the underlying iid random variables are rarely analytically expressible (Kimball, 1946; Jowitt, 1979) and there is no guarantee  $Q^{\star}(s,a)\approx y_{\bar{\theta}}(s,a)$ .
3 Deriving a Deep Q-Learning Algorithm from First Principles
We find an analytic expression that relates the limiting Gumbel distribution to parameters of the underlying noise distribution from function approximation. To do so, we assume that noise in the Thrun and Schwartz (1993) model is a heteroscedastic Gumbel noise with state-dependent spread
$$
Q _ {\theta} (s, a) = Q ^ {\star} (s, a) - g _ {\theta , a} (s), \quad g _ {\theta , a} (\cdot) \stackrel {{\mathrm {i i d}}} {{\sim}} \mathcal {G} (0, \beta_ {\theta} (\cdot)), \quad \text {f o r a l l} a \in \mathcal {A}, \tag {2}
$$
Appendix A.1 restates the finding of McFadden et al. (1973) and Rust (1994), which results in the following expression with the soft value function  $V_{\theta}^{\mathrm{soft}}$
$$
\begin{array}{l} \max _ {a} Q ^ {\star} (s, a) = \max _ {a} \left[ Q _ {\theta} (s, a) + g _ {\theta , a} (s) \right] = \beta_ {\theta} (s) \log \int \exp \left(\frac {Q _ {\theta} (s , a)}{\beta_ {\theta} (s)}\right) \mathrm {d} a + g _ {\theta} (s) \\ = V _ {\theta} ^ {\text {s o f t}} (s) + g _ {\theta} (s), \quad \text {w h e r e} g _ {\theta , a} (\cdot), g _ {\theta} (\cdot) \stackrel {{\text {i i d}}} {{\sim}} \mathcal {G} (0, \beta_ {\theta} (\cdot)). \tag {3} \\ \end{array}
$$
Substituting Equations 2 and 3 into 1 yields a new noise model of the Soft Bellman Optimality Equation with two function approximators and two heteroscedastic Gumbel noise sources
$$
Q _ {\theta} (s, a) + g _ {\theta , a} (s) = \underset {p \left(s ^ {\prime} \mid s, a\right)} {\mathbb {E}} \left[ r + \gamma V _ {\bar {\theta}} ^ {\text {s o f t}} \left(s ^ {\prime}\right) + \gamma g _ {\bar {\theta}} \left(s ^ {\prime}\right) \right]. \tag {4}
$$
We now develop this noise model into a loss function used in our algorithm, DoubleGum.
2This equation derives from an unnumbered equation at the top of Page 3 of Thrun and Schwartz (1993), and we have renamed their variables to fit our notation. The authors add the noise  $\epsilon$  to  $Q^{\star}$ , but we subtract it for notational clarity later on without loss of generality. We additionally subscript the noise by  $\theta$  to clarify that the random variable is resampled for different  $\theta$ s in the function approximator.
<sup>3</sup>Ernst et al. (2005) and Riedmiller (2005) introduced the MSBE to  $Q$ -Learning, arguing that MSE was popular for regression problems in supervised learning. In regression, the MSE arises from MLE parameter estimation of a homoscedastic normal, which is assumed to exist because of the CLT.
3.1 Merging Two Distributions
We replace  $Q_{\theta}$  with a new function approximator  $Q_{\theta}^{\mathrm{new}}(s,a) = Q_{\theta}(s,a) + g_{\theta}(s)$ , resulting in a new soft value function on the RHS of Equation 4
$$
V _ {\theta} ^ {\text {s o f t}} (s) + g _ {\theta} (s) = \beta_ {\theta} (s) \log \int \exp \left(\frac {Q _ {\theta} ^ {\text {n e w}} (s , a)}{\beta_ {\theta} (s)}\right) \mathrm {d} a = V _ {\theta} ^ {\text {s o f t}, \text {n e w}} (s) \tag {A.2}
$$
and a LHS with a heteroscedastic logistic random variable obtained in Appendix A.3 from the difference of two Gumbel random variables with equation spreads. Note that this operation is only permissible in MDPs without terminating states. Equation 4 simplifies into
$$
Q _ {\theta} ^ {\text {n e w}} (s, a) + l _ {\theta , a} (s) = \underset {p \left(s ^ {\prime} \mid s, a\right)} {\mathbb {E}} \left[ r + \gamma V _ {\hat {\theta}} ^ {\text {s o f t}, \text {n e w}} \left(s ^ {\prime}\right) \right], l _ {\theta , a} (\cdot) \sim \mathcal {L} \left(0, \beta_ {\theta} (\cdot)\right), \tag {5}
$$
the Soft Bellman Equation with heteroscedastic logistic noise. The bootstrapped targets in this equation is the RHS, which we will denote by  $y_{\hat{\theta}}^{\mathrm{soft}}(s,a)$ .
3.2 Parameter Estimation with Moment Matching
Equation 5 does not allow MLE of  $\theta$  because the heteroscedastic logistic is not a member of the exponential family and lacks a sufficient statistical estimator. We learn  $\theta$  with moment matching by MLE of a heteroscedastic normal  $y_{\bar{\theta}}^{\mathrm{soft}}(s,a)\sim \mathcal{N}(Q_{\theta}^{\mathrm{new}}(s,a),\sigma_{\theta}(s))$  given by
$$
\min  _ {\theta} \left[ \log \sigma_ {\theta} (s) + \left(\frac {y _ {\theta} ^ {\text {s o f t}} (s , a) - Q _ {\theta} ^ {\text {n e w}} (s , a)}{\sigma_ {\theta} (s)}\right) ^ {2} \right], \tag {6}
$$
and recovering the logistic location and spread as  $Q_{\theta}^{\mathrm{new}}(s,a)$  and  $\beta_{\theta}(s) = \sigma_{\theta}(s)\frac{\sqrt{3}}{\pi}$ .
3.3 Computing the Bootstrapped Targets
$y_{\theta}^{\mathrm{soft}}$  is calculated with one sample from  $s^\prime$  and  $a^\prime$  where appropriate. In discrete control,  $\mathcal{A}$  is finite and the expression  $V_{\theta}^{\mathrm{soft,new}}$  is a sum. The resultant bootstrapped target in discrete control is
$$
y _ {\theta} ^ {\mathrm {d i s c r e t e}} (s, a, r, s ^ {\prime}) = r + \gamma V _ {\theta} ^ {\mathrm {s o f t , n e w}} (s ^ {\prime}).
$$
In continuous control,  $V_{\theta}^{\mathrm{soft},\mathrm{new}}$  is intractable because  $\mathcal{A}$  contains infinitesimally small quantities. Appendix A.4, adapted from Equation 19 in Haarnoja et al. (2017), variationally approximates the integral with respect to a policy neural network  $\pi_{\phi}(a|s)$  with parameters  $\phi$ , producing
$$
V _ {\theta} ^ {\text {s o f t , n e w}} (s) = \underset {\pi_ {\phi} (a | s)} {\mathbb {E}} \left[ Q _ {\theta} ^ {\text {n e w}} (s, a) \right] + \beta_ {\theta} (s) \mathbb {C} \left[ \pi_ {\phi} \mid \mid p _ {\theta} \right], \text {w h e r e} p _ {\theta} (a \mid s) \propto \exp \frac {Q _ {\theta} ^ {\text {n e w}} (s , a)}{\beta_ {\theta} (s)}. \tag {7}
$$
Denoting  $\mathbb{C}[\pi_{\phi}||p_{\theta}]$  as a constant  $\frac{\pi}{\sqrt{3}} c$  yields bootstrapped targets of
$$
y _ {\theta} ^ {\text {c o n t i n u o u s}} (s, a, r, s ^ {\prime}) = r + \gamma_ {\pi_ {\phi} \left(a ^ {\prime} \mid s ^ {\prime}\right)} \mathbb {E} \left[ Q _ {\theta} ^ {\text {n e w}} \left(s ^ {\prime}, a ^ {\prime}\right) \right] + \gamma c \sigma_ {\theta} \left(s ^ {\prime}\right). \tag {8}
$$
We treat  $c$  as a hyperparameter named the pessimism factor defined before training and held constant throughout. Appendix F.1 empirically finds that a default value of  $c = -0.1$  and details how  $c$  may be adjusted. Appendix B.3 shows that the pessimism factor performs the same role as pessimistic ensembles but with more computational efficiency and fine-grained control.
3.4 Algorithm
Algorithm 1 presents pseudocode for DoubleGum, an off-policy Deep  $Q$ -Learning algorithm that learns  $\theta$  in both discrete and continuous control. DoubleGum is named after the two Gumbels used in the derivation of its noise model, Equation 4. We outline some implementation details that empirically improved the stability of training.
Target Networks: As is standard,  $y^{\mathrm{discrete}}$  and  $y^{\mathrm{continuous}}$  are computed with target networks, whose parameters we denote by  $\psi$ . We used target networks with exponential moving weight updating. This
Algorithm 1: DoubleGum: An Off-Policy Deep  $Q$ -Learning Algorithm
Input: MDP  $(S, A, r, p)$ , replay buffer  $\mathcal{D} = \emptyset$  (Lin, 1992), initial parameters  $\theta_0$  (and  $\phi_0$  in continuous control), initial state  $s$ , learning rates  $\eta_{\theta}$  and  $\eta_{\phi}$ , target network EMA rate  $\eta_{\psi}$ , batch size  $n$ .  
Output: Parameters  $\theta$  (and  $\phi$  in continuous control)  
if discrete control then  
define  $\pi(s) = \max_a Q_{\theta_i}^{\text{new}}(s, a)$  at iteration  $i$   
define  $y = y^{\text{discrete}}$   
define  $\sigma = \sigma_{\theta_i}(s)$  at iteration  $i$   
else if continuous control then  
define  $\pi(s) = \pi_{\phi_i}(s)$  at iteration  $i$   
define  $y = y^{\text{continuous}}$   
define  $\sigma = \sigma_{\theta_i}(s, a)$  at iteration  $i$ $\psi_0 \gets \theta_0$   
for iteration  $i$  do  
 $a \gets \pi(s)$ $s' \sim p(s | s, a)$ $\mathcal{D} \gets \mathcal{D} \cup \{(s, a, r(s, a, s'), s')\}$ $s \gets s'$ $\theta_{i+1} \gets \theta_i - \eta_{\theta} \nabla_{\theta} \underset{(s, a, r, s')_{1:n} \sim \mathcal{D}}{\mathbb{E}}[J_{\theta, y_{\psi_i}, \sigma}(s, a, r, s')] \Big|_{\theta_i}$ $\psi_{i+1} \gets \eta_{\psi} \psi_i + (1 - \eta_{\psi}) \theta_{i+1}$   
if continuous control then  
 $\phi_{i+1} \gets \phi_i + \eta_{\phi} \nabla_{\phi} \underset{(s, a, r, s')_{1:n} \sim \mathcal{D}}{\mathbb{E}}[Q_{\theta_i}^{\text{new}}(s, \pi_{\phi}(s))] \Big|_{\phi_i}$
is commonly used in continuous control (Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018b), and has been recently been shown to improve discrete control (D'Oro et al., 2022).
Q-Functions with Variance Networks: We learn  $\sigma$  with variance networks (Nix and Weigend, 1994; Kendall and Gal, 2017). In discrete control,  $\sigma_{\theta}(s)$  is a variance head added to the  $Q$ -network, and is approximated as  $\sigma_{\theta}(s,a)$  in continuous control. The variance head is a single linear layer whose input is the penultimate layer of the main  $Q$ -network and output is a single unit with a SoftPlus activation function (Dugas et al., 2000) that ensures its positivity. Seitzer et al. (2022) improves stability during training by multiplying the entire loss by  $\sigma_{\bar{\theta}}$ , the numerical value of the variance network, yielding a loss function for  $\theta$  which for continuous control is
$$
J _ {\theta , y _ {\psi}, \sigma_ {\theta}} (s, a, r, s ^ {\prime}) = \left[ \sigma_ {\bar {\theta}} (s, a) \left(\log \sigma_ {\theta} (s, a) + \left(\frac {y _ {\psi} (s , a , r , s ^ {\prime}) - Q _ {\theta} ^ {\text {n e w}} (s , a)}{\sigma_ {\theta} (s , a)}\right) ^ {2}\right) \right] \quad . \tag {9}
$$
Note that in the above equation  $\sigma_{\bar{\theta}}(s,a)$  would be  $\sigma_{\bar{\theta}}(s)$  in discrete control. We do not have any convergence guarantees for this loss function, and Appendix C discusses this issue in more detail.
Policy: In discrete control, actions are taken by  $\max_{a} Q_{\theta}^{\mathrm{new}}(s, a)$  as standard. Exploration was performed by policy churn (Schaul et al., 2022). In continuous control, the policy  $\pi_{\phi}$  is a separate network. We use a DDPG fixed-variance actor because Yarats et al. (2021) showed it trained more stably than a SAC actor with learned variance. Following Laskin et al. (2021), the actor's outputs were injected with zero-mean Normal noise with a standard deviation of 0.2 truncated to 0.3.
Network Architecture: All networks had two hidden layers of size 256, ReLU activations (Glorot et al., 2011), orthogonal initialization (Saxe et al., 2013) with a gain of  $\sqrt{2}$  for all layers apart from the last layer of the policy and variance head, which had gains of 1. This initialization had been shown to be empirically advantageous in policy-gradient and  $Q$ -Learning methods in Huang et al. (2022a) and Kostrikov (2021), respectively. Ball et al. (2023) improves stability in continuous control using LayerNorm (Ba et al., 2016). We find that the similar method of GroupNorm (Wu and He, 2018) with 16 groups without a shift and scale (Xu et al., 2019) worked better, but only in continuous control. All parameters were optimized by Adam (Kingma and Ba, 2014) with default hyperparameters.
Figure 1: The empirical distribution of noise in discrete control CartPole-v1. (a, b): Fitted normal, logistic, and Gumbel distributions against (a) unstandardized and (b) standardized empirical distributions in Deep  $Q$ -Learning at the end of training. (c): Negative Log-Likelihoods (NLLs) of the noise in Deep  $Q$ -Learning under different distributions throughout training (lower is better). Appendix D.1 presents further results in more discrete and continuous control tasks.
Figure 2: The effect of changing pessimism factor  $c$  on the target  $Q$ -value in continuous control. Appendix D.2 presents further results in more tasks.
4 Empirical Evidence for Theoretical Assumptions
4.1 Noise Distributions in Deep Q-Learning
Figure 1 shows the evolution of noise during training DoubleGum in the classic control task of CartPole-v1. Histograms in Figures 1(a) and 1(b) were generated from  $10^{4}$  samples from the replay buffer after 1 million training timesteps. Figure 1(a) computes TD-errors by  $y_{\psi}(s,a,r,s') - Q_{\theta}^{\mathrm{new}}(s,a)$ , while Figure 1(b) computes standardized TD-errors by dividing the previous equation by  $\sigma_{\theta}(s)$  or  $\sigma_{\theta}(s,a)$  where appropriate. The normal, logistic, and Gumbel distributions were fitted by moment matching, and homo(scedastic) and hetero(scedastic) distributions were respectively fitted to unstandardized and standardized data. Figure 1(c) shows the mean and standard deviation over 12 Negative Log-Likelihoods (NLLs), each computed from a different training run with a different (randomly chosen) initial seed. Every NLL was computed over  $10^{4}$  samples from the replay buffer every 1000 timesteps of training.
Figure 1(a) shows that a homoscedastic normal coarsely fits the empirical distribution, forming a good estimate of the mean but not the spread. Our result contradicts Garg et al. (2023) that fitted a Gumbel to the empirical distribution. We discuss this discrepancy in Appendix E.1. Figure 1(b) shows that a heteroscedastic Logistic captures both the mean and spread of the TD-errors, validating Equations 5 and its derivation. Finally, Figure 1(c) shows that a moment-matched heteroscedastic normal used in DoubleGum is a suitable approximation to the heteroscedastic logistic throughout training, validating Equation 6. Appendix D.1 shows that the trend in Figure 1(c) holds for other discrete control environments as well as continuous control.