SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \min  _ {\phi} \beta \mathrm {D} _ {\mathrm {K L}} \left[ \pi_ {\phi} \mid p _ {\theta} \right] = \min  _ {\phi} \beta \int \pi_ {\phi} (a \mid s) \log \frac {\pi_ {\phi} (a \mid s)}{p _ {\theta} (a \mid s)} \mathrm {d} a \\ = \max  _ {\phi} \left[ \beta \mathbb {H} [ \pi_ {\phi} ] + \beta \int \pi_ {\phi} (a \mid s) \log \frac {\exp Q _ {\theta} ^ {\text {n e w}} (s , a) / \beta}{\int \exp Q _ {\theta} ^ {\text {n e w}} (s , a ^ {\prime}) / \beta \mathrm {d} a ^ {\prime}} \mathrm {d} a \right] \\ = \max  _ {\phi} \left[ \beta \mathbb {H} \left[ \pi_ {\phi} \right] + \beta \int \pi_ {\phi} (a \mid s) \frac {Q _ {\theta} ^ {\text {n e w}} (s , a)}{\beta} \mathrm {d} a \right] \\ \end{array}
$$
which is then estimated by Monte-Carlo samples from  $\pi_{\phi}$  as
$$
\max  _ {\phi} \underset {\pi_ {\phi} (a | s)} {\mathbb {E}} \left[ Q _ {\theta} ^ {\text {n e w}} (s, a) - \beta \log \pi_ {\phi} (a \mid s) \right]. \tag {15}
$$
SAC (Haarnoja et al., 2018a,b) has a policy with learned variance and state-independent  $\beta$ . DDPG (Lillicrap et al., 2015) has a fixed-variance policy which removes the second term in Equation 15 as it is constant with respect to the maximization. DoubleGum has a state-dependent  $\beta(s)$ , but uses the same actor loss as DDPG because DoubleGum uses a DDPG fixed-variance policy.
B.2 Maximum-Entropy Reinforcement Learning
SACv1 (Haarnoja et al., 2018b) is a special case of DoubleGum and DDPG (Lillicrap et al., 2015) is a special case of SAC. All three continuous control algorithms have an actor and critic loss derived from the same principle. Section B.1 shows this for the actor losses of DoubleGum, SAC, and DDPG. We now relate the critic losses to each other, starting from the most general case, DoubleGum. In continuous control, DoubleGum uses the following noise model, formed from substituting Equation 7 into Equation 5:
$$
Q _ {\theta} ^ {\text {n e w}} (s, a) + l _ {\theta , a} (s) = \underset {p \left(s ^ {\prime} \mid s, a\right)} {\mathbb {E}} \left[ r + \gamma \underset {\pi_ {\phi} \left(a ^ {\prime} \mid s ^ {\prime}\right)} {\mathbb {E}} \left[ Q _ {\theta} ^ {\text {n e w}} \left(s ^ {\prime}, a ^ {\prime}\right) \right] + \gamma \beta_ {\theta} (s) \mathbb {C} \left[ \pi_ {\phi} \mid p _ {\theta} \right] \right]. \tag {16}
$$
Here,  $l_{\theta ,a}(\cdot)\sim \mathcal{L}(0,\beta_{\theta}(\cdot))$  is a logistic distribution and  $p_{\theta}(a\mid s)\propto \exp \frac{Q_{\theta}^{\mathrm{new}}(s,a)}{\beta_{\theta}(s)}$ . The DoubleGum critic loss is derived from this noise model by approximating the RHS with Equation 8 and learning  $\theta$  with moment matching in Section 3.2.
The SAC noise model is derived from Equation 16 in three ways. First, SAC approximates  $l_{\theta,a}(s) \sim \mathcal{L}(0, \beta_{\theta}(\cdot))$  as  $n_{\theta,a} \sim \mathcal{N}(0,\sigma)$ , motivated by the fact that both distributions have the same mean/mode. Secondly, SAC approximates the DoubleGum state-dependent logistic spread  $\beta_{\theta}(s)$  as temperature parameter  $\beta$  learned not as a part of the critic but by itself with Lagrangian dual gradient descent. Thirdly, SAC breaks down  $\mathbb{C}[\pi_{\phi}||p_{\theta}] = \mathbb{H}[\pi_{\phi}] + \mathbb{D}_{\mathrm{KL}}[\pi_{\phi}||p_{\theta}]$  before assuming that the KL-Divergence is negligible, given that it is minimized by the actor loss. These three approximations yield the SAC noise model as
$$
Q _ {\theta} ^ {\text {n e w}} (s, a) + n _ {\theta , a} = \underset {p \left(s ^ {\prime} \mid s, a\right)} {\mathbb {E}} \left[ r + \gamma \underset {\pi_ {\phi} \left(a ^ {\prime} \mid s ^ {\prime}\right)} {\mathbb {E}} \left[ Q _ {\theta} ^ {\text {n e w}} \left(s ^ {\prime}, a ^ {\prime}\right) + \beta \log \pi_ {\phi} \left(a ^ {\prime} \mid s ^ {\prime}\right) \right] \right]. \tag {17}
$$
MLE of  $\theta$  wrt the above noise model yields the MSBE critic loss.
DDPG is a special case of SAC that assumes  $\beta \to 0$ , removing the last term in Equation 17.  $\lim_{\beta \to 0}p_{\theta}(a\mid s)$  becomes deterministic, so  $\pi_{\phi}$  may be modelled by a deterministic policy.
B.3 Interpreting the Cross-Entropy as a Pessimism Factor
In continuous control, Fujimoto et al. (2018) introduced Twin Networks, a method that improved sample-efficiency with pessimistic bootstrapped targets computed by returning a sample-wise minimum from an ensemble of two  $Q$ -functions. Follow-up work selects a quantile estimate from an ensemble (Kuznetsov et al., 2020; Chen et al., 2021; Ball et al., 2023), which we demonstrate is equivalent to estimating  $V_{\theta,\beta}^{\mathrm{soft,new}}$ .
Suppose there is an ensemble of  $n$  networks where the  $i^{\text{th}}$  network follows  $Q_{\theta_i}(s, a) = Q_\theta(s, a) + z_i(s, a)$ . Here,  $Q_\theta$  is an 'ideal' function approximator never instantiated nor computed and  $z$  is an arbitrary noise source. When  $n$  is sufficiently large,
$$
\begin{array}{l} \min  _ {i} \underset {\pi_ {\phi} (a | s)} {\mathbb {E}} [ Q _ {\theta_ {i}} (s, a) ] = \min  _ {i} \underset {\pi_ {\phi} (a | s)} {\mathbb {E}} [ Q _ {\theta} (s, a) + z _ {i} (s, a) ] = \underset {\pi_ {\phi} (a | s)} {\mathbb {E}} [ Q _ {\theta} (s, a) ] + \min  _ {i} z _ {i} (s) \\ = \underset {\pi_ {\phi} (a | s)} {\mathbb {E}} \left[ Q _ {\theta} (s, a) \right] - g (s), \text {w h e r e} g (s) \sim \mathcal {G} (\alpha (s), \beta (s)). \\ \end{array}
$$
A Gumbel random variable  $g \sim \mathcal{G}(\alpha, \beta)$  has  $\mathbb{E}[g] = \alpha + \gamma_e\beta$ , where  $\gamma_e$  is the Euler-Mascheroni constant, so for a deterministic environment the bootstrapped targets become
$$
r + \gamma_ {\pi_ {\phi} (a ^ {\prime} | s ^ {\prime})} \mathbb {E} \left[ Q _ {\theta} \left(s ^ {\prime}, a ^ {\prime}\right) \right] - \gamma \alpha \left(s ^ {\prime}\right) - \gamma \gamma_ {e} \beta \left(s ^ {\prime}\right),
$$
recovering Equation 8, the DoubleGum continuous control targets, up to an additive term  $\gamma \alpha(s')$ , while  $-\gamma \gamma_{e}\beta(s')$  recovers the spread  $\gamma c\sigma_{\theta}(s')$  up to a negative scaling factor, indicating that the default  $c$  should be negative. Moskovitz et al. (2021) and Ball et al. (2023) showed that the appropriate ensemble size and selected quantile changes the overestimation bias, so appropriate values would ensure  $\alpha(s') = 0$ .
B.4 Comparison between DoubleGum and XQL
We present an explanation of Extreme  $Q$ -Learning (XQL) as presented in Appendix C.1 of Garg et al. (2023). XQL can be derived from Soft Bellman Equation backups given by
$$
Q (s, a) \leftarrow \underset {p (s ^ {\prime} | s, a)} {\mathbb {E}} \left[ r (s, a, s ^ {\prime}) + \gamma V ^ {\mathrm {s o f t}} (s) \right], \text {w h e r e} V ^ {\mathrm {s o f t}} (s) = \beta \log \sum_ {a ^ {\prime}} \exp \left(\frac {Q (s ^ {\prime} , a ^ {\prime})}{\beta}\right)
$$
and  $\beta$  is a fixed hyperparameter. Computing the log-sum-exp of  $V^{\mathrm{soft}}$  is intractable in continuous control, as the sum over  $a^\prime$  becomes an integral in continuous control tasks.
Garg et al. (2023) present a method of estimating its value using Gumbel regression. Given a (potentially infinite) set of scalars  $x \in X$ , Gumbel regression provides a method to estimate the numerical value of log-sum-exp  $\beta(x) = \beta \log \sum_{X} \exp x / \beta$ . Gumbel regression assumes  $x \sim \mathcal{G}(\alpha, \beta)$ , where  $\mathcal{G}$  is a homoscedastic Gumbel distribution, and  $\beta$  is a fixed (hyper)parameter.  $\alpha$  estimated by MLE tends towards log-sum-exp  $\beta(x)$ . MLE is performed by numerically maximizing the log-likelihood of a Gumbel distribution, which recovers the LINear-EXponential (LINEX) loss function introduced by Varian (1975).
Garg et al. (2023) incorporate Gumbel regression into deep  $Q$ -Learning in two ways, which they name X-SAC and X-TD3. X-SAC combines Gumbel regression to estimate the soft value function used in SACv0 (Haarnoja et al., 2018a). The soft value function  $V_{\rho}^{\mathrm{soft}}(s)$  is a neural network whose parameters  $\rho$  are learned by Gumbel regression from  $Q_{\psi}(s,a)\sim \mathcal{G}(V_{\rho}(s),\beta)$ , where  $\psi$  are target network parameters. A neural network  $Q_{\theta}$  with parameters  $\theta$  may then be learned by the MSE between itself and  $\mathbb{E}_{p(s'|s,a)}[r(s,a,s') + \gamma V_{\rho}^{\mathrm{soft}}(s)]$ . X-SAC is vastly different from DoubleGum, because our algorithm does not estimate the soft value function with a separate neural network.
Gumbel regression is directly used to learn the  $Q$ -values in X-TD3. First, the bootstrapped targets are thusly rewritten
$$
\begin{array}{l} y ^ {\text {s o f t}} (s, a) = \underset {p \left(s ^ {\prime} \mid s, a\right)} {\mathbb {E}} \left[ r + \gamma \beta \log \sum_ {a ^ {\prime}} \exp \left(\frac {Q _ {\phi} \left(s ^ {\prime} , a ^ {\prime}\right)}{\beta}\right) \right] \\ = \underset {p (s ^ {\prime} | s, a)} {\mathbb {E}} \left[ \gamma \beta \log \sum_ {a ^ {\prime}} \exp \left(\frac {r + \gamma Q _ {\psi} (s ^ {\prime} , a ^ {\prime}) - Q _ {\theta} (s , a)}{\gamma \beta}\right) \right] \\ \end{array}
$$
In environments with deterministic environments, which comprise all environments considered by Garg et al. (2023) and our paper, Lemma C.1 of Garg et al. (2023) provides a method of learning the soft value function with Gumbel regression on  $y^{\mathrm{soft}}(s,a) \sim \mathcal{G}(Q_{\theta}(s,a),\gamma \beta)$ . The Gumbel regression objective used in X-TD3 to learn  $\theta$  is vastly different from the moment matching with the logistic distribution DoubleGum uses to learn  $\theta$ .
To motivate their use of Gumbel regression, Garg et al. (2023) derived a noise model which they use to present empirical evidence of homoscedastic Gumbel noise. In contrast, we presented empirical evidence of heteroscedastic logistic noise formed from a noise model with two heteroscedastic Gumbel distributions.
C A Discussion on The Convergence of DoubleGum
To the best of our knowledge, there are two types of convergence analysis in  $Q$ -Learning: 1) operator-theoretic analysis over tabular  $Q$ -functions, and 2) training dynamics of neural network parameters. We believe the second is more appropriate for DoubleGum, because our theory addresses issues in using neural networks (and not tables) for  $Q$ -learning. Nevertheless, for completeness, we discuss convergence guarantees for the tabular setting and the function approximation setting. While we can guarantee convergence for the former setting, we have no guarantees for the second.
C.1 Tabular Q-Functions
Appendices B.1 and B.2 present DoubleGum as a MaxEnt RL algorithm. When  $Q$ -functions are tabular, Appendix A of Haarnoja et al. (2018a) shows that MaxEnt RL algorithms may be derived
Algorithm 2: DoubleGum Soft Policy Iteration
Input: Finite MDP  $(S, A, r, p)$ , initial tables  $Q$ ,  $\beta$ ,  $y^{\mathrm{soft}}$ , initial policy  $\pi$   
Output: Optimal Tabular  $Q$ -function  $Q^{\star}$   
1 for training iteration  $i$  do  
2 for all  $s$  do  
3 for all  $a$  do  
4  $y^{\mathrm{soft}}(s, a) \gets \underset{p(s'|s, a)}{\mathbb{E}} \left[ r(s, a, s') + \gamma \beta_i(s') \log \sum_{a'} \exp \left( \frac{Q_i(s', a')}{\beta_i(s')} \right) \right]$   
5  $Q_{i+1}(s, a) \gets y^{\mathrm{soft}}(s, a)$   
6  $\beta_{i+1}(s) \gets \frac{\sqrt{3}}{\pi} \sqrt{\underset{a \sim \pi(a|s)}{\mathbb{V}}} [y^{\mathrm{soft}}(s, a)]$   
7 define  $\pi(a | s) \gets \frac{\exp(Q_{i+1}(s, a) / \beta_{i+1}(s))}{\sum_{a'} \exp(Q_{i+1}(s, a') / \beta_{i+1}(s))}$
from soft policy iteration. We therefore present a convergence proof for DoubleGum with tabular  $Q$ -functions based on soft policy iteration.
DoubleGum treats the return as coming from a logistic distribution and learns its location and spread. In the tabular setting, two tables would need to be learned,  $Q(s,a)$  and  $\beta (s)$ . An algorithm to learn these tables in a finite MDP with soft policy iteration is presented in Algorithm 2. Policy evaluation is done by Lines 4-6 while Line 7 performs policy improvement.
Proof of convergence of Algorithm 2 is similar to the SAC proof of convergence in Appendix B of Haarnoja et al. (2018a). This should not be surprising, given that Appendix B.2 shows SAC as a special case of DoubleGum. We first show that policy evaluation converges and that a new policy found by policy improvement does not reduce the magnitude of the value function.
Lemma 5 (Soft Policy Evaluation). Consider the Soft Policy Evaluation operator given by
$$
Q _ {i + 1} (s, a) \leftarrow \underset {p \left(s ^ {\prime} \mid s, a\right)} {\mathbb {E}} \left[ r (s, a, s ^ {\prime}) + \gamma \beta_ {i} \left(s ^ {\prime}\right) \log \sum_ {a ^ {\prime}} \exp \left(\frac {Q _ {i} \left(s ^ {\prime} , a ^ {\prime}\right)}{\beta_ {i} \left(s ^ {\prime}\right)}\right) \right] o v e r a l l (s, a) p a i r s.
$$
$\lim_{i\to \infty}Q_i$  converges to the soft  $Q$  -value.
Proof. Following Appendix A.4
$$
\beta (s) \log \sum_ {a} \exp \left(\frac {Q (s , a)}{\beta (s)}\right) = \underset {\pi (a | s)} {\mathbb {E}} [ Q _ {\theta} (s, a) ] + \beta (s) \mathbb {C} [ \pi \mid p ]
$$
$$
\text {w h e r e} p (a \mid s) = \frac {\exp (Q (s , a) / \beta (s))}{\sum_ {a ^ {\prime}} \exp (Q (s , a ^ {\prime}) / \beta (s))}
$$
the bootstrapped targets may be thusly rewritten
$$
\begin{array}{l} \underset {p \left(s ^ {\prime} \mid s, a\right)} {\mathbb {E}} \left[ r (s, a, s ^ {\prime}) + \gamma \beta \left(s ^ {\prime}\right) \log \sum_ {a ^ {\prime}} \exp \left(\frac {Q \left(s ^ {\prime} , a ^ {\prime}\right)}{\beta \left(s ^ {\prime}\right)}\right) \right] \\ = \underset {p (s ^ {\prime} | s, a)} {\mathbb {E}} \left[ r (s, a, s ^ {\prime}) + \gamma \underset {\pi (a ^ {\prime} | s ^ {\prime})} {\mathbb {E}} [ Q (s ^ {\prime}, a ^ {\prime}) ] + \beta (s ^ {\prime}) \mathbb {C} [ \pi | | p ] \right] \\ = \underset {p (s ^ {\prime} | s, a)} {\mathbb {E}} \left[ r ^ {\prime} (s, a, s ^ {\prime}) + \gamma \underset {\pi \left(a ^ {\prime} | s ^ {\prime}\right)} {\mathbb {E}} [ Q \left(s ^ {\prime}, a ^ {\prime}\right) ] \right] \\ \end{array}
$$
where  $r^{\prime}(s,a,s^{\prime}) = r(s,a,s^{\prime}) + \beta (s^{\prime})\mathbb{C}[\pi ||p]$
Following Lemma 1 in Haarnoja et al. (2018a), Sutton and Barto (1998) gives convergence of  $Q_{i + 1}(s,a)\gets \mathbb{E}_{p(s^{\prime}|s,a)}\left[r^{\prime}(s,a,s^{\prime}) + \gamma \mathbb{E}_{\pi (a^{\prime}|s^{\prime})}[Q_{i}(s^{\prime},a^{\prime})]\right]$
The proof of Soft Policy Improvement should be identical to SAC, given that Appendix B.1 shows that DoubleGum and SAC use identical actor losses. As such, Lemma 5 can be used in place of Lemma 1 in Theorem 1 of Haarnoja et al. (2018a), thus showing convergence of DoubleGum in the tabular setting.
C.2 Deep Q-Functions
Parameters of the deep  $Q$ -function used by DoubleGum in Algorithm 1 are learned by a loss function equivalent to that of heteroscedastic normal regression. Convergence of DoubleGum in the function approximation setting would therefore rely on convergence of heteroscedastic normal regression.
Zhang et al. (2023) introduces PAC-bounds for heteroscedastic normal regression, but on the condition that the mean-estimate is close to the ground truth mean, as mentioned in Paragraph 1 of Section 4. This is empirically achieved by Seitzer et al. (2022), who analyze heteroscedastic normal regression and find that the mean-estimate frequently converges to an underfitting solution. This is because the Negative Log-Likelihood (NLL) of a normal distribution is minimized when the variance becomes large – in Equation 6, this term is denoted with  $\sigma_{\theta}^{2}$ . As such, changes in  $Q_{\theta}^{\mathrm{new}}$  will not change the loss function much. To rectify this, Seitzer et al. (2022) multiplies the NLL of the normal with the numerical value of the standard deviation, reducing the dominance of  $\sigma_{\theta}$  on the loss function.
D Further Empirical Evidence for Theoretical Assumptions
D.1 Noise Distributions in Deep Q-Learning
Figure 6 presents graphs corresponding to Figure 1c for all environments considered in this paper. Continuous control results were generated from DoubleGum with default pessimism ( $c = -0.1$ ).
D.2 Adjusting The Pessimism Factor
Figure 7 presents graphs corresponding to Figure 2 for all continuous control environments considered in this paper.
E.1 Noise Distribution Discrepancy with Extreme Q-Learning
In Appendix D.2 of Page 19, Garg et al. (2023) fitted a Gumbel distribution to the TD errors on three continuous control environments. The Gumbel distribution was a good fit in two of the three environments they benchmarked on. We could not reproduce this result and attribute the discrepancy to experimental differences.
Garg et al. (2023) logged their batch of 256 TD errors once every 5,000 steps during training for 100,000 timesteps, producing  $\approx 4000$  samples which were aggregated. They also computed bootstrapped targets with online parameters. In contrast, we sample 10,000 TD errors with bootstrapped targets computed from target parameters at a single timestep instance, and we do not aggregate samples across timesteps.
E.2 Continuous Control Benchmarks and Evaluation
As mentioned in Section 6.2, the evaluation metric in continuous control was the normalized IQM with  $95\%$  stratified bootstrap confidence intervals from Agarwal et al. (2021). Returns were normalized by a minimum value computed from the mean of 100 rollouts sampled from a uniform policy and the maximum possible return from the environment. When the maximum value was not specified, we used the maximum value of any single rollout attained by any of the baselines.
We benchmarked on four continuous control suites: DeepMind Control (Tassa et al., 2018; Tunyasuvunakool et al., 2020), MuJoCo (Todorov et al., 2012; Brockman et al., 2016), MetaWorld (Yu et al., 2020), and Box2D (Brockman et al., 2016). These environments were selected to be as extensive as possible. DeepMind Control and MetaWorld were chosen because of their diversity of tasks, while the MuJoCo and Box2D environments are popular benchmarks within the common interface of OpenAI Gym (Brockman et al., 2016), now Gymnasium (Farama Foundation, 2023). No citation exists for Gymnasium as of writing this paper, and we link to their GitHub repository https://github.com/Farama-Foundation/Gymnasium as suggested in https://github.com/Farama-Foundation/Gymnasium/issues/82.
(a) Discrete Control
(b) DeepMind Control
(c) MuJoCo
(d) MetaWorld
(e) Box2D
Figure 6: Negative Log-Likelihoods (NLLs) of the noise in Deep  $Q$ -Learning under different distributions throughout training (lower is better). Mean calculated per-task  $\pm$  standard deviation. The legend for all graphs is in Figure 6e.
Figure 7: The effect of changing pessimism factor  $c$  on the target  $Q$ -value in continuous control. IQM calculated per-task  $\pm$  standard deviation. The legend for all graphs is in Figure 7d.
DeepMind Control (DMC) was designed to benchmark continuous control, over a broad range of agent morphologies. We selected agent morphologies that could be trained from states with a broad range of action spaces from 1 (acrobot) to 38 (dog). We did not benchmark on humanoid_CMU as this environment was not intended to be solved with RL from scratch, unlike the other baselines. The hardest task was selected from each of the agent morphologies. Properties of the 11 DMC tasks are presented in Table 1a.
MetaWorld was designed to have a diverse range of tasks to evaluate the generalization ability of learned policies. Each environment within MetaWorld is therefore made up of multiple tasks, all with the same underlying structure of an MDP but with different numerical values of their parameters. We follow the method of Seyde et al. (2022) to benchmark on a single MetaWorld task by first selecting an environment and then randomly selecting a set of numerical parameters. Each new instantiation of a MetaWorld task would result in a different set of hyperparameters. As such, we expect the error bars in the aggregate statistics of MetaWorld to be substantially larger than the other environments. We benchmark on tasks formed from the union of the ML1, MT10, and ML10 train tasks that a policy in MetaWorld would be trained on, as well as the five environments benchmarked in Seyde et al. (2022). Properties of the 15 MetaWorld tasks are presented in Table 1c.
MuJoCo was evaluated on the same tasks as SAC (Haarnoja et al., 2018b). These tasks were all locomotion-based. Properties of the 5 MuJoCo tasks are presented in Table 1b.
Box2D was evaluated on all continuous control tasks from states. Properties of the 2 Box2D tasks are presented in 1d.
E.3 Discrete Control Baselines
Discrete control algorithms were implemented as described in Section 3.4. Hyperparameters used in discrete control algorithms are detailed in Tables 2 and 3. We provide explanations for these design choices as follows.
DQN: The original DQN algorithm in Mnih et al. (2015) was designed for pixel inputs. We modified DQN to use state inputs by using an architecture described in Section 3.4 we used in continuous control that was popular for use with state inputs. Conversely to the continuous control architecture, we found removing GroupNorm (Wu and He, 2018) was crucial to getting DQN to work. Similarly to the continuous control architecture, we found that changing the initialization and target network updating drastically improved performance. We also used the MSE and Adam (Kingma and Ba, 2014) optimizers as Ceron and Castro (2021) showed that this yielded improved performance over the Huber Loss (Huber, 1992) and RMSProp (Hinton et al., 2012) of the original DQN. Our implementation of DQN solves classic discrete control tasks that the CleanRL (Huang et al., 2022b) reproduction of the original DQN paper at https://docs.cleanrl.dev/rl-algorithms/dqn/#experiment-results_1 could not solve.
Dueling Double DQN (Dueling DDQN) was a baseline modified from Hessel et al. (2018) designed to be as compatible with DoubleGum as possible. Rainbow evaluated six innovations to DQN: Double DQN (Van Hasselt et al., 2016), Dueling DQN (Wang et al., 2016), noisy networks (Fortunato et al., 2017),  $n$ -step returns, C51 distributional RL (Bellemare et al., 2017), and prioritized replay (Schaul et al., 2016). We only used the first two of these six innovations in DoubleGum. We did not find  $n$ -step returns effective in discrete domains we considered, nor prioritized replay. Distributional RL was incompatible with DoubleGum, while Schwarzer et al. (2023) did not find noisy networks advantageous.
DoubleDQN was implemented following Van Hasselt et al. (2016) by computing bootstrapped targets of  $Q_{\psi}^{\mathrm{new}}(s, \max_a Q_\theta^{\mathrm{new}}(s, a))$ . Dueling DQN was implemented following Wang et al. (2016), with the advantage and value heads having two layers with a hidden layer of size 256 and ReLU activations. The stability of Dueling DQN was greatly improved by setting the biases of bothueling heads to 0.
DoubleGum was implemented as Dueling DDQN with an additional variance head described in Section 3.4.
Table 1: Properties of Continuous Control Environments
(a) DeepMind Control
Environment	Task	Action Dimension	Maximum Return	Minimum Return
acrobot	swingup	1	1000	3.252
reacher	hard	2	1000	8.547
finger-turn	hard	2	1000	67.78
hopper	hop	4	1000	0.07236
fish	swim	5	1000	70.99
cheetah	run	6	1000	3.647
walker	run	6	1000	22.96
quadruped	run	12	1000	108.2
swimmer	swimmer15	14	1000	157
humanoid	run	21	1000	0.877
dog	run	38	1000	4.883
(b) MuJoCo
Task	Action Dimension	Maximum Return	Minimum Return
Hopper-v4	3	3572	18.52
HalfCheetah-v4	6	11960	-283.4
Walker2d-v4	6	5737	2.753
Ant-v4	8	6683	-60.06
Humanoid-v4	17	6829	122.5
(c) MetaWorld
Task	Action Dimension	Maximum Return	Minimum Return
button-press-v2	4	10000	187.5
door-open-v2	4	10000	277.1
drawer-close-v2	4	10000	842.5
drawer-open-v2	4	10000	631.8
peg-insert-side-v2	4	10000	8.083
pick-place-v2	4	10000	5.449
push-v2	4	10000	30.62
reach-v2	4	10000	776.1
window-open-v2	4	10000	230.3
window-close-v2	4	10000	306.7
basketball-v2	4	10000	10.2
dial-turn-v2	4	10000	125.6
sweep-into-v2	4	10000	63.41
hammer-v2	4	10000	395.1
assembly-v2	4	10000	226.3