SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4.2 Adjusting The Pessimism Factor
Figure 2 plots a 12-sample IQM and standard deviation over  $\frac{1}{256}\sum_{i=1}^{256}Q_{\psi}^{\mathrm{new}}(s_i,a_i)$ , the average magnitude of the target  $Q$ -value used in bootstrapped targets. Figure 2 and Appendix D.2 show that the average magnitude increases as the pessimism factor  $c$  increases, validating its effectiveness.
5.1 Theoretical Analyses of the Noise in Q-Learning
Logistic Q-Learning Bas-Serrano et al. (2021) presents a similar noise model to DoubleGum but with a homoscedastic logistic instead of a heteroscedastic logistic in Equation 5. Their distribution is derived from the linear-programming perspective of value-function learning described in Section 6.9 of Puterman (2014)) and Peters et al. (2010). While DoubleGum learns a  $Q$ -function off-policy, Logistic  $Q$ -Learning uses on-policy rollouts to compute the  $Q$ -values of the current policy. We do not benchmark against Logistic  $Q$ -Learning because their method is on-policy and only uses linear function approximation.
Extreme Q-Learning (XQL) Garg et al. (2023) presents a noise model for Deep  $Q$ -Learning with one homoscedastic Gumbel noise source, as opposed to the two heteroscedastic Gumbels in Equation 4. Parameters are learned by the LINear-EXponential (LINEX) loss Varian (1975) formed from the log-likelihood of a Gumbel distribution. XQL is presented in more detail in Appendix B.4.
Gumbel Noise in Deep Q-Learning: Thrun and Schwartz (1993) argued that the max-operator in bootstrapped targets transformed zero-mean noise from function approximation into statistically biased noise. The authors did not recognize that the resultant noise distribution was Gumbel, and this was realized by Lee and Powell (2012). Unlike DoubleGum, these two works did not make assumptions about the distribution of function approximator noise but instead focused on mitigating the overestimation bias of bootstrapped targets. In economics, McFadden et al. (1973) and Rust (1994) assume the presence of Gumbel noise from noisy reward observations (unlike DoubleGum, which assumes that noise comes from function approximation) and derives the soft value function we present in Appendix A.1 for static and dynamic discrete choice models. XQL brings the Rust-McFadden et al. model to deep reinforcement learning to tackle continuous control.
The soft value function was introduced to model stochastic policies (Rummery and Niranan, 1994; Fox et al., 2015; Haarnoja et al., 2017, 2018b). The most prominent algorithm that uses the soft value function is Soft Actor-Critic (SAC) (Haarnoja et al., 2018a,b), which Appendix B.2 shows is a special case of DoubleGum when  $Q_{\theta}^{\mathrm{new}}$  is learned by homoscedastic instead of heteroscedastic normal regression and the spread is a tuned scalar parameter instead of a learned state-dependent standard deviation. Appendix B.2 also shows that Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al., 2015) is a simpler special case of SAC that has recently been shown to outperform and train more stably than SAC (Yarats et al., 2021).
5.2 Empirically Similar Methods to DoubleGum
Distributional RL models the bootstrapped targets and  $Q$ -function as distributions. In these methods, a  $Q$ -function is learned by minimizing the divergence between itself and a target distribution. The most similar distributional RL method to DoubleGum is Mixture-of-Gaussian (MoG) Critics, introduced in Appendix A of Barth-Maron et al. (2018). DoubleGum is a special case of MoG-Critics with only one Gaussian (such as in Morimura et al. (2012)) and mean samples of the target distribution. Curiously, Shahriari et al. (2022) shows that sample-efficiency of training improves when bootstrapped target sampled are increasingly near the mean but did not try mean sampling. Nevertheless, Shahriari et al. (2022) show that MoG-Critics outperforms a baseline with the C51 distributional head (Bellemare et al., 2017) popular in discrete control, obtaining state-of-the-art results in DeepMind Control. In discrete control with distributional RL, C51 has been superseded by Quantile Regression (QR) methods (Dabney et al., 2018b,a; Yang et al., 2019) that predict quantile estimates of a distribution. Ma et al. (2020); Wurman et al. (2022) and Teng et al. (2022) apply QR to continuous control.
Adjusting pessimism greatly improves the sample efficiency of RL. Fujimoto et al. (2018) showed empirical evidence of overestimation bias in continuous control and mitigated it with pessimistic estimates computed by Twin Networks. However, Twin Networks may sometimes harm sample efficiency because the optimal degree of pessimism varies across environments. To address this, the degree of pessimism is adjusted during training (Lan et al., 2020; Wang et al., 2021; Karimpanal et al., 2021; Moskovitz et al., 2021; Kuznetsov et al., 2020, 2022; Ball et al., 2023), often with the help of an ensemble. In contrast, DoubleGum uses one  $Q$ -network and one scalar hyperparameter fixed throughout training.
Figure 3: Discrete control, IQM of returns  $\pm$  standard deviation over 12 seeds.
Many other RL methods add or subtract the learned standard deviation to bootstrapped targets. Risk-Aware RL subtracts a learned standard deviation to learn a low-variance policy (La and Ghavamzadeh, 2013; Tamar and Mannor, 2013; Tamar et al., 2016). Upper-Confidence Bounded (UCB) methods add a learned standard deviation to explore high-variance regions (Lee et al., 2021; Teng et al., 2022). These methods use a combination of ensembles and variance networks, but it is also possible to derive a Bellman equation for the variance following Dearden et al. (1998). The current state-of-the-art method in RL with variance estimation is Mai et al. (2022), which uses both variance networks and ensembles. The use of ensembles is motivated by the need to capture model uncertainty – differences between  $Q$ -functions with different parameters. We believe that ensembles are unnecessary because model uncertainty will be expressed in bootstrapped targets from two different timesteps as parameters change through learning and that all variation in bootstrapped targets will henceforth be captured by variance networks.
6.1 Discrete Control
We benchmarked DoubleGum on classic discrete control tasks against two baselines from the DQN family of algorithms (Figure 3). All algorithms were implemented following Section 3.4. Appendix E.3 discusses the baseline algorithms in more detail.
Performance was evaluated by the InterQuartile Mean (IQM) over 12 runs, each one with a different randomly initialized seed. Agarwal et al. (2021) showed that the IQM was a robust performance metric in RL. 12 was chosen because it was the smallest multiple of four (so a IQM could be computed) greater than the 10 seeds recommended by Henderson et al. (2018). In each run, the policy was evaluated by taking the mean of 10 rollouts every 1000 timesteps, following Fujimoto et al. (2018).
Figure 3 shows that DoubleGum sometimes obtains better sample efficiency than baselines, but not significantly more to necessitate further discrete control experiments. In the remainder of this work, we focus on continuous control, where we found DoubleGum to be more effective.
6.2 Continuous Control
We present two modes of evaluating DoubleGum because our algorithm has a pessimism factor hyperparameter we choose to tune per suite. We, therefore, benchmark all continuous control algorithms with default pessimism (Figure 4) and without pessimism-tuning per-suite (Figure 5). Table 5 presents default and per-suite pessimism values.
We compare against seven baselines. The first five are popular algorithms in the continuous control literature: DDPG (Lillicrap et al., 2015), TD3 (Fujimoto et al., 2018), SAC (Haarnoja et al., 2018b), MoG-Critics (Shahriari et al., 2022) and XQL (Garg et al., 2023). Here, DDPG, TD3, and SAC represent MaxEnt RL, MoG-Critics represent Distributional RL, and XQL is the algorithm with the most similar noise model to ours. We introduce two further baselines: QR-DDDPG, a stronger Distributional RL baseline that combines quantile regression (Dabney et al., 2018b) with DDPG, and FinerTD3, TD3 with an ensemble of five networks as opposed to a network of two ensembles
Figure 4: Continuous control with default parameters, IQM normalized score over 33 tasks in 4 suites with  $95\%$  stratified bootstrap CIs. Methods that default to use Twin Networks are dashed. Appendix F.4 presents per-suite and per-task aggregate results.
Figure 5: Continuous control with the best pessimism hyperparameters tuned per suite, IQM normalized score over 33 tasks in 4 suites with  $95\%$  stratified bootstrap CIs. Appendix F.5 presents per-suite and per-task aggregate results.
in the original TD3 that enables finer control over the degree of pessimism with an ensemble of five networks. All algorithms were implemented following design decisions outlined in Section 3.4. Appendix E.4 discusses the algorithms in more detail.
The pessimism of DoubleGum was tuned by changing its pessimism factor. The pessimism of baseline algorithms was tuned by manually choosing whether to use Twin Networks (Fujimoto et al., 2018) or not. Note that we refer to DDPG with Twin Networks as TD3. The pessimism of FinerTD3 was tuned by selecting which quantile estimate to use from an ensemble of five networks. The pessimism of MoG-Critics could not be tuned because its critic does not support paired sampling between ensemble members. Appendices F.1 and F.2 respectively detail how the pessimisms of DoubleGum and baseline algorithms were adjusted.
We benchmarked DoubleGum on 33 tasks over 4 continuous control suites comprising respectively of 11 DeepMind Control (DMC) tasks (Tassa et al., 2018; Tunyasuvunakool et al., 2020), 5 MuJoCo tasks (Todorov et al., 2012; Brockman et al., 2016), 15 MetaWorld tasks (Yu et al., 2020) and 2 Box2D tasks (Brockman et al., 2016). We follow Agarwal et al. (2021) and evaluate performance with the normalized IQM with  $95\%$  stratified bootstrap confidence intervals aggregated over all 33 tasks. 12 runs from each task was collected by a similar method to that described in Section 6.1. Further details of the tasks and their aggregate metric are detailed in Appendix E.2.
Figure 4 shows that DoubleGum outperformed all baselines in aggregate over 33 tasks when all algorithms used their default pessimism settings. Figure 5 shows that DoubleGum outperformed all baselines in aggregate over 33 tasks when the pessimism of all algorithms is adjusted per suite. Comparing the figures shows that adjusting the pessimism of DoubleGum per suite also attained a higher aggregate score than DoubleGum with default pessimism.
This paper studied the noise distribution in Deep  $Q$ -Learning from first principles. We derived a noise model for Deep  $Q$ -Learning that used two heteroscedastic Gumbel distributions. Converting our noise model into an algorithm yielded DoubleGum, an off-policy Deep  $Q$ -Learning algorithm applied to both discrete and continuous control.
In discrete control, our algorithm attained competitive performance to the baselines. Despite having an numerically exact loss function in discrete control, DoubleGum was very sensitive to hyperparameters.
Practically, using Dueling DQN (Wang et al., 2016) to learn a  $Q$ -function was crucial to getting DoubleGum to work. Appendix D.1 shows that our noise model fits the underlying noise distribution of Deep  $Q$ -Learning and we therefore suspect that instability in discrete control might be due to the training dynamics of deep learning and not our theory.
In continuous control, we introduced a pessimism factor hyperparameter to approximate our otherwise intractable noise model. We provided a default value for the pessimism factor that outperformed popular  $Q$ -Learning baselines in aggregate over 33 tasks. Tuning this hyperparameter yielded even greater empirical gains. Our method of tuning pessimism was more computationally efficient and finer-grained than popular methods that tuned a quantile estimate from an ensemble.
In continuous control, DoubleGum outperformed all baselines in aggregate. We hypothesize that DoubleGum outperformed MaxEnt RL baselines because DoubleGum is a more expressive generalization of SAC, which is itself more expressive than DDPG as shown in Appendix B.2. Our theory showed that TD-errors follow a heteroscedastic Logistic, and we believe that modeling this distribution should be sufficient for distributional RL. We hypothesize that more complex distributions considered by the Distributional RL methods QR and MoG overfit to the replay buffer and might not generalize well to online rollouts. FinerTD3 performs marginally poorer than DoubleGum, even when pessimism was adjusted per-suite. We believe this is because FinerTD3 adjusts pessimism finer than other baselines, but still not as fine as the continuous scalar in DoubleGum. Finally, we hypothesize that DoubleGum outperformed XQL because our noise model better fits the underlying noise distribution in Deep Q-Learning, as shown in Appendix D.1.
This paper shows that better empirical performance in Deep RL may be attained through a better understanding of theory. To summarize, we hope that our work encourages the community to increase focus on reducing the gap between theory and practice to create reinforcement learning algorithms that train stably across a wide variety of environments.
Theoretically, our work lacks a convergence guarantee for DoubleGum. This is exceptionally challenging because to the best of our knowledge there are currently no convergence guarantees for heteroscedastic regression. Appendix C discusses convergence in more detail.
Experimentally, there are many areas left open for future work. For speed in proof of concept experiments, we only swept over five values for the pessimism factor hyperparameter and only tuned per-suite. We anticipate that a more thorough search of the pessimism factor combined with a per-task selection will improve our results even further. An obvious follow-up would be an algorithm that automatically adjusted the pessimism factor during training. Additionally, we only focused on environments with state observations to not deal with representation learning from visual inputs. Another obvious next step would be to train DoubleGum on visual inputs or POMDPs.
We would like to thank Clement Gehring, Victor Hui, Sobhan Mohammadpour, Tianwei Ni, Anushree Rankawat, Fabrice Normandin, Ryan D'Orazio, Jesse Farebrother, Valentin Thomas, Matthieu Geist, Olivier Pietquin, Emma Frejinger, and the five anonymous reviewers.
All algorithms and code were implemented in JAX (Bradbury et al., 2018) and Haiku (Hennigan et al., 2020) and cpprb (Yamada, 2019). Helper functions for running RL experiments were taken from Kostrikov (2021); Wu (2021); Hoffman et al. (2020) and Agarwal et al. (2021). Compute requirements are listed in Appendix E.5.
This research was funded by CIFAR and Hitachi and enabled in part by compute resources, software and technical help provided by Mila (mila.quebec), Calcul Québec (calculquebec.ca) and the Digital Research Alliance of Canada (alliance.can.ca).
Hui carried out the majority of his work in Section J of Mila and would like to express his wish for the silent (but vibrant) working conditions there to continue for many years to come.
A Proofs 17
A.1 The SoftMax Location of Gumbel Distribution 17
A.2 Shifting the Value Function 18
A.3 The Difference between Two Gumbel Random Variables is a Logistic 18
A.4 Soft Q-Learning Identity 19
B Further Theory and Derivations 19
B.1 Actor Loss 19
B.2 Maximum-Entropy Reinforcement Learning 20
B.3 Interpreting the Cross-Entropy as a Pessimism Factor 20
B.4 Comparison between DoubleGum and XQL 21
C A Discussion on The Convergence of DoubleGum 21
C.1 Tabular Q-Functions 21
C.2 Deep Q-Functions 23
D Further Empirical Evidence for Theoretical Assumptions 23
D.1 Noise Distributions in Deep Q-Learning 23
D.2 Adjusting The Pessimism Factor 23
E Further Experimental Details 23
E.1 Noise Distribution Discrepancy with Extreme Q-Learning 23
E.2 Continuous Control Benchmarks and Evaluation 23
E.3 Discrete Control Baselines 26
E.4 Continuous Control Baselines 29
E.5 Compute Requirements 29
F Further Results 30
F.1 Adjusting the Pessimism of DoubleGum 30
F.2 Adjusting the Pessimism of Baseline Algorithms 30
F.3 Discrete Control 30
F.4 Continuous Control with Default Pessimism 33
F.5 Continuous Control with Pessimism adjusted Per-Suite 33
A.1 The SoftMax Location of Gumbel Distribution
We refer to the log-sum-exp operator as the SoftMax operator. This is not the same-named operator in Bridle (1989, 1990), which we suggest should be (re-)named SoftArgMax.
Theorem 1.
$$
\max  _ {i} \left[ \alpha_ {i} + g _ {i} \right] = \beta \log \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta}\right) + g, \text {w h e r e} g, g _ {i} \sim \mathcal {G} (0, \beta)
$$
where  $\mathcal{G}$  is a Gumbel distribution. A Gumbel random variable  $g\sim \mathcal{G}(\alpha ,\beta)$  specified by location  $\alpha \in \mathbb{R}$  and spread  $\beta >0$  has PDF  $p(g) = \frac{1}{\beta}\exp (-z - \exp (-z))$  with  $z = (g - \alpha) / \beta$  and CDF  $P(g) = \frac{1}{\beta}\exp (-\exp (-z))$  (Gumbel, 1935).
Proof. First note
$$
\alpha_ {i} + g _ {i}, \quad g _ {i} \sim \mathcal {G} (0, \beta) \Rightarrow \alpha_ {i} + g _ {i} \sim \mathcal {G} _ {i} (\alpha_ {i}, \beta)
$$
Then, denoting  $y = \max_{i}[\alpha_{i} + g_{i}]$
$$
\begin{array}{l} P \left(X \leqslant y\right) = \prod_ {i} P _ {i} (X \leqslant y) \\ = \prod_ {i} \mathcal {G} _ {i} (X \leqslant y) \\ = \prod_ {i} \exp \left(- \exp \left(- \frac {x - \alpha_ {i}}{\beta}\right)\right) \\ = \exp \left(- \sum_ {i} \exp \left(- \frac {x - \alpha_ {i}}{\beta}\right)\right) \\ = \exp \left(- \exp \left(- \frac {x}{\beta}\right) \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta}\right)\right) \\ = \exp \left(- \exp \left(- \frac {x}{\beta} + \log \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta}\right)\right)\right) \\ = \exp \left(- \exp \left(- \frac {1}{\beta} \left(x - \beta \log \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta}\right)\right)\right)\right) \\ = \mathcal {G} \left(\beta \log \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta}\right), \beta\right) \\ = \beta \log \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta}\right) + g, \text {w h e r e} g \sim \mathcal {G} (0, \beta) \\ \end{array}
$$
When applied to discrete  $Q$ -Learning, we produce
$$
\max  _ {\mathcal {A}} \left[ Q _ {\theta} (s, a) + g _ {\theta , a} (s) \right] = \beta_ {\theta} (s) \log \int_ {\mathcal {A} _ {s}} \exp \left(\frac {Q _ {\theta} (s , a)}{\beta_ {\theta} (s)}\right) \mathrm {d} a + g _ {\theta} (s) \tag {10}
$$
where  $g_{\theta ,a}(\cdot),g_{\theta}(\cdot)\sim \mathcal{G}\left(0,\beta_{\theta}(\cdot)\right)$  , for all  $a\in \mathcal{A}$
We assume that the same result holds for the continuous case if  $|\mathcal{A}| < \infty$ , an assumption first used in Lemma 1, Appendix B.1, Page 11 of Haarnoja et al. (2018a) to ensure boundedness. Here, we require the output of the max-operator to be bounded, which cannot be the case when the number of its arguments is  $\infty$ .
A.2 Shifting the Value Function
Theorem 2.
$$
\beta \log \sum_ {i} \exp \left(\frac {\alpha_ {i} + g}{\beta}\right) = \beta \log \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta}\right) + g
$$
$$
\begin{array}{l} \beta \log \sum_ {i} \exp \left(\frac {\alpha_ {i} + g}{\beta}\right) = \beta \log \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta} + \frac {g}{\beta}\right) (11) \\ = \beta \log \exp \left(\frac {g}{\beta}\right) \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta}\right) (12) \\ = \beta \log \exp \left(\frac {g}{\beta}\right) + \beta \log \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta}\right) (13) \\ = \beta \log \sum_ {i} \exp \left(\frac {\alpha_ {i}}{\beta}\right) + g (14) \\ \end{array}
$$
A.3 The Difference between Two Gumbel Random Variables is a Logistic
Theorem 3.
$$
x _ {1}, x _ {2} \sim \mathcal {G} (0, \beta) \Rightarrow z = x _ {1} - x _ {2}, \quad z \sim \mathcal {L} (0, \beta)
$$
where  $\mathcal{L}$  is a logistic distribution. A logistic random variable  $l\sim L(\alpha ,\beta)$  with location  $\alpha$  and spread  $\beta$  has PDF  $\frac{\exp(-(l - \alpha) / \beta)}{\beta(1 + \exp(-(l - \alpha) / \beta))^2}$  and CDF  $\frac{1}{1 + \exp(-(l - \alpha) / \beta)}$ .
Proof. First, construct the convolution based on the joint PDF
$$
\begin{array}{l} \mathcal {L} (z) = P (Z \leqslant z) \\ = P \left(X _ {1} - X _ {2} \leqslant z\right) \\ = P \left(X _ {1} \leqslant z + x _ {2}\right) \\ = \int_ {- \infty} ^ {\infty} \int_ {- \infty} ^ {z + x _ {2}} g (x _ {1}) g (x _ {2}) d x _ {1} d x _ {2} \\ = \int_ {- \infty} ^ {\infty} G (z + x _ {2}) g (x _ {2}) \mathrm {d} x _ {2}. \\ \end{array}
$$
Rewriting  $x_{2}$  as  $x$  yields
$$
\begin{array}{l} \mathcal {L} (z) = \int_ {- \infty} ^ {\infty} G (z + x) g (x) d x \\ = \int_ {- \infty} ^ {\infty} \exp \left(\exp - \frac {z + x}{\beta}\right) \exp \left(- \frac {x}{\beta} - \exp - \frac {x}{\beta}\right) d x \\ = \int_ {- \infty} ^ {\infty} \exp \left(- e ^ {- \frac {x}{\beta}} \left(1 + e ^ {- \frac {z}{\beta}}\right)\right) e ^ {- \frac {x}{\beta}} d x \\ = \int_ {0} ^ {\infty} \frac {1}{\beta} \exp \left(- u \left(1 + e ^ {- \frac {z}{\beta}}\right)\right) d u, \text {w h e r e} u = e ^ {- \frac {x}{\beta}}, d u = - \frac {1}{\beta} e ^ {- \frac {z}{\beta}} d x \\ = \left. \frac {1}{\beta} \frac {1}{1 + e ^ {- \frac {z}{\beta}}} e ^ {\left(- u \left(1 + e ^ {- \frac {z}{\beta}}\right)\right)} \right| _ {0} ^ {\infty} \\ = \frac {1}{1 + e ^ {- \frac {z}{\beta}}} . \\ \end{array}
$$
which is the CDF of  $\mathcal{L}(0,\beta)$
A.4 Soft Q-Learning Identity
Theorem 4. For an arbitrary  $p(x)$
$$
\beta \log \int \exp \left(\frac {E (x)}{\beta}\right) d x = \underset {x \sim p (\cdot)} {\mathbb {E}} [ E (x) ] + \beta \mathbb {C} [ p | | p ^ {\star} ], w h e r e p ^ {\star} (x) = \frac {\exp \frac {E (x)}{\beta}}{\int \exp \frac {E (x)}{\beta} d x}
$$
$$
\begin{array}{l} \underset {p (x)} {\mathbb {E}} [ E (x) ] + \beta \mathbb {C} [ p \mid | p ^ {\star} ] = \underset {p (x)} {\mathbb {E}} [ E (x) ] - \beta \int p (x) \log \frac {\exp \frac {E (x)}{\beta}}{\int \exp \frac {E \left(x ^ {\prime}\right)}{\beta} \mathrm {d} x ^ {\prime}} \mathrm {d} x \\ = \beta \int p (x) \log \int \exp \frac {E \left(x ^ {\prime}\right)}{\beta} d x ^ {\prime} d x \\ = \beta \log \int \exp \frac {E (x)}{\beta} d x \\ \end{array}
$$
When applied to  $Q$ -Learning, the following identity produces
$$
\begin{array}{l} \beta (s) \log \int \exp \left(\frac {Q _ {\theta} (s , a)}{\beta (s)}\right) d a = \underset {\pi_ {\phi} \left(a ^ {\prime} \mid s ^ {\prime}\right)} {\mathbb {E}} \left[ Q _ {\theta} (s, a) \right] + \beta (s) \mathbb {C} \left[ \pi_ {\phi} \right| | p _ {\theta} ] \\ \text {w h e r e} p _ {\theta} (a \mid s) = \frac {\exp \frac {Q _ {\theta} (s , a)}{\beta (s)}}{\int \exp \frac {Q _ {\theta} (s , a ^ {\prime})}{\beta (s)} d a ^ {\prime}} \\ \end{array}
$$
B Further Theory and Derivations
B.1 Actor Loss
The actor losses used in DoubleGum, SAC, and DDPG are all derived from the same principle. For a given  $s$ , the actor loss function should minimize the following (reverse) KL-Divergence, previously presented in Equation 7.
$$
\min  _ {\phi} \beta \mathbb {D} _ {\mathrm {K L}} \left[ \pi_ {\phi} \mid | p _ {\theta} \right], \text {w h e r e} p _ {\theta} (a \mid s) = \frac {\exp Q _ {\theta} ^ {\text {n e w}} (s , a) / \beta}{\int \exp Q _ {\theta} ^ {\text {n e w}} (s , a ^ {\prime}) / \beta \mathrm {d} a ^ {\prime}}.
$$
This simplifies as