SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(d) Box2D
Task	Action Dimension	Maximum Return	Minimum Return
BipedalWalker-v3	4	300	-99.97
BipedalWalkerHardcore-v3	4	300	-107.9
Table 2: Shared Hyperparameters of Benchmarked Algorithms
Hyperparameter	Value
Evaluation Episodes	10
Evaluation Frequency	Maximum Timesteps / 100
Discount Factor γ	0.99
n-Step Returns	1 step
Replay Ratio	1
Replay Buffer Size	1,000,000
Maximum Timesteps	1,000,000
Table 3: Hyperparameters for Discrete Control
Hyperparameter	Value
Starting Timesteps	2,000
Maximum Timesteps	100,000
Exploration	Policy Churn
Optimizer	Adam
Learning rate	3e-4
Number of groups in network GroupNorm	0
Network structure	Linear(256),ReLU,Linear(256),ReLU
Table 4: Hyperparameters for Continuous Control
Hyperparameter	Value
Starting Timesteps	10,000
Maximum Timesteps	1,000,000
Exploration Noise	0.2
Policy Noise in Critic Loss	0.1
Policy Noise in Actor Loss	0.1
Actor optimizer	Adam
Actor learning rate	3e-4
Critic optimizer	Adam
Critic learning rate	3e-4
Number of groups in Actor GroupNorm	16
Number of groups in Critic GroupNorm	16
Critic target networks EMA ηφ	5e-3
Actor target networks EMA	1
Linear(256), GroupNorm, ReLU
Critic structure	Linear(256), GroupNorm, ReLU
Linear(256), GroupNorm, ReLU
Actor structure	Linear(256), GroupNorm, ReLU
Table 5: Pessimism Hyperparameters in Continuous Control
Algorithm	Pessimism Hyperparameter
Default	DeepMind Control	MuJoCo	MetaWorld	Box2D
DoubleGum (ours)	-0.1	-0.1	-0.5	0.1	-0.1
DDPG/TD3	Twin	Single	Twin	Single	Twin
SAC	Twin	Single	Twin	Single	Twin
XQL	Twin (β = 5)	Single (3)	Single (5)	Twin (2)	Twin (5)
QR-DDPG	Single	Single	Twin	Single	Twin
FinerTD3	1	1	3	3	1
E.4 Continuous Control Baselines
Continuous control algorithms were implemented as described in Section 3.4. Hyperparameters used in continuous control algorithms are detailed in Tables 2 and 4. Pessimism hyperparameters are presented in Table 5 and were found following results in Appendix F.2.
As mentioned, all implementations used networks with two hidden layers of width 256, with orthogonal initialization (Saxe et al., 2013) and GroupNorm (Wu and He, 2018). Following Kostrikov (2021), target network parameters were updated with an EMA of  $5e - 3$  in the critic and 0 in the actor. All these design choices differ from their original implementations but improved aggregate performance. We provide explanations for these design choices as follows.
DDPG was introduced in Lillicrap et al. (2015) and Fujimoto et al. (2018) updated the design choices of DDPG to empirically improve its performance. In addition to the existing changes, our implementation uses the noise clipping scheme in the actor specified by Laskin et al. (2021).
TD3 was implemented with three changes from Fujimoto et al. (2018). First, we update the actor once per critic update - ie using a delay of 1. This is such that the only hyperparameter change between our DDPG and TD3 is the use of Twin Networks. Secondly, we update the actor to maximize the mean of two critics rather than a single critic, a design choice we found empirically reduced variance between training runs. Thirdly, we do not compute the EMA of actor-network parameters. Removing this EMA improves sample efficiency but at the cost of higher variance.
FinerTD3 (our introduced baseline) was implemented with the same hyperparameters as TD3 but with an ensemble of 5 critic networks. We chose to use 5 networks because we tuned the pessimism factor hyperparameter of DoubleGum over 5 values. The 5 critics in FinerTD3 enable five values of pessimism to be used. Pessimism of FinerTD3 is adjusted in the bootstrapped targets. The 5 critic values are sorted by decreasing positivity, and the  $i^{\text{th}}$  smallest value is used as the target critic value in the bootstrapped targets.
SAC was implemented with hyperparameters from Kostrikov (2021), which we found improved performance. Kostrikov (2021) differs from Haarnoja et al. (2018b) in two additional ways from those mentioned. The standard deviation in the actor was clipped to  $[-10, 2]$ , and the target entropy was the action dimension divided by 2 instead of just the action dimension.
XQL Garg et al. (2023) presents two off-policy algorithms: X-TD3 and X-SAC. We use X-TD3 to be consistent with the DDPG fixed-variance actor of DoubleGum and refer to it throughout as XQL. XQL tunes two hyperparameters per task: the use of twin networks/not and scalar hyperparameter  $\beta$ . We swept over the same  $\beta$ -values as Garg et al. (2023): 1, 2, 5 without Twin Critics and 3, 4, 10 and 20 with Twin Critics.  $\beta$  was tuned in the same way as pessimism - we found a default  $\beta$  value and a  $\beta$  tuned per-suite.  $\beta$  values are presented in Table 5 and were found following results in Appendix F.2.
MoG-DDPG is formed by combining a Mixture-of-Gaussians (MoG) critic with DDPG. The MoG critic was introduced in Appendix A of Barth-Maron et al. (2018) and improved by Shahriari et al. (2022). The latter paper combines the MoG critic with DDPG with distributed training, but we remove the distributed training component because we do not use it in DoubleGum.
QR-DDPG (our introduced baseline) combines the quantile regression method of Dabney et al. (2018b) with a DDPG actor. Although Ma et al. (2020); Wurman et al. (2022) and Teng et al. (2022) have combined quantile regression with SAC, we combine it with DDPG because DoubleGum is built on top of DDPG. Like Dabney et al. (2018b), we use 201 quantiles, but these are initialized with orthogonal initialization and are optimized with the MSE, rather than the Huber loss. QR was developed for discrete control and uses the Huber loss with the RMSProp optimizer popular in discrete control methods. We found better performance with the MSE and Adam optimizer, perhaps confirming the result of Ceron and Castro (2021) in distributional RL for continuous control.
DoubleGum was implemented as DDPG with a variance head described in Section 3.4.
E.5 Compute Requirements
A single training run for discrete control may take up to 3 to 5 minutes on a laptop with an Intel Core i9 CPU, NVIDIA 1050 GPU and 31.0 GiB of RAM. On the same system, a single training run for continuous control takes 1 - 2 hours.
Table 6: Discrete Control Numerical Results
Task	Score at 100K timesteps (IQM over 12 seeds)
DoubleGum (ours)	DQN	DuelingDDQN
CartPole-v1	500 ± 113.4	475 ± 105.5	496.9 ± 89.1
Acrobot-v1	-62.78 ± 1.775	-73.52 ± 5.191	-64.12 ± 17.15
MountainCar-v0	-98.17 ± 2.45	-99.37 ± 5.914	-98.75 ± 30.73
The overwhelming majority of our experiments were run on private infrastructure. This cluster had a mixture of Intel Broadwell, Skylake, Cascade Lake, AMD Rome, AMD Milan CPUs, and NVIDIA P100s, V100s, and A100s GPUs. Benchmarking continuous control took roughly ten times longer than benchmarking discrete control. Multi-threaded experiments for continuous control running twelve seeds in parallel took 5 - 8 hours. 8 algorithms (DoubleGum, DDPG, TD3, MoG-Critics, SAC, XQL, QR-DDPG, FinerTD3) were benchmarked over 33 continuous control environments, and there were further runs for hyperparameter sweeps (4 for DoubleGum, 1 for SAC, 6 for XQL, 1 for QR-DDPG and 4 for FinerTD3), yielding 24 runs in total. These algorithms were run at least 10 times for development and hyperparameter tuning. This yields a lower bound of  $8 \times 33 \times 24 \times 10 = 63360$  hours (7.23 years) of computation.
Assuming that all experiments were run on Tesla V100-SXM2-16GB (TDP of 250W), the cluster it was run on had a carbon efficiency of  $0.0006\mathrm{kgCO_2eq / kWh}$  (that of the surrounding power grid) and that there were 63360 hours of cumulative computation, the total emissions were  $9.51\mathrm{kgCO_2eq}$ , equivalent to driving  $36\mathrm{km}$  in an average car. Estimations were conducted using the MachineLearning Impact calculator presented in Lacoste et al. (2019).
F Further Results
F.1 Adjusting the Pessimism of DoubleGum
Figure 8 shows that sample efficiency is sensitive to the pessimism factor  $c$  adjusting pessimism per suite greatly impacts sample efficiency. The best performing  $c$  was  $c = -0.1$ , and was thus set as the default pessimism factor value.
Figure 9 shows that the performance of DoubleGum may be improved when the degree of pessimism is changed per suite. This graph was used to determine what pessimism factor to use in each suite, whose values are reported in Table 5.
F.2 Adjusting the Pessimism of Baseline Algorithms
This section presents graphs used to determine which pessimism values to use for baseline algorithms. All final values are reported in Table 5.
Figure 10 shows that sample efficiency is sensitive to the use of pessimism determined by the use of Twin Networks/not. In aggregate, each method was improved by using Twin Networks. Twin networks were therefore set as the default pessimism option for all baseline algorithms apart from QR-DDDPG, because Twin Networks was not used with quantile regression in (Dabney et al., 2018b). Figure 11 was used to determine whether to use Twin Networks/not on a per suite basis.
Similarly, Figures 14 and 15 were respectively used to determine pessimism hyperparameters for FinerTD3. In these two graphs, numbers refer to the  $i^{\text{th}}$  smallest value returned by the ensemble of target critics. Finally, Figures 12 and 13 were respectively used to determine pessimism and  $\beta$  hyperparameters for XQL.
F.3 Discrete Control
Table 6 presents results for discrete control at 100K timesteps.
Figure 8: Adjusting the pessimism factor  $c$  in DoubleGum, IQM normalized score over 33 tasks in 42 suites with  $95\%$  stratified bootstrap CIs.
Figure 9: Adjusting pessimism in DoubleGum, per-suite IQM with  $95\%$  stratified bootstrap CIs.
Figure 10: Adjusting pessimism of baseline algorithms with the use of Twin Networks/not, IQM normalized score over 33 tasks in 4 suites with  $95\%$  stratified bootstrap CIs. Methods that default to use Twin Networks are dashed.
Figure 11: Adjusting pessimism of baseline algorithms with the use of Twin Networks/not, per-suite IQM normalized score with  $95\%$  stratified bootstrap CIs. Methods that default to use Twin Networks are dashed.
Figure 12: Adjusting pessimism of XQL, IQM normalized score over 33 tasks in 4 suites with  $95\%$  stratified bootstrap CIs. Methods that use Twin Networks are dashed.
Timesteps (in millions)
Figure 13: Adjusting pessimism of XQL, per-suite IQM normalized score with  $95\%$  stratified bootstrap CIs. Methods that use Twin Networks are dashed.
Figure 14: Adjusting pessimism of FinerTD3, IQM normalized score over 33 tasks in 4 suites with  $95\%$  stratified bootstrap CIs.
Figure 15: Adjusting pessimism of FinerTD3, per-suite IQM normalized score with  $95\%$  stratified bootstrap CIs.
Figure 16: Continuous control with default parameters, per-suite IQM normalized score with  $95\%$  stratified bootstrap CIs. Methods that default to use Twin Networks are dashed.
Figure 17: Continuous control with the best pessimism hyperparameters tuned per suite, per-suite IQM normalized score with  $95\%$  stratified bootstrap CIs.
F.4 Continuous Control with Default Pessimism
Figures 16 and 18 respectively present aggregate per-suite and per-task results of DoubleGum benchmarked against baseline algorithms with default pessimism values.
Table 7 presents the performance of all algorithms with default pessimism at 1 million timesteps. This graph has six subsections. The first four subsections present per-task results from DeepMind Control, MuJoCo, MetaWorld, and Box2D, respectively, corresponding to results from 18. The next subsection presents per-suite aggregate results, corresponding to Figure 16, while the last subsection presents aggregate results over all tasks and suites, corresponding to Figure 1. In aggregate results, only the IQM is reported.
F.5 Continuous Control with Pessimism adjusted Per-Suite
Figures 17 and 19 respectively present aggregate per-suite and per-task results of DoubleGum benchmarked against baseline algorithms with the best pessimism values adjusted per-suite.
Table 8 presents the performance of all algorithms with default pessimism at 1 million timesteps. This graph has six subsections. The first four subsections present per-task results from DeepMind Control, MuJoCo, MetaWorld, and Box2D, respectively, corresponding to results from 19. The next subsection presents per-suite aggregate results, corresponding to Figure 17, while the last subsection presents aggregate results over all tasks and suites, corresponding to Figure 1. In aggregate results, only the IQM is reported.
(a) DeepMind Control
(b) MuJoCo
Timesteps (in millions)
(c) MetaWorld
No
Figure 18: Continuous control with default pessimism hyperparameters, per-task IQM ± standard deviation. Methods that default to use Twin Networks are dashed. The legend for all graphs is in Figure 18d.
Timesteps (in millions)
(d) Box2D
DoubleGum, c = -0.1 (Ours)	Twin-SAC
MoG-DDPG	Twin-XQL
DDPG	QR-DDPG
TD3	FinerTD3
(c) MetaWorld
Timesteps (in millions)
(d) Box2D
Figure 19: Continuous control with the best pessimism hyperparameters adjusted per suite, per-task IQM ± standard deviation. The legend for all graphs is in Figure 19d.