SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 8: Performance of DDQN and SUPER-DDQN (gaussian experience selection, target bandwidth 0.1) for differing hyperparameter settings of the underlying DDQN algorithm. Top: Different learning rates and rollout fragment lengths in Pursuit. Bottom: Different exploration settings in Pursuit and co-evolving variants of Batle and Adversarial-Pursuit. Hyperparameters otherwise identical to those used in Figure 1. Performance measured at 1M timesteps in Pursuit, 300k timesteps in Battle, 400k timesteps in Adversarial-Pursuit.
Figure 9: Performance of quantile SUPER vs share-all and uniform random experience sharing in Pursuit at 800k timesteps.
D Additional Analysis of Bandwidth Sensitivity
We present here a more detailed analysis of bandwidth sensitivity of SUPER-DDQN in the three experience selection modes we discuss in the main text. Figure 10 shows the mean performance across five seeds for gaussian (left), quantile (middle) and stochastic (right) experience selection, at 1-2M timesteps (top) and at 250k timesteps (bottom). We can see that at 1-2M timesteps and a target bandwidth of 0.1, all three experience selection criteria perform similar. One thing that stands out is that stochastic selection has much lower performance at other target bandwidths, and also much less performance uplift compared to no-sharing DDQN at 250k timesteps at any bandwidth. Gaussian experience selection appears to be less sensitive to target bandwidth, but upon closer analysis we found that it also was much less responsive in terms of how much actual bandwidth it used at different
Figure 10: Performance of SUPER with different experience selection and varying bandwidth in Pursuit at 1-2M timesteps (top) and at 250k timesteps (bottom).
settings. Figure 11 (left) shows the actual bandwidth used by each selection criterion at different target bandwidths. We can see that quantile and stochastic experience hit their target bandwidth very well in general. $^{5}$  What stands out, however, is that gaussian selection vastly overshoots the target bandwidth at lower settings, never going significantly below 0.01 actual bandwidth.
What is a fairer comparison therefore is to look at performance versus actual bandwidth used for each of the approaches, which we do in Figure 11 (middle, at 1-2M timesteps, and right, at 250k timesteps). For these figures, we did the following: First, for each experience selection approach and target bandwidth, we computed the mean performance and mean actual bandwidth across the five seeds. Then, for each experience selection mode, we plotted these (meanactualbandwidth, meanperformance) (one for each target bandwidth) in a line plot. $^6$  The result gives us a rough estimate of how each approach's performance varies with actual bandwidth used. We see again that stochastic selection shows worse performance than quantile at low bandwidths, and early in training. We also see that gaussian selection very closely approximates quantile selection. Notice that gaussian selection never hits an exact actual bandwidth of 0.1, and so we cannot tell from these data if it would match quantile selection's performance at its peak. However, we can see that at the actual bandwidths that gaussian selection does hit, it shows very similar performance to quantile selection. As stated in the main text, our interpretation of this is that using mean and variance to approximate the exact distribution of absolute td-error is a reasonable approximation, but that we might need to be more clever in selecting  $c$  in equation 3.
5Quantile selection overshoots at 1e-4 (0.0001) target bandwidth and is closer to 1e-3 actual bandwidth usage, which we attribute to a rounding error, as we ran these experiments with a window size of  $1500(1.5\mathrm{e} + 3)$  and a quantile of less than a single element is not well-defined.
<sup>6</sup>Because each data point now has variance in both  $x$ - and  $y$ -directions, it is not possible to draw error bars for these.
Figure 11: Left: Actual bandwidth used (fraction of experiences shared) at different target bandwidths. Middle, right: Performance compared to actual bandwidth used at 1-2M and 250k timesteps.
E Experiment Hyperparameters & Details
We performed all experiments using the open-source library RLlib [20]. Experiments in Figure 1 and 6 were ran using RLlib version 2.0.0; experiments in other figures were run using version 1.13.0. Environments used are from PettingZoo [38], including SISL [11] and MAgent [45]. The SUPER algorithm was implemented by modifying RLlib's standard DQN algorithm to perform the SUPER experience sharing between rollout and training. Table 2 lists all the algorithm hyperparameters and environment settings we used for all the experiments. Experiments in the "Stability across hyperparameters" section had hyperparameters set to those listed in Table 2 except those specified in Figure 8. Any parameters not listed were left at their default values. Hyperparameters were tuned using a grid search; some of the combinations tested are also discussed in the "Stability across hyperparameters" section. For DQN, DDQN and their SUPER variants, we found hyperparameters using a grid search on independent DDQN in each environment, and then used those hyperparameters for all DQN/DDQN and SUPER variants in that environment. For all other algorithms we performed a grid search for each algorithm in each environment. For MADDPG we attempted further optimization using the Python HyperOpt package [5], however yielding no significant improvement over our manual grid search. For SEAC, we performed a grid search in each environment, but found no better hyperparameters than the default. We found a CNN network architecture using manual experimentation in each environment, and then used this architecture for all algorithms except MADDPG where we used a fully connected net for technical reasons. We tested all other algorithms using both the hand-tuned CNN as well as a fully connected network, and found that the latter performed significantly worse, but still reasonable (and in particular significantly better than MADDPG using the same fully connected network, on all domains).
All experiments were repeated with three seeds. All plots show the mean and standard deviation of these seeds at each point in training. For technical reasons, individual experiment runs did not always report data at identical intervals. For instance, one run might report data when it had sampled 51000 environment timesteps, and another run might report at 53000 environment timesteps. In order to still be able to report a meaningful mean and standard deviation across repeated runs, we rounded down the timesteps reported to the nearest  $k$  steps, i.e. taking both the data above to represent each run's performance at 50000 steps. We set  $k$  to the target reporting interval in each domain (8000 timesteps in Pursuit, 6000 timesteps in the other two domains). Where a run reported more than once in a 10000 step interval, we took the mean of its reports to represent that run's performance in the interval. Mean and standard deviation were calculated across this mean performance for each of the five seeds. To increase legibility, we applied smoothing to Figures 1 and 6 using an exponential window with  $\alpha = 0.3$  for Pursuit,  $\alpha = 0.1$  for Battle, and  $\alpha = 0.25$  for Adversarial-Pursuit, and  $\alpha = 0.3$  for Atari.
No smoothing was found to be necessary for MeltingPot results. This removes some noise from the reported performance, but does not change the relative ordering of any two curves.
F Implementation & Reproducibility
All source code is included in the appendix and will be made available on publication under an open-source license. We refer the reader to the included README file, which contains instructions to recreate the experiments discussed in this paper.
Table 2: Hyperparameter Configuration Table - SISL: Pursuit
Environment Parameters
HyperParameters	Value	HyperParameters	Value
max cycles	500	x/y sizes	16/16
shared reward	False	num evaders	30
horizon	500	n catch	2
surrounded	True	num agents(pursuers)	8
tag reward	0.01	urgency reward	-0.1
constrained window	1.0	catch rewards	5
obs range	7
CNN Network
CNN layers	[32,64,64]	Kernel size	[2,2]
Strides	1
SUPER / DQN / DDQN
learning rate	0.00016	final exploration epsilon	0.001
batch size	32	nframework	torch
prioritized replay_alpha	0.6	prioritized replay eps	1e-06
dueling	True	target network update_freq	1000
buffer size	120000	rollout fragment length	4
initial exploration epsilon	0.1
MADDPG
Actor lr	0.00025	Critic lr	0.00025
NN(FC)	[64,64]	tau	0.015
framework	tensorflow	actor feature reg	0.001
SEAC
learning rate	3e-4	adam eps	0.001
batch size	5	use gae	False
framework	torch	gae lambda	0.95
entropy coef	0.01	value loss coef	0.5
max grad norm	0.5	use proper time limits	True
recurrent policy	False	use linear lr decay	False
seac coef	1.0	num processes	4
num steps	5
Table 3: Hyperparameter Configuration Table- MAgent: Battle
Environment Parameters
HyperParameters	Value	HyperParameters	Value
minimap mode	False	step reward	-0.005
Num blue agents	6	Num red agents	6
dead penalty	-0.1	attack penalty	-0.1
attack opponent reward	0.2	max cycles	1000
extra features	False	map size	18
CNN Network
CNN layers	[32,64,64]	Kernel size	[2,2]
Strides	1
SUPER / DQN / DDQN
learning rate	1e-4	batch size	32
framework	torch	prioritized replay_alpha	0.6
prioritized replay eps	1e-06	horizon	1000
dueling	True	target network update_freq	1200
rollout fragment length	5	buffer size	90000
initial exploration epsilon	0.1	final exploration epsilon	0.001
MADDPG
Actor lr	0.00025	Critic lr	0.00025
NN(FC)	[64,64]	tau	0.015
framework	tensorflow	actor feature reg	0.001
SEAC
learning rate	3e-4	adam eps	0.001
batch size	5	use gae	False
framework	torch	gae lambda	0.95
entropy coef	0.01	value loss coef	0.5
max grad norm	0.5	use proper time limits	True
recurrent policy	False	use linear lr decay	False
seac coef	1.0	num processes	4
num steps	5
Table 4: Hyperparameter Configuration Table - MAgent: Adversarial Pursuit
Environment Parameters
HyperParameters	Value	HyperParameters	Value
Number predators	4	Number preys	8
minimap mode	False	tag penalty	-0.2
max cycles	500	extra features	False
map size	18
Policy Network
CNN layers	[32,64,64]	Kernel size	[2,2]
Strides	1
SUPER / DQN / DDQN
learning rate	1e-4	batch size	32
framework	torch	prioritized replay alpha	0.6
prioritized replay eps	1e-06	horizon	500
dueling	True	target network update_freq	1200
buffer size	90000	rollout fragment length	5
initial exploration epsilon	0.1	final exploration epsilon	0.001
MADDPG
Actor lr	0.00025	Critic lr	0.00025
NN(FC)	[64,64]	tau	0.015
framework	tensorflow	actor feature reg	0.001
SEAC
learning rate	3e-4	adam eps	0.001
batch size	5	use gae	False
framework	torch	gae lambda	0.95
entropy coef	0.01	value loss coef	0.5
max grad norm	0.5	use proper time limits	True
recurrent policy	False	use linear lr decay	False
seac coef	1.0	num processes	4
num steps	5
Table 5: Hyperparameter Configuration Table - MeltingPot
Policy Network
CNN layers	[16,128]	Kernel size	[8,8], [sprite_x, sprite_y]
Strides	8,1
SUPER / DDQN
learning rate	0.00016	batch size	32
framework	torch	prioritized replay alpha	0.6
prioritized replay eps	1e-06	horizon	500
dueling	True	target network update_freq	1000
buffer size	30000	rollout fragment length	4
initial exploration epsilon	0.1	final exploration epsilon	0.001
Table 6: Hyperparameter Configuration Table - Atari
Policy Network
Model Configuration	RLlib Default
SUPER / DDQN
learning rate	0.0000625	batch size	32
framework	torch	adam_Epsilon	0.00015
dueling	True	target network update_freq	8000
buffer size	100000	rollout fragment length	4
initial exploration epsilon	0.1	final exploration epsilon	0.01