SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Central to our method is the estimation of uncertainty in Q-values. We found that what kind of uncertainty to model and how we model it are crucial to the performance of EDE The first important consideration is what kind of uncertainty to model. Specifically, we argue that only epistemic uncertainty is useful for exploration since they represent the reducible uncertainty. This is a longstanding view shared by many different sub-field of machine learning [74, 98]. The aleatoric uncertainty on the other hand is intrinsic to the environment (e.g., label noise) and cannot be reduced no matter how many samples we gather. In standard regression, even with ensembles, the estimated uncertainty (i.e., variance) contains both epistemic and aleatoric uncertainty. In Atari, the main source of aleatoric uncertainty comes from the sticky action which has a relatively simple structure. However, in CMDPs such as Progen, the aleatoric uncertainty coming from having different environments is much more complicated and possibly multimodal, making it important to model them explicitly. Empirically, using aleatoric uncertainty to explore is detrimental to the test performance which may partially explain DQN+UCB's extremely poor performance (Figure 5), even compared to DQN+ε-greedy (i.e., uncertainty estimation based on ensemble of non-distributional DQNs contains both types of uncertainties). To further test this hypothesis, we run QR-DQN+UCB with both aleatoric and epistemic uncertainty and a version with only epistemic uncertainty (Figure 16c). The results show that using only epistemic uncertainty performs significantly better than using both uncertainties in terms of both training and test performance.
Some additional important detail about uncertainty estimation with an ensemble are the number of ensembles and how the members of the ensemble are updated. These details have implications for both the computational complexity of inference and the quality of estimated uncertainty. For the main experiments, we use 5 ensemble members which, without specific optimization, results in about 2.5 times more time per algorithm step. The fol lowing are the time complexity of our algorithm on a Tesla V100 (note that this only occurs at the training time since at test time all heads share the same feature extractor):
EDE 5 ensemble members): 0.3411 seconds per algorithm step
EDE 3 ensemble members): 0.2331 seconds per algorithm step
- QR-DQN: 0.1356 seconds per algorithm step
In Figure 17b, we found that using 3 ensemble heads achieves comparable performance. On the other hand, using deep ensemble seems to be crucial for a good estimation of epistemic uncertainty as UA-DQN [20], which uses MAP sampling with 3 ensemble members, performs much worse than QR-DQN with 3 ensemble members trained with different minibatch and initialization (which are presumably more independent from each other). We believe further understanding this phenomenon would be an exciting future research direction.
I Architecture and Hyperparameters
I.1 Progen
Architecture. We use a standard ResNet [42] feature extractor  $(f)$  from IMPALA [27], which was adopted by Ehrenberg et al. [26] for value-based methods. The feature maps are flattened  $(d = 2048)$  and processed by  $M$  copies of 2-layer MLPs with the same architecture to output  $M \times |\mathcal{A}| \times N$  values (for other models, this shape is changed accordingly for the appropriate output dimension).  $|\mathcal{A}| = 15$  for all Progen games. Each MLP consists of two liner layers (both of dimension 512) separated by a ReLU activation; the first layer is  $g^{(1)}: \mathbb{R}^d \to \mathbb{R}^{512}$  and the second layer is  $g^{(2)}: \mathbb{R}^{512} \to \mathbb{R}^{|\mathcal{A}| \times N}$ . The overall architecture is
$$
g \circ f = g ^ {(2)} \circ \operatorname {R e L U} \circ g ^ {(1)} \circ \text {F l a t t e n} \circ f.
$$
The hyperparameters for the main method is shown in Table 4, and are directly adapted from Ehrenberg et al. [26] other than those of new algorithmic components.
For hyperparameters specific to EDE, we search over the following values on the game bigfish:
$\varphi : \{10, 20, 30, 50\}$
-  $\lambda : \{0.5, 0.6, 0.7\}$
$\alpha : \{6, 7, 8, 9\}$
and picked the best combination.
Name	Value
number of training envs	200
procgen distribution mode	easy
minibatch size	512
number of parallel workers (K)	64
replay buffer size	106
discount factor (γ)	0.99
n-step return	3
frame stack	1
dueling network	no
target network update frequency	32000 (algo step)
initial random experience collection steps	2000 × K (env step)
total number of steps	25 × 106 (env step)
update per algo step	1
env step per algo step	K
Adam learning rate	2.5 × 10-4
Adam epsilon	1.5 × 10-4
prioritized experience replay	no
number of quantiles (N)	200
number of ensemble heads (M)	5
Huber loss κ	1
gradient clip norm	10
φ (UCB)	30
λ (TEE)	0.6
α (TEE)	7
Table 4: Hyperparameters for the main Procgen experiments. One algorithm step (algo step) can have multiple environment steps (env step) due to the distributed experience collection.
Points of comparisons. We try to keep the hyperparameters the same as much as possible and only make necessary changes to keep the comparison fair:
- For base DQN and QR-DQN, we use  $\varepsilon$ -greedy exploration with decaying  $\varepsilon$ . Specifically, at every algorithmic step, the epsilon is computed with the following equation:
$$
\varepsilon (t) = \min  \left(0. 1 + 0. 9 \exp \left(- \frac {t - 2 0 0 0}{8 0 0 0}\right), 1\right)
$$
- For Bootstrapped DQN [79], we use bootstrapping to train the model and uniformly sample a value head for each actor at the beginning of each episode that is used for the entire episode. We use 5 ensemble members just like the other methods.
- For UCB, we use the estimated epistemic uncertainty when combined with QR-DQN and use the standard variance when combined with DQN [19] and use  $\varphi = 30$ .
- For TEE without UCB, we use  $\varepsilon$ -greedy exploration without decaying  $\varepsilon$  for both QR-DQN and DQN, and EDE ith the same  $\bar{\lambda}$  and  $\alpha$ .
- For  $\epsilon z$ -greedy, we use the same  $\varepsilon$  decay schedule in combination with  $n = 10000$  and  $\mu = 2.0$  which are the best performing hyperparameter values from Dabney et al. [24]. Each parallel worker has its own  $\epsilon z$ -greedy exploration process.
- For NoisyNet, we modify the base fully-connected layer with the noisy implementation [32] with  $\sigma = 0.5$ .
I.2 Crafter
For Crafter, we use the default architecture and hyperparameters that are used for the Rainbow experiments in Hafner [37] with the following changes:
- Change the distributional RL component from C51 to QR-DQN with appropriate architecture change and use EDE for exploration
- Change the minibatch size from 32 to 64 to speed up training
- Remove prioritized experience replay
Once again, we emphasize that everything else stays the same, so the performance gain can be largely attributed to the proposed method. For EDE since the default configuration from Hafner [37] is single-thread, we use Thompson sampling instead of UCB and do not use TEE.
For  $\varphi$ , we searched over  $\{0.5, 1.0, 2.0\}$  and picked the best value.
Name	Value
minibatch size	64
number of parallel workers (K)	1
replay buffer size	10^6
discount factor (γ)	0.99
n-step return	3
frame stack	4
dueling network	no
target network update frequency	8000 (algo step)
initial random experience collection steps	20000 (env steps)
total number of steps	10^6 (env steps)
algo step per update	4
env step per algo step	1
Adam learning rate	6.25 × 10^-5
Adam epsilon	1.5 × 10^-4
prioritized experience replay	no
number of quantiles (N)	200
number of ensemble heads (M)	5
Huber loss κ	1
gradient clip norm	10
φ (Thompson sampling)	0.5
Table 5: Hyperparameters for the final Crafter experiments.
J Sensitivity Study
In this section, we study the sensitivity of the test performance (i.e., aggregated min-max normalized scores) to various hyperparameters on a subset of games. First, without TEE, we study the sensitivity to different  $\varphi$  (UCB exploration coefficients), different  $M$  (number of ensemble members), and whether the feature extractor is trained with gradients from each ensemble member.
For  $\varphi$  and  $M$ , we run the algorithm on: bossfight, ninja, plunder, starpilot, caveflyer, jumper, bigfish, leaper for 1 seed and report the aggregated min-max normalized scores in Figure 17a and Figure 17b. We observe that the algorithm is slightly sensitive to the choice of  $\varphi$ , but not sensitive at all to  $M$ . This is encouraging since a large amount of computational overhead comes from the ensemble. Note that while smaller  $\varphi$  performs the best here, we use a larger value ( $\varphi = 30$ ) in combination with TEE.
In Figure 17c, we show the effect of only training the feature extractor using the gradient from one member of the ensemble at every iteration. The results are computed on: ninja, plunder, jumper, caveflyer, bigfish, leaper, climber for 1 seed. We observe that always training the feature extractor leads to lower performance, corroborating our intuition that the feature extractor should be trained at the same speed as the individual ensemble members.
In Figure 18, we study the performance under different hyperparameter values of TEE. We use fixed  $M = 5$  and  $\varphi = 30$  and vary the values of either  $\alpha$  or  $\lambda$  while holding the other one fixed. We observe no significant difference across these, suggesting that the algorithm is robust to the values of  $\alpha$  and  $\lambda$ .
K Hardware
The experiments are conducted on 2080 and V100 and take approximately 250 GPU days.
(a)  $\varphi$  ablation.
(b)  $M$  ablation.
(c) Feature extractor ablation.
Figure 17: Aggregated min-max normalized test scores for  $\varphi$  (for fixed  $M = 3$ , and training feature extractor for all value heads),  $M$  (for fixed  $\varphi = 50$  and training feature extractor for all value heads), and whether feature extractor is trained with all value head (for fixed  $\varphi = 50$  and  $M = 3$ ).
Figure 18: Aggregated min-max normalized test scores for  $\lambda$  (for fixed  $\alpha = 7$ ) and  $\alpha$  (for fixed  $\lambda = 0.6$ ) on 4 games: bossfight, climber, plunder, starpilot.