SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, the bottom row of Fig. 11a details the success rate as the ratio of instances in which the agent successfully reached the oracle's score (meaning that the entire traversable region has been explored). For the Maze environment, the agent achieves a success-rate of  $70\%$ , whereas for the Jumper and Miner the agent fails to meet the oracle's score. This indicates that the oracle, as per our implementation for Jumper and Miner, captures states that are in-effect unreachable, and thus the agent is unable to match the oracle's score.
B.3 The Importance of a Memory Unit for maxEnt
We evaluate the importance of memory for maxEnt: In Fig. 11a we train a maxEnt agent with and without memory (GRU). The results indicate that for Maze, the GRU is vital in order to maximize performance for all various sizes of training set. In Jumper, we see that the GRU provides an advantage when training on 200 levels, however the benefit diminishes when additional training levels are available. For Miner, there appears to be no significant advantage for incorporating memory with 200 training levels. However, the benefit of a GRU becomes noticeable once more training levels are available (beyond 500).
When looking at the extrinsic reward (Fig. 11b), we see an interesting effect after introducing a GRU. Maze and Jumper suffer a degradation in performance with 200 levels (indicating overfit), while Miner appears to be unaffected.
In summary, we empirically show that memory is beneficial for the maxEnt policy on Maze, Jumper and Miner. Interestingly, we demonstrate that the introduction of memory for training to maximize extrinsic reward causes the agent to overfit in Maze and Jumper with 200 training levels.
This section describes the constants and hyperparameters used as part of the evaluation of our algorithm.
C.1 Normalization Constants
In Figures 2 and 12 we compare the performance of the various algorithms. The results are normalized in accordance with [Cobbe et al., 2020], which defines the normalized return as
$$
R _ {n o r m} = (R - R _ {m i n}) / (R _ {m a x} - R _ {m i n}).
$$
The  $R_{min}$  and  $R_{max}$  of each environment are detailed in Table 3. Note that the test performance of PPO on Heist (see Fig. 2) is lower than  $R_{min}$  (the trivial performance), indicating severe overfitting.
As described in Section 5.1, the hyperparameters of our algorithm are the number of agents  $m$  that form the ensemble, of which  $k$  agents are required to be in agreement for the ensemble to achieve a consensus on its action, and  $\alpha$  as the parameter of  $n_{\pi_{\mathcal{H}}}\sim Geom(\alpha)$  that represents the number of maxEnt steps taken when the ensemble fails to reach a consensus. An additional hyperparameter is the neighborhood size  $k_{\mathrm{NN}}$  of the  $k$ -NN estimator (Section 4.1), used to determine the reward of the maxEnt policy.
We conducted a hyperparameter search over the ensemble size  $m \in \{4, 6, 8, 10\}$  for different values of ensemble agreement  $k \in \{2, 4, 6, 8\}$ , and values of  $\alpha \in \{0.2, 0.5, 0.8\}$ . We found that a value of  $\alpha = 0.5$ , and an ensemble size of  $m = 10$  produce the best results for all games, whereas the value of  $k$  varies from game to game, as detailed in table 4. For example, table 5 shows the results for varying values of  $m$  and  $k$  for the Maze environment. Throughout our experiments, we train our networks using the Adam optimizer [Kingma and Ba, 2014]. For the PPO hyperparameters we use the hyperparameters found in [Cobbe et al., 2020] as detailed in Table 6.
Hard
Easy
Environment	Rmin	Rmax	Rmin	Rmax
CoinRun	5	10	5	10
StarPilot	1.5	35	2.5	64
CaveFlyer	2	13.4	3.5	12
Dodgeball	1.5	19	1.5	19
FruitBot	-.5	27.2	-1.5	32.4
Chaser	.5	14.2	.5	13
Miner	1.5	20	1.5	13
Jumper	1	10	3	10
Leaper	1.5	10	3	10
Maze	4	10	5	10
BigFish	0	40	1	40
Heist	2	10	3.5	10
Climber	1	12.6	2	12.6
Plunder	3	30	4.5	30
Ninja	2	10	3.5	10
BossFight	.5	13	.5	13
Table 3: Normalization Constants.
Game	k
Maze	6
Jumper	4
Miner	2
Heist	8
BigFish	8
StarPilot	1
FruitBot	1
BossFight	1
Plunder	2
CaveFlyer	2
CoinRun	1
Chaser	2
Climber	2
Dodgeball	2
Leaper	1
Ninja	2
Table 4: Consensus size  $k$  as hyperparameter for each game.
Ensemble size	4	6	8	10
ExpGen	8.02 ± 0.06	8.15 ± 0.19	8.00 ± 0.12	8.22 ± 0.11
Table 5: Ablation study of ensemble size and its effect on the test score. Each network in the ensemble is trained on 200 instances of Maze. The results show improved performance for large ensemble size. The mean and standard deviation are computed using 10 runs with different seeds.
Table 7 shows an evaluation of the maxEnt gap and ExpGen using different neighbor size  $k_{\mathrm{NN}}$  on the Maze environment. We found that the best performance is obtained for  $k_{\mathrm{NN}} \in \{1,2,3,4,5\}$ . Thus, we choose  $k_{\mathrm{NN}} = 2$ , the second nearest neighbor, for all games.
D Results for all ProcGen Games
Figure 12 details the normalized test performance for all ProcGen games. Normalization is performed according to [Cobbe et al., 2020] as described in Appendix C.1. The figure demonstrates
Parameter	Value
γ	.999
λ	.95
# timesteps per rollout	512
Epochs per rollout	3
# minibatches per epoch	8
Entropy bonus (kH)	.01
PPO clip range	.2
Reward Normalization?	Yes
Learning rate	5e-4
# workers	1
# environments per worker	32
Total timesteps	25M
GRU?	Only for maxEnt
Frame Stack?	No
Table 6: PPO Hyperparameters.
Neighbor Size kNN	maxEnt (Train)	Maze maxEnt (Test)	Environment maxEnt Gap [%]	ExpGen (Test)
1	17.5 ± 0.5	15.3 ± 1.4	12.7%	7.9 ± 0.2
2	33.9 ± 0.2	31.3 ± 1.5	7.7%	8.3 ± 0.2
3	50.2 ± 2.0	42.5 ± 2.8	15.3%	8.2 ± 0.2
4	62.9 ± 2.6	52.3 ± 3.0	16.8%	8.2 ± 0.1
5	70.6 ± 3.3	57.8 ± 4.6	15.5%	8.2 ± 0.1
6	80.0 ± 3.2	65.3 ± 1.4	18.4%	8.1 ± 0.2
7	87.2 ± 1.4	71.1 ± 1.1	18.5%	8.0 ± 0.1
8	93.8 ± 1.0	75.3 ± 2.3	19.7%	7.8 ± 0.2
9	96.6 ± 4.8	78.0 ± 2.0	19.3%	8.1 ± 0.2
10	102.2 ± 1.8	89.1 ± 0.6	12.8%	7.9 ± 0.1
that ExpGen establishes state-of-the-art results on several challenging games and achieves on-par performance with the leading approach on the remaining games.
E Results after convergence
Tables 8 and 9 detail the train and test performance of ExpGen, LEEP and PPO, when trained for  $50M$  environment steps. Table 9 shows that ExpGen surpasses LEEP and PPO in most games.
Table 7: Hyperparameter search for neighborhood size  $k_{\mathrm{NN}}$  for the Maze environment. The table presents the maxEnt gap and the performance of ExpGen for varying values of  $k_{\mathrm{NN}}$ . The mean and variance are computed for 3 seeds.
Game	PPO	LEEP	ExpGen
Maze	9.45 ± 0.21	9.84 ± 0.05	9.71 ± 0.11
Heist	7.97 ± 0.56	6.86 ± 0.68	9.62 ± 0.08
Jumper	8.62 ± 0.08	6.1 ± 0.4	8.00 ± 0.13
Miner	12.86 ± 0.06	1.9 ± 0.2	12.53 ± 0.12
BigFish	14.24 ± 3.36	8.82 ± 0.36	5.95 ± 0.32
Climber	8.76 ± 0.41	4.6 ± 0.4	8.89 ± 0.29
Dodgeball	8.88 ± 0.38	6.22 ± 0.55	8.73 ± 0.40
Plunder	9.64 ± 1.85	5.1 ± 0.1	8.06 ± 0.98
Ninja	9.10 ± 0.32	5.2 ± 0.3	8.90 ± 0.30
CaveFlyer	8.98 ± 0.59	5.4 ± 0.1	8.75 ± 0.59
Table 8: Train score of ProcGen environments trained on 200 instances for 50M environment steps. We compare our algorithm to the baselines LEEP and PPO. The mean and standard deviation are computed over 8 runs with different seeds.
Figure 12: Normalized test Performance for PPO, IDAAC, and ExpGen+IDAAC, on all ProcGen games. ExpGen achieves state-of-the-art performance on test levels of Maze, Heist, and Jumper and on-par performance in the remaining games.
Game	PPO	LEEP	ExpGen
Maze	5.78 ± 0.39	6.78 ± 0.21	8.33 ± 0.14
Heist	2.54 ± 0.45	4.42 ± 0.57	6.91 ± 0.24
Jumper	5.78 ± 0.28	6.4 ± 0.4	6.64 ± 0.15
Miner	8.76 ± 0.33	0.8 ± 0.1	9.48 ± 0.39
BigFish	3.82 ± 1.98	5.5 ± 0.41	5.99 ± 0.64
Climber	6.14 ± 0.50	2.6 ± 0.4	6.29 ± 0.54
Dodgeball	3.71 ± 0.55	4.58 ± 0.47	3.84 ± 0.56
Plunder	7.78 ± 1.74	4.2 ± 0.2	6.91 ± 1.00
Ninja	6.94 ± 0.30	4.9 ± 0.8	6.75 ± 6.75
CaveFlyer	6.19 ± 0.66	2.6 ± 0.2	6.36 ± 0.49
Table 9: Test score of ProcGen environments trained on 200 instances for  $50M$  environment steps. We compare our algorithm to the baselines LEEP and PPO. The mean and standard deviation are computed over 8 runs with different seeds.
In the following sections, we provide ablation studies of an ExpGen variant that combines random actions and an evaluation of  $L_{0}$  state similarity measure.
F.1 Ensemble Combined with Random Actions
We compare the proposed approach to a variant of ExpGen denoted by Ensemble+random where we train an ensemble and at test time select a random action if the ensemble networks fail to reach a consensus. The results are shown in Table 10, indicating that selecting the maximum entropy policy upon ensemble disagreements yields superior results.
Algorithm	Train	Test
ExpGen	9.6 ± 0.2	8.2 ± 0.1
Ensemble + random	9.3 ± 0.1	6.2 ± 0.2
LEEP	9.4 ± 0.3	6.6 ± 0.2
PPO + GRU	9.5 ± 0.2	5.4 ± 0.3
PPO	9.1 ± 0.2	5.6 ± 0.1
Table 10: Ablation study of ExpGen on Maze. The table shows testing scores of networks trained on 200 maze instances. We present a comparison between the proposed approach and LEEP, PPO+GRU and PPO, as well as an alternative ensemble policy with random actions upon ensemble disagreement. The mean and standard deviation are computed using 10 runs with different seeds.
F.2 Evaluation of Various Similarity Measures for maxEnt
Tables 11 and 12 present the results of our evaluation of ExpGen equipped with a maxEnt exploration policy that uses either the  $L_{0}$  or  $L_{2}$  norms. The experiment targets the Maze and Heist environments and uses the same train and test procedures as in the main paper (25M training steps, score mean and standard deviation are measured over 10 seeds).
Game	ExpGen L0(Train)	ExpGen L2(Train)	PPO(Train)
Heist	9.4 ± 0.3	9.4 ± 0.1	6.1 ± 0.8
Maze	9.6 ± 0.2	9.6 ± 0.1	9.1 ± 0.2
Table 11: Train scores of ExpGen using maxEnt policy with either  $L_0$  or  $L_2$  compared with PPO. The mean and standard deviation are measured over 10 seeds.
Game	ExpGen L0 (Test)	ExpGen L2 (Test)	PPO (Test)
Heist	7.4 ± 0.1	7.4 ± 0.2	2.4 ± 0.5
Maze	8.2 ± 0.1	8.3 ± 0.2	5.6 ± 0.1
Table 12: Test scores of ExpGen using maxEnt policy with either  $L_0$  or  $L_2$  compared with PPO. The mean and standard deviation are measured over 10 seeds.
The results demonstrate that both  $L_{2}$  and  $L_{0}$  allow ExpGen to surpass the PPO baseline for Maze and Heist environments, in which they perform similarly well at test time. This indicates that both are valid measures of state similarity for the maxEnt policy.
G Sample Complexity
One may wonder whether the leading approaches would benefit from training on additional environment steps. A trained agent can still fail at test time either due to poor generalization performance (overfitting on a small number of training domains) or due to insufficient training steps of the policy (underfitting). In this work, we are interested in the former and design our experiments such that no method underfits. Figure 13 shows IDAAC training for  $100M$  steps on Maze and Jumper, illustrating that the best test performance is obtained at around  $25M$  steps, and training for longer does not contribute further (and can even degrade performance). Therefore, although ExpGen requires more environment steps (on account of training its ensemble of constituent reward policies), training for longer does not place our baseline (IDAAC) at any sort of a disadvantage.
Figure 13: The mean and median of the accumulated reward for IDAAC trained for  $100M$  steps, averaged over 10 runs with different seeds. The curves show that test-reward stagnates and even decreases beyond  $25M$  steps.