SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our main insight is that, given the generalization property of the entropy maximization policy established above, an agent can apply this behavior in a test MDP and expect effective exploration at test time. In the following, we pair this insight with the epistemic POMDP idea, and propose to play the exploration policy when the agent faces epistemic uncertainty, hopefully driving the agent to a different state where the reward-seeking policy is more certain. This can be seen as an adaptation of the seminal explicit explore or exploit idea [Kearns and Singh, 2002], to the setting of ZSG-RL.
5.1 Algorithm
Our framework comprises two parts: an entropy maximizing network and an ensemble of networks that maximize an extrinsic reward to evaluate epistemic uncertainty. The first step entails training a network equipped with a memory unit to obtain a maxEnt policy  $\pi_{\mathcal{H}}$  that maximizes entropy, as described in section 4.1. Next, we train an ensemble of memory-less policy networks  $\{\pi_r^j\}_{j=1}^m$  to maximize extrinsic reward. Following Ghosh et al. [2021], we shall use the ensemble to assess epistemic uncertainty. Different from Ghosh et al. [2021], however, we do not change the RL loss function, and use an off-the-shelf RL algorithm (such as PPO [Schulman et al., 2017] or IDAAC [Stooke et al., 2021]).
At test time, we couple these two components into a combined agent  $\pi$  (detailed as pseudo-code in Algorithm 1). We consider domains with a finite action space, and say that the policy  $\pi_r^i$  is certain at state  $s$  if its action  $a_{i}\sim \pi_{r}^{i}(a|s)$  is in consensus with the ensemble:  $a_{i} = a_{j}$  for the majority of  $k$  out of  $m$ , where  $k$  is a hyperparameter of our algorithm. When the networks  $\{\pi_r^j\}_{j = 1}^m$  are not in consensus, the agent  $\pi$  takes a sequence of  $n_{\pi_{\mathcal{H}}}$  actions from the entropy maximization policy  $\pi_{\mathcal{H}}$ , which encourages exploratory behavior.
Algorithm 1 Explore to Generalize (ExpGen)
1: Input: ensemble size  $m$ ,  
2: initial state  $s_0 = \text{ENVIRONMENT reset}$ .  
3:  $n_{\pi_{\mathcal{H}}} = 0$   
4: Train maxEnt policy  $\pi_{\mathcal{H}}$  using intrinsic reward  $r_I$  (Eq: 7).  
5: Train  $m$  policies  $\pi_r^1, \pi_r^2 \ldots \pi_r^m$  using extrinsic reward  $r_{ext}$ .  
6: for  $t = 1$  to  $H$  do  
7:  $a_i \sim \pi_r^i(\cdot|s_t)$   
8:  $n_{\pi_{\mathcal{H}}} \gets n_{\pi_{\mathcal{H}}} - 1$   
9: if  $a_i \in \text{Consensus}(a_j | j \in \{1 \ldots m\})$  and  $n_{\pi_{\mathcal{H}}} < 0$  then  
10:  $a_t = a_i$   
11: else  
12:  $a_{\mathcal{H}} \sim \pi_{\mathcal{H}}(\cdot|h_t)$   
13:  $a_t = a_{\mathcal{H}}$   
14:  $n_{\pi_{\mathcal{H}}} \sim \text{Geom}(\alpha)$   
15: end if  
16:  $s_{t+1} \gets \text{ENVIRONMENT STEP}(a_t)$   
17: end for
Agent meta-stability Switching between two policies may result in a case where the agent repeatedly toggles between two states—if, say, the maxEnt policy takes the agent from state  $s_1$  to a state  $s_2$ , where the ensemble agrees on an action that again moves to state  $s_1$ . To avoid such "meta-stable" behavior, we randomly choose the number of maxEnt steps  $n_{\pi_{\mathcal{H}}}$  from a Geometric distribution,  $n_{\pi_{\mathcal{H}}} \sim \text{Geom}(\alpha)$ .
We evaluate our algorithm on the ProcGen benchmark, which employs a discrete 15-dimensional action space and generates RGB observations of size  $64 \times 64 \times 3$ . Our experimental setup follows ProcGen's 'easy' configuration, wherein agents are trained on 200 levels for  $25M$  steps and subsequently tested on random levels [Cobbe et al., 2020]. All agents are implemented using the IMPALA convolutional architecture [Espeholt et al., 2018], and trained using PPO [Schulman et al., 2017] or
Figure 5: Test performance of PPO trained using the reward  $r_{total}$  that combines intrinsic and extrinsic rewards, weighted by  $\beta$  (Eq. 9). Each figure details the results for different values of discount factor  $\gamma$ . All networks are randomly initialized and trained on 200 maze levels, and their mean is computed over 4 runs with different seeds. The figures show an improvement over the PPO baseline for  $\gamma = 0.5$ . In all cases, ExpGen outperforms the combined reward agent.
IDAAC [Raileanu and Fergus, 2021]. For the maximum entropy agent  $\pi_{\mathcal{H}}$  we incorporate a single GRU [Cho et al., 2014] at the final embedding of the IMPALA convolutional architecture. For all games, we use the same parameter  $\alpha = 0.5$  of the Geometric distribution and form an ensemble of 10 networks. For further information regarding our experimental setup and specific hyperparameters, please refer to Appendix C.
6.1 Generalization Performance
We compare our algorithm to six leading algorithms: vanilla PPO [Schulman et al., 2017], PLR [Jiang et al., 2021] that utilizes automatic curriculum-based learning, UCB-DrAC [Raileanu et al., 2021], which incorporates data augmentation to learn policies invariant to different input transformations, PPG [Cobbe et al., 2021], which decouples the optimization of policy and value function during learning, and IDAAC [Raileanu and Fergus, 2021], the previous state-of-the-art algorithm on ProcGen that decouples policy learning from value function learning and employs adversarial loss to enforce invariance to spurious features. Lastly, we evaluate our algorithm against LEEP [Ghosh et al., 2021], the only algorithm that, to our knowledge, managed to improve upon the performance of vanilla PPO on Maze and Heist. The evaluation matches the train and test setting detailed by the contending algorithms and their performance is provided as reported by their authors. For evaluating LEEP and IDAAC, we use the original implementation provided by the authors.
Tables 2 and 1 show the train and test scores, respectively, for all ProcGen games. The tables show that ExpGen combined with PPO achieves a notable gain over the baselines on Maze, Heist and Jumper, while on other games, invariance-based approaches perform better (for example, IDAAC leads on BigFish, Plunder and Climber, whereas PPG leads on CaveFlyer, and UCB-DrAC leads on Dodgeball). These results correspond to our observation that for some domains, invariance cannot be used to completely resolve epistemic uncertainty. We emphasize that ExpGen substantially outperforms LEEP on all games, showing that our improved exploration at test time is significant. In Appendix E we compare ExpGen with LEEP trained for  $50M$  environment steps, showing a similar trend. When combining ExpGen with the leading invariance-based approach, IDAAC, we establish that ExpGen is in-fact complementary to the advantage of such algorithms, setting a new state-of-the-art performance in ProcGen. A notable exception is Dodgeball, where all current methods still fail.
Figures 6 and 7 show aggregate statistics of ExpGen, PPO, PLR, UCB-DrAC, PPG and IDAAC for all games  $^{6}$ , affirming the dominance of ExpGen+IDAAC as the state-of-the-art. The results are obtained using 10 runs per game, with scores normalized as in Appendix C.1. The shaded regions indicate  $95\%$  Confidence Intervals (CIs) and are estimated using the percentile stratified bootstrap with 2,000 (Fig. 6) and 50,000 (Fig. 7) bootstrap re-samples. Fig. 6 (Left) compares algorithm score-distribution, illustrating the advantage of the proposed approach across all games. Fig. 6 (Right)
For LEEP and IDAAC, we followed the prescribed hyperparameter values of the papers' authors (we directly corresponded with them). For some domains, we could not reproduce the exact results, and in those cases, we used their reported scores, giving them an advantage.
See riable [Agarwal et al., 2021] for additional details on the various performance measures and protocols.
Figure 6: Comparison across all ProcGen games, with  $95\%$  bootstrap CIs highlighted in color. Left. Score distributions of ExpGen, PPO, PLR, UCB-DrAC, PPG and IDAAC. Right. Shows in each row, the probability of algorithm  $X$  outperforming algorithm  $Y$ . The comparison illustrates the superiority of ExpGen over the leading contender IDAAC with probability 0.6, as well as over other methods with even higher probability.
shows the probability of improvement of algorithm  $X$  against algorithm  $Y$ . The first row (ExpGen vs. IDAAC) demonstrates that the proposed approach surpasses IDAAC with probability 0.6 and subsequent rows emphasize the superiority of ExpGen over contending methods at an even higher probability. This is because ExpGen improves upon IDAAC in several key challenging tasks and is on-par in the rest. Fig. 7 provides aggregate metrics of mean, median and IQM scores and optimality gap (as  $1 - \text{mean}$ ) for all ProcGen games. The figure shows that ExpGen outperforms the contending methods in all measures.
Ablation Study One may wonder if the ensemble in ExpGen is necessary, or whether the observation that the maxEnt policy generalizes well can be exploited using a single policy. We investigate the effect of combining the intrinsic and extrinsic rewards,  $r_I$  and  $r_{ext}$ , respectively, into a single reward as a weighted sum:
$$
r _ {\text {t o t a l}} = \beta r _ {I} + (1 - \beta) r _ {\text {e x t}}, \tag {9}
$$
and train for  $\beta = \{0.1, 0.3, 0.5, 0.7, 0.9\}$  on Maze. Figure 5 shows the train and test scores over  $50M$  steps for different values of discount factor  $\gamma$ . We obtain the best test score for  $\gamma = 0.5$  and  $\beta = 0.1$ , illustrating an improvement compared with the PPO baseline. When comparing with ExpGen, the combined reward (Eq. 9) exhibits inferior performance with slightly higher variance. In Appendix C.2 and F, we also provide an ablation study of ensemble size and draw comparisons to variants of our algorithm.
Game	PPO	PLR	UCB-DrAC	PPG	IDAAC	LEEP	ExpGen (PPO)	ExpGen (IDAAC)
BigFish	2.9 ± 1.1	10.9 ± 2.8	9.2 ± 2.0	11.2 ± 1.4	18.5 ± 1.2	4.9 ± 0.9	6.0 ± 0.5	18.5 ± 1.9
StarPilot	24.9 ± 1.0	27.9 ± 4.4	30.0 ± 1.3	47.2 ± 1.6	37.0 ± 2.3	3.2 ± 2.2	31.0 ± 0.9	39.8 ± 2.9
FruitBot	26.2 ± 1.2	28.0 ± 1.4	27.6 ± 0.4	27.8 ± 0.6	27.9 ± 0.5	16.4 ± 1.6	26.2 ± 0.6	28.4 ± 0.4
BossFight	7.4 ± 0.4	8.9 ± 0.4	7.8 ± 0.6	10.3 ± 0.2	9.8 ± 0.6	0.5 ± 0.3	7.7 ± 0.2	9.8 ± 0.5
Ninja	6.1 ± 0.2	7.2 ± 0.4	6.6 ± 0.4	6.6 ± 0.1	6.8 ± 0.4	4.4 ± 0.5	6.6 ± 0.2	6.6 ± 0.3
Plunder	7.8 ± 1.6	8.7 ± 2.2	8.3 ± 1.1	14.3 ± 2.0	23.3 ± 1.4	4.4 ± 0.3	5.5 ± 1.3	23.6 ± 1.4
CaveFlyer	5.5 ± 0.5	6.3 ± 0.5	5.0 ± 0.8	7.0 ± 0.4	5.0 ± 0.6	4.9 ± 0.2	5.7 ± 0.3	5.3 ± 0.7
CoinRun	8.6 ± 0.2	8.8 ± 0.5	8.6 ± 0.2	8.9 ± 0.1	9.4 ± 0.1	7.3 ± 0.4	8.8 ± 0.1	9.3 ± 0.3
Jumper	5.8 ± 0.3	5.8 ± 0.5	6.2 ± 0.3	5.9 ± 0.1	6.3 ± 0.2	5.4 ± 1.2	6.7 ± 0.3	6.8 ± 0.5
Chaser	3.1 ± 0.9	6.9 ± 1.2	6.3 ± 0.6	9.8 ± 0.5	6.8 ± 1.0	3.0 ± 0.1	3.6 ± 1.6	7.1 ± 1.4
Climber	5.4 ± 0.5	6.3 ± 0.8	6.3 ± 0.6	2.8 ± 0.4	8.3 ± 0.4	2.6 ± 0.9	5.9 ± 0.5	9.5 ± 0.3
Dodgeball	2.2 ± 0.4	1.8 ± 0.5	4.2 ± 0.9	2.3 ± 0.3	3.2 ± 0.3	1.9 ± 0.2	2.9 ± 0.3	2.8 ± 0.2
Heist	2.4 ± 0.5	2.9 ± 0.5	3.5 ± 0.4	2.8 ± 0.4	3.5 ± 0.2	4.5 ± 0.3	7.4 ± 0.2	7.2 ± 0.5
Leaper	4.9 ± 2.2	6.8 ± 1.2	4.8 ± 0.9	8.5 ± 1.0	7.7 ± 1.0	4.4 ± 0.2	4.0 ± 1.7	7.6 ± 1.2
Maze	5.6 ± 0.1	5.5 ± 0.8	6.3 ± 0.1	5.1 ± 0.3	5.6 ± 0.3	6.6 ± 0.2	8.3 ± 0.2	7.8 ± 0.2
Miner	7.8 ± 0.3	9.6 ± 0.6	9.2 ± 0.6	7.4 ± 0.2	9.5 ± 0.4	1.1 ± 0.1	8.0 ± 0.7	9.8 ± 0.3
Table 1: Test score of ProcGen games trained on 200 levels for  ${25M}$  environment steps. We compare our algorithm to PPO, PLR, UCB-DrAC, PPG, IDAAC and LEEP. The mean and standard deviation are computed over 10 runs with different seeds.
Game	PPO	PLR	UCB-DrAC	PPG	IDAAC	LEEP	ExpGen (PPO)	ExpGen (IDAAC)
BigFish	8.9 ± 2.0	7.8 ± 1.0	12.8 ± 1.8	19.9 ± 1.7	21.8 ± 1.8	8.9 ± 0.9	7.0 ± 0.4	21.5 ± 2.3
StarPilot	29.0 ± 1.1	2.6 ± 0.3	33.1 ± 1.3	49.6 ± 2.1	38.6 ± 2.2	5.3 ± 0.3	34.3 ± 1.6	40.0 ± 2.7
FruitBot	28.8 ± 0.6	15.9 ± 1.3	29.3 ± 0.5	31.1 ± 0.5	29.1 ± 0.7	17.4 ± 0.7	28.9 ± 0.6	29.5 ± 0.5
BossFight	8.0 ± 0.4	8.7 ± 0.7	8.1 ± 0.4	11.1 ± 0.1	10.4 ± 0.4	0.3 ± 0.1	7.9 ± 0.6	9.9 ± 0.7
Ninja	7.3 ± 0.2	5.4 ± 0.5	8.0 ± 0.4	8.9 ± 0.2	8.9 ± 0.3	4.6 ± 0.2	8.5 ± 0.3	7.9 ± 0.6
Plunder	9.4 ± 1.7	4.1 ± 1.3	10.2 ± 1.8	16.4 ± 1.9	24.6 ± 1.6	4.9 ± 0.2	5.8 ± 1.4	26.1 ± 2.7
CaveFlyer	7.3 ± 0.7	6.4 ± 0.1	5.8 ± 0.9	9.5 ± 0.2	6.2 ± 0.6	4.9 ± 0.3	6.8 ± 0.4	5.5 ± 0.5
CoinRun	9.4 ± 0.3	5.4 ± 0.4	9.4 ± 0.2	9.9 ± 0.0	9.8 ± 0.1	6.7 ± 0.1	9.8 ± 0.1	9.1 ± 0.4
Jumper	8.6 ± 0.1	3.6 ± 0.5	8.2 ± 0.1	8.7 ± 0.1	8.7 ± 0.2	5.7 ± 0.1	7.9 ± 0.2	8.1 ± 0.4
Chaser	3.7 ± 1.2	6.3 ± 0.7	7.0 ± 0.6	10.7 ± 0.4	7.5 ± 0.8	2.6 ± 0.1	4.7 ± 1.8	6.9 ± 1.1
Climber	6.9 ± 1.0	6.2 ± 0.8	8.6 ± 0.6	10.2 ± 0.2	10.2 ± 0.7	3.5 ± 0.3	7.7 ± 0.4	11.4 ± 0.2
Dodgeball	6.4 ± 0.6	2.0 ± 1.1	7.3 ± 0.8	5.5 ± 0.5	4.9 ± 0.3	3.3 ± 0.1	5.8 ± 0.5	5.3 ± 0.5
Heist	6.1 ± 0.8	1.2 ± 0.4	6.2 ± 0.6	7.4 ± 0.4	4.5 ± 0.3	7.1 ± 0.2	9.4 ± 0.1	7.0 ± 0.6
Leaper	5.5 ± 0.4	6.4 ± 0.4	5.0 ± 0.9	9.3 ± 1.1	8.3 ± 0.7	4.3 ± 2.3	4.3 ± 2.0	8.1 ± 0.9
Maze	9.1 ± 0.2	4.1 ± 0.5	8.5 ± 0.3	9.0 ± 0.2	6.4 ± 0.5	9.4 ± 0.3	9.6 ± 0.1	7.4 ± 0.4
Miner	11.3 ± 0.3	9.7 ± 0.4	12.0 ± 0.3	11.3 ± 1.0	11.5 ± 0.5	1.9 ± 0.6	9.0 ± 0.8	11.9 ± 0.2
Table 2: Train score of ProcGen games trained on 200 levels for  ${25M}$  environment steps. We compare our algorithm to PPO, PLR, UCB-DrAC, PPG, IDAAC and LEEP. The mean and standard deviation are computed over 10 runs with different seeds.
Figure 7: Aggregate metrics for all ProcGen games: mean, median and IQM scores (higher is better) and optimality gap (lower is better), with  $95\%$  CIs highlighted in color. ExpGen outperforms the contending methods in all measures.
We observed that policies trained to explore, using maximum entropy RL, exhibited generalization of their exploration behavior in the zero-shot RL setting. Based on this insight, we proposed ExpGen—a ZSG-RL algorithm that takes a maxEnt exploration step whenever an ensemble of policies trained for reward maximization does not agree on the current action. We demonstrated that this simple approach performs well on all ZSG-RL domains of the ProcGen benchmark.
One burning question is why does maxEnt exploration generalize so well? An intuitive argument is that the maxEnt policy in an MDP is invariant to the reward. Thus, if for every training MDP there are many different rewards, each prescribing a different behavior, the maxEnt policy has to be invariant to this variability. In other words, the maxEnt policy contains no information about the rewards in the data, and generalization is well known to be bounded by the mutual information between the policy and the training data [Bassily et al., 2018]. Perhaps an even more interesting question is whether the maxEnt policy is also less sensitive to variations in the dynamics of the MDPs. We leave this as an open theoretical problem.
Another consideration is safety. In some domains, a wrong action can lead to a disaster, and in such cases, exploration at test time should be hedged. One possibility is to add to ExpGen's policy ensemble an ensemble of advantage functions, and use it to weigh the action agreement [Rotman et al., 2020]. Intuitively, the ensemble should agree that unsafe actions have a low advantage, and not select them at test time.
Finally, we point out that while our work made significant progress on generalization in several ProcGen games, the performance on Dodgeball remains low for all methods we are aware of. An interesting question is whether performance on Dodgeball can be improved by combining invariance-based techniques (other than IDAAC) with exploration at test time, or whether Dodgeball represents a different class of problems that requires a completely different approach.
Acknowledgments The research of DS was Funded by the European Union (ERC, A-B-C-Deep, 101039436). The research of EZ and AT was Funded by the European Union (ERC, Bayes-RL, 101041250). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of the Schmidt Career Advancement Chair in AI.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29320, 2021.
Raef Bassily, Shay Moran, Ido Nachum, Jonathan Shafer, and Amir Yehudayoff. Learners that use little information. In Algorithmic Learning Theory, pages 25-55. PMLR, 2018.
Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der Meulen, et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997.
Martin Bertran, Natalia Martinez, Mariano Phielipp, and Guillermo Sapiro. Instance-based generalization in reinforcement learning. Advances in Neural Information Processing Systems, 33: 11333-11344, 2020.
Dimitri Bertsekas. Dynamic programming and optimal control: Volume I, volume 1. Athena scientific, 2012.
Craig Boutilier, Chih-wei Hsu, Branislav Kveton, Martin Mladenov, Csaba Szepesvari, and Manzil Zaheer. Differentiable meta-learning of bandit policies. Advances in Neural Information Processing Systems, 33, 2020.
Kyunghyun Cho, Bart Van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1282-1289. PMLR, 2019.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048-2056. PMLR, 2020.
Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In International Conference on Machine Learning, pages 2020-2027. PMLR, 2021.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407-1416. PMLR, 2018.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123, 2018.
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why generalization in rl is difficult: Epistemic pomdpds and implicit partial observability. Advances in Neural Information Processing Systems, 34:25502-25515, 2021.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pages 2681-2691. PMLR, 2019.
YF Hendrawan. Comparison of hand follower and dead-end filler algorithm in solving perfect mazes. In Journal of Physics: Conference Series, volume 1569, page 022059. IOP Publishing, 2020.
Huaxiu Yao, Linjun Zhang, and Chelsea Finn. Meta-learning with fewer tasks through task interpolation. arXiv preprint arXiv:2106.02695, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 11920-11931. PMLR, 2021.
Chang Ye, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. Rotation, translation, and cropping for zero-shot generalization. In 2020 IEEE Conference on Games (CoG), pages 57-64. IEEE, 2020.
A Hidden Maze Experiment
In this section we empirically validate our thought experiment described in the Introduction: we train a recurrent policy on hidden mazes using 128 training levels (with fixed environment colors). Figure 8a depicts the observations of an agent along its trajectory; the agent sees only its own location (green spot), whereas the entire maze layout, corridors and goal, are hidden. In Fig. 8b we visualize the full, unobserved, state of the agent. The train and test results are shown in Fig 9, indicating severe overfitting to the training levels: test performance failed to improve beyond the random initial policy during training. Indeed, the recurrent policy memorizes the agent's training trajectories instead of learning generalized behavior. Hiding the maze's goal and corridors leads to agent-behavior that is the most invariant to instance specific observations—the observations include the minimum information, such that the agent can still learn to solve the maze.
This experiment demonstrates why methods based on observation invariance (e.g., IDAAC Raileanu and Fergus [2021]) do not improve performance on Maze-like tasks, despite significantly improving performance on games such as BigFish and Plunder, where invariance to colors helps to generalize.
Figure 9: PPO performance on the hidden maze task, indicating severe overfitting. Train (Orange) and Test (blue) performance are displayed for 10 seeds, alongside their means (highlighted in bold).
B Maximum Entropy Policy
This section elaborates on the implementation details of the maxEnt oracle and provides a performance evaluation of the maxEnt policy.
B.1 Generalization Gap of maxEnt vs PPO across all ProcGen Environments
Recall from Section 4.2 that the (normalized) generalization gap is described by
$$
\left(\mathcal {R} _ {e m p} (\pi) - \mathcal {R} _ {p o p} (\pi)\right) / \mathcal {R} _ {e m p} (\pi),
$$
where  $R_{emp}(\pi) = \mathcal{R}_{train}(\pi)$  and  $R_{pop}(\pi) = \mathcal{R}_{test}(\pi)$ . Fig 10 shows the generalization ability of the maxEnt exploration policy compared to PPO, obtained from training on 200 training levels.
Figure 10: The normalized generalization gap  $[\%]$  of maxEnt and PPO for all ProcGen games, trained on 200 training levels and averaged over 4 seeds (lower is better).
The figure demonstrates that the maxEnt exploration policies transfer better in zero-shot generalization (achieve a smaller generalization gap) across all ProcGen games apart from Ninja. This holds true
even in environments where the ExpGen algorithm is on par but does not exceed the baseline, pointing to the importance of exploratory behavior in some environments, but not in others.
B.2 Computing the maxEnt Oracle
The maxEnt oracle prescribes the maximum intrinsic return achievable per environment instance. To simplify the computation of the maxEnt oracle score, in this section, we evaluate the maxEnt score using the  $L_{0}$  instead of the  $L_{2}$  norm and use the first nearest neighbor ( $k = 1$ ). In Maze, we implement the oracle using the right-hand rule [Hendrawan, 2020] for maze exploration: If there is no wall and no goal on the right, the agent turns right, otherwise, it continues straight. If there is a wall (or goal) ahead, it goes left (see Figure 3). In Jumper, we first down-sample (by average pooling) from  $64 \times 64$  to  $21 \times 21$  pixels to form cells of  $3 \times 3$ , and the oracle score is the number of background cells (cells that the agent can visit). Here we assume that the agent has a size of  $3 \times 3$  pixels. In Miner, the "easy" environment is partitioned into  $10 \times 10$  cells. The maximum entropy is the number of cells that contain "dirt" (i.e., cells that the agent can excavate in order to make traversable).
Figure 11: Generalization ability of maximum entropy and extrinsic reward policy: (a.top row) Score of maximum entropy policy, normalized by the oracle score. (a(bottom row) Success Rate of maximum entropy policy. (b) Score of extrinsic reward policy. Training for maximum entropy exhibits a small generalization gap in Maze, Jumper and Miner. Average and standard deviation are obtained using 4 seeds.
In Fig. 11a the top row describes the intrinsic return of the maxEnt policy, normalized by the oracle's return (described above). We draw a comparison between agents with memory (GRU) and without. For the Maze environment (top-right), the agent achieves over  $90\%$  of the oracle's performance when employing a memory unit (GRU). For the Jumper and Miner environments, the agents approach  $60\%$  and  $80\%$  of the oracle's score, respectively.