Implicit Curriculum in Procgen Made Explicit
Authors: Zhenxiong Tan
Kaixin Wang
Xinchao Wang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Reinforcement Learning
Curriculum Learning
Procedurally Generated Environment
Primary area: reinforcement_learning
TL;DR: We find an implicit curriculum in Procgen and introduce C-Procgen for controlled context learning research.

Abstract: Procedurally generated environments such as Procgen Benchmark provide a testbed for evaluating the agent's ability to robustly learn a relevant skill, by situating the agent in ever-changing levels. The diverse levels associated with varying contexts are naturally connected to curriculum learning. Existing works mainly focus on arranging the levels to explicitly form a curriculum. In this work, we take a close look at the learning process itself under the multi-level training in Procgen. Interestingly, the learning process exhibits a gradual shift from easy contexts to hard contexts, suggesting an implicit curriculum in multi-level training. Our analysis is made possible through C-Procgen, a benchmark we build upon Procgen that enables explicit control of the contexts. We believe our findings will foster a deeper understanding of learning in diverse contexts, and our benchmark will benefit future research in curriculum reinforcement learning.

Implicit Curriculum in Procgen Made Explicit
Zhenxiong Tan' National University Singapore chenxiong@u.nus..... .uu..uuuuu
Kaixin Wang"t
National University of Singapore kaixin96. wang@gmail. com
Xinchao Wang
National University of Singapore xinchaoθnus sdu.sg
Abstract
Procedurally generated environments such as Procgen Benchmark provide testbed for evaluating the agent's ability to robustly learn a relevant skill, by situating the agent in ever-changing levels. The diverse levels associated with varying contexts are naturally connected to curriculum learning. Existing works mainly focus on arranging the levels to explicitly form curriculum. In this work, take close look at the learning process itself under the multi-level training in Procgen. Interestingly, the learning process exhibits gradual shift from easy contexts to hard contexts, suggesting an implicit curriculum in multi-level training. Our analysis is made possible through C-Procgen, benchmark we build upon Procgen that enables explicit control of the contexts. We believe our findings will foster a deeper understanding of learning in diverse contexts, and benchmark will benefit future research in curriculum reinforcement learning.
Introduction
Training deep reinforcement learning agents in singleton environments is often susceptible to over- fitting (Zhang al., [2018a Song al. 2019 Cobbe et al. 2019 2020). Recently, procedurally generated environments, which take into account the agent's generalization ability, have received increasing attention (Justesen al. 2018 Risi and Togelius 2020 Cobbe et al. 2020). In those environments, the agent faces an entirely new level in each episode (e.g.. different maze layouts). Succeeding in such ever-changing levels demands that the agent learn robust policies. Compared to singleton environments, the diverse levels in procedurally generated environments pose new challenges for reinforcement learning training, calling for a better understanding of how learning progresses in this multi-level setting.
In procedurally generated environments, each level is associated with some environment parame- ters (Dennis et al. (2020) that vary across levels, e.g., the agent's 's max speed, the size of the maze, or the number obstacles. We refer these various combinations contexts of the environment. The varying contexts give rise to levels of varying difficulties, suggesting possible connection curriculum learning (Bengio et al. (009). While some previous works explore this connection, they mainly focus on arranging the order of levels to form curriculum (Jing al., 2021b), or adaptively generating new levels to create a curriculum (Wang et al. 2019 2020 Dennis al. 2020 Jiang et al. 222 Parker-Holder et al. 2022). The learning process itself in the face of diverse levels less investigated.
environment parameters as the
'Equal Contributions
CCurently Microsoft Research Asia, work done during his time in NUS.
Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To bridge this gap, we take closer look at how reinforcement learning progresses under multi-level training Procgen. In particular, we do not consider any form of level prioritization, i.e., the levels are sampled uniformly as in the standard setting of Proegen. To facilitate our study, we build new benchmark, C-Procgen, which provides explicit access and control over the environment parameters. We then investigate how some metrics (e.g., the agent's i reward and the loss) under different contexts evolve as the learning progresses. In addition, we inspect how learning will be affected when some contexts are masked out or additional contexts are included.
We find that the learning process exhibits gradual shift from easy contexts to hard contexts, despite the absence of explicit curriculum (e.g., level prioritization). The return and the loss concentrate on easy tasks in the early stage of training, and gradually shift towards hard tasks as the training goes on. In addition, we observe a mismatch between the loss and the number of samples across different contexts in the middle stage of training. Specifically, a large portion of samples collected by the agent are from hard contexts but the loss is concentrated in easy tasks. This implies that considerable number of samples might be wasted and contribute little to updating the policy. Moreover, we find that this mismatch has potential connection to how much performance gain can be achieved from modifying the sample distribution by prioritizing levels (Jiang et al., 2021b).
When the training contexts are partially masked out (easy, medium, or hard), we observe mixed results: in some games, the excluded contexts are essential for the agent form an implicit curriculum, and masking them out will impact the performance; in other games, the remaining contexts are still able to provide useful information for the agent learn transferrable policies. Besides, we find that expanding the set of training contexts helps expedite the learning shift from easy to hard contexts.
Our C-Procgen benchmark exposes the environment parameters in the black-box context generation process in Procgen. It preserves the advantages of the Procgen benchmark, such as being compre- hensive and challenging, while greatly expanding its use by adding explicit control of the contexts. C-Procgen can be useful in curriculum learning and contextual reinforcement learning.
In summary, paper makes the following contributions:
We build C-Procgen, benchmark that enhances Procgen with accessible and controllable environment parameters, which can be of interest to curriculum learning and other areas.
Our work offers insights into the learning dynamics under the multi-context training in Procgen, and makes several interesting observations.
We reveal that an implicit curriculum occurs in the multi-context training in Procgen despite that the contexts are uniformly sampled.
2 Related works
2.1 Procedurally generated environments
Recently, procedural content generation has been adopted to design RL environments that consider the agent's ability to generalize as a central component of success. In procedurally generated environments, each episode begins with a new context created from a generation process, which is analogous to a "level" in video games. Across different episodes, the context is varied in certain task-relevant (e.g., number of obstacles) or task-irrelevant (e.g.. background image) aspects. One popular procedurally generated environment suite is the Proegen benchmark (Cobbe et al., 2020), which consists of 16 challenging Atari-like video games. However, in Procgen, multiple choices during the procedural generation process are governed by a single random seed. The underlying parameters that control the context generation are not accessible and visible to researchers. In other words, the context is implicitly determined by a black-box process. As pointed out by (Kirk et al., 2023). the evaluation protocols supported by such environments are limited to varying the size of the context set. Our work empowers Procgen by making many factors of variation visible and controllable, allowing researchers to explicitly configure the environment parameters.
Additionally, several works like Minigrid (Chevalier- Shhvallie-Booiserrtt et al. 2018) and Minihack (Samvelyan et al.] (021) offer flexible frameworks for creating customizable reinforcement learning environments with adjustable parameters to generate different contexts. However, their environments are lightweight and lack convenient APIs for controlling environment parameters.

---

Original Procgen
Our C-Procgen
explicitly
visibility
gravity
game mode
implicitly
air control
level ID
bomb probability
nax speed
nax jump
jump charge
gap range
gap increment
section number
Figure I: An illustrative example comparing the original Procgen and our C-Procgen.
Closely related to our work is the CARL benchmark (Benjamins al. 2021 which collects environments from several domains and makes their contexts configurable. While CARL mostly focuses on control tasks (including classic control environments from OpenAI Gym (Brockman et al.) 2016) and physical simulations), our C-Procgen aims to extend the widely used Procgen benchmark.
2.2 Curriculum learning
Curriculum learning (Bengio al. 2009 is a general strategy in machine learning that organizes training examples or tasks in order to boost learning efficiency. The varying contexts in Procgen pose challenges at different levels of difficulty, which is naturally connected curriculum learning. Jiang et al. (b propose to prioritize the contexts that have higher estimated learning potential when sampling new contexts for the next training levels. This induces an emergent curriculum of increasingly difficult levels. Another line of work aims to automatically generate novel contexts that adapt to the agent's evolving ability (Wang et al. 2019 2020; Dennis et al.) 2020] Jiang et al. 202 Parker-Holder et al. 2022). While these works focus manipulating the distribution of contexts to explicitly form a curriculum, our paper reveals that even when the contexts are uniformly sampled, the agent's learning process still implicitly follows a curriculum.
The prior works mentioned above can also be viewed as form of automatic curriculum learn- ing (Portelas et al., 2020). In this regard, our work is also closely related to the TeachMy benchmark (Romac al.] 202T). TeachMy/Agen extends the Bipedal Walker environment from Ope- nAI Gym (Brockman et al. | (0116) to parametric ones that enable controllable procedural generation, providing a testbed for benchmarking automatic curriculum learning algorithms. Similarly, our work aims to make the more challenging Procgen benchmark configurable.
3 C-Procgen: Controllable Contextual Procgen
In this work, we first build new benchmark that enhances the challenging Procgen suite aiming to investigate how learning progresses under the multi-context setting. Specifically, we carefully refactor the source code of Procgen and expose over 100 environment parameters that determine the context generation, giving users more control and insight into the previously opaque generation process. The resulting benchmark is C-Procgen Controllable Contextual environment parameters and is well-suited for Curriculum reinforcement learning research. Throughout this paper, use the term "environment parameter" (Dennis et al.) 2020> to refer to any configurable factor that can vary across episodes (e.g.. the agent's 's max speed, maze size, or number of obstacles). The term "context" refers specific configuration determined by the combination all environment parameters.
which augments the original Proegen with
As illustrated in Figure T the original Procgen only provides level ID to implicitly control the generation of context. In contrast, our C-Procgen allows users to directly control the context
'The source code of C-Procgen be found GitHub: http:://gttubbcoom///xxa///CCCC

---

via explicit configuration of the environment parameters. Thus, C-Procgen can serve as a high- quality testbed for research in the area of Contextual Reinforcement Learning (Hallak et al. (0135). Furthermore, C-Procgen enables users to integrate the context information into the learning algorithms, which can be useful in Automatic Curriculum Learning (Portelas et al. 2020).
In addition, C-Proggen also provides several engineering enhancements for improved usability:
Unlike the original Procgen, which assigns the same configuration to all environments during the initialization of vectorized environments, C-Proegen allows distinct contexts to be assigned to each environment. This flexibility helps expose the algorithm to more diverse game contexts during training.
Users can dynamically modify the context of each environment between episodes without creating new instance. This feature is particularly beneficial for curriculum reinforcement learning, where environments need dynamically changing contexts throughout the learning process.
C-Procgen offers interface that lets users track detailed context information during the learning process, enhancing transparency and facilitating better analysis.
In summary, we would like to highlight that C-Procgen enhances Procgen with controllable envi- ronment parameters while the introduced overhead during simulation is negligible. In the following sections, we will take advantage of the features provided by C-Procgen to conduct a detailed analysis of how the learning dynamics evolve across contexts in Procgen.
Learning Dynamics in Procgen
In this section, investigate the learning progress of Procgen under various contexts. For our experiments, we used C-Procgen, which faithfully simulates the same game logic and context distributions as the easy mode of the original Proegen benchmark. By leveraging the flexible context control features of C-Procgen, we recorded key metrics such as loss, entropy, episode length, average score, and the number of samples for each context. This approach provides a more detailed view of learning progress across different contexts.
Specifically, select nine enviromments from Procgen due their episodic contexts, which change with each game reset, resulting in unique configurations for each playthrough. We utilize the Prox- imal Policy Optimization (PPO) algorithm (Schulman et al. 2017) in our reinforcement learning experiments. For each of these selected environments, we perform five individual runs, each encom- passing 25 million steps, to ensure comprehensive and robust analysis. Notably, the experiment with C-Procgen allows for the generation of diverse environments under all possible circumstances, following the sample efficiency experimental protocols of the original Procgen benchmark.
4.1 A Shift from Easy Hard Contexts
To gain deeper insights into the learning process in C-Procgen, we first focus on the game Ninja. In the game Ninja, the agent navigates through field of suspended platforms while avoiding bombs. This environment contains six unique contexts, each determined by different environment parameters including section_nuu _um, gap_range and bomb_prob. For instance, the number of platforms in the game, determined by the contextual factor section. num, significantly impacts the gameplay. Moreover, the environment parameter gap_rre lapaggggeee determines the distance between the two platforms. Smaller values correspond to smaller gaps, thus implying easier transitions for the agent. Furthermore, a higher bomb_ bomb_prob escalates the likelihood of encountering bombs. For instance, Figure [ presents heat maps that visualize some metrics of the learning process across different contexts within the game Ninja a, where each grid indicates a unique context. [ Take, for example, the grid at coordinates context where the game features two platforms, and the gap_range and bomb_prob are set to their respective minimum values.
(2, 1); it corresponds
*Due their direct correlation, the environment parameters gap_range bomb_prob are combined on the same axis for simplification. Specifically, as gap_range increases, bomb_prob also increases. They are correlated in the following three combinations: when sap_range is [0], bomb_pooba __tobbrrbbb is O; when gap_range is [o, 1]. bomb_prob is 0.25; and when gap_range is [0, 1, 2], bomb _ombbb_robbbo is 0.5.

---

11
T2
T3
T4
T5
T5
range
T2
Score
T1
Std
Number of sections
0
High
Law
High
Fewer
More
Steps(M)
Score
Loss Sample
Samples
Dissecting the learning dynamies in Procgen with game Ninja as an example. Left: The training curve, where we mark five stages during training (TI-T5). Right: Heatmaps of three metrics at stage TI-T5, including average score (top row), loss per sample (middle row), and the number of samples (bottom row) across different contexts.
Figure 2:
Average Score. A notable phenomenon observed the Average Score for Ninja (depicted in blue heat maps, see Figure 2). The average score, which measures the agent's performance, varies significantly across different contexts. Notably, as early as stage T2, the agent demonstrated high scoring capability in the context (1, 0), while in other more challenging contexts such as (5, 2), high scores were only achieved at a later stage as T3- This led us to categorize the contexts into two types: "Easy contexts". where high scores scores were achieved much later. This progression mastering different contexts effectively forms a curriculum, subtly guiding the agent's learning process.
obtained early in training, and "Hard contexts", where high
Loss per Sample. Another important insight concerns the Loss per Sample Ninja (depicted in orange color heat maps, Figure 2 This metric represents the average absolute value of the sample-wise loss generated at each algorithmic step within a specific context.
More specifically, throughout the learning process, the focus of the loss per sample metric appears to shift progressively from simpler contexts to more challenging ones, potentially reflecting the order in which the agent learns to master different scenarios. Notably, this metric tends to concentrate on contexts that pose moderate level of difficulty for the agent those that are neither too easy nor too difficult. This phenomenon can be understood as follows: in contexts where the agent is either highly proficient or completely unfamiliar, it can make more accurate value judgments, resulting in fewer losses. However, in moderately challenging contexts, the agent encounters more variability rewards, leading to errors in value estimation and, consequently, higher number loss signals to drive its updates. This observation aligns with the core principle of curriculum learning, which stresses the importance guiding the agent through tasks that are neither too easy nor too difficult.
Samples Distribution. An analysis of the Samples Distribution (depicted in green color heat maps, see reveel insights about the distribution samples across different contexts within single iteration. While the probability of sampling each context remains constant during environment resets and does not change over time, the length of the episode within each context is not constant, leading to variation in the samples across contexts. Aside from the earliest stage at Ti, it is observed from the heat map that for majority of the learning process within the game Ninja, samples tend to concentrate within the hard contexts.
Figure 3 expands the scope of observation beyond the Ninja environment to encompass wider range of environments. We found that the observations made in the Ninja environment can be generalized other environments of the Procgen benchmark.
'The value ranges in the heatmaps of Figure 2 and Figure have been normalized. The range for the score is based the maximum and minimum values achieved by the agent the entire learning process, while the ranges loss per sample the number samples are based maximum minimum values each specific time point. The loss from PPO (Schulman et al.
2017 is in weighted of three terms: lpolicy dvalue and bentrop.

---

Climber
Maze
Dodge Ball
Platforms num
Maze size
Enemiees num
Coin Run
Helst
Boss Fight
T1
T2
T3
T4
T3
Uhreehoold
Keys
Round
Section num
Maze size
Leaper
Fruit Bot
Road lanes num
things num
Score
Low
High
Loss / Sample
Fower
More
Samples
Bad things
Figure 3: Score, loss per sample, and sample num across different environments and contexts. Each subplot represents a different environment, with heatmaps depicting the metric values at different time points and contexts.
4.2 A Mismatch between Loss and Sample Distribution
The metric "Loss per sample" indicates whether the agent has learned content in specific context to certain extent. As previously discussed, the agent's focus tends to gravitate toward contexts of moderate difficulty during learning. However, on which the agent focuses do not match the distribution of samples in the Procgen environments. This discrepancy suggests that significant portion of the samples may not providing sufficient loss signals to effectively guide the agent's 's learning. Particularly in the early stages of the process, easy contexts often promote the agent's initial progress, yet a majority of the samples the agent encounters come from hard contexts, which yield limited learning.
observe inconsistency: the contexts
To gain deeper understanding of this inconsistency, introduce measure called "Loss Production Efficiency" that captures the effectiveness of sample distribution in generating loss signals at the ith iteration throughout the learning process in Procgen. The Loss Production Efficiency is defined as the ratio between the loss generated by the current sample distribution across different contexts and the maximum achievable loss:
Enec Pf -1 E.
max.Ec 19
In this equation, Es represents the Loss Production Efficiency (LPE) for specific context from context space C the th iteration. Here, Iς refers the loss per sample for context e iteration i, and of denotes the proportion of samples for context out of the total samples in the same iteration. The

---

LPE for the entire learning process is represented by the average of the Loss Production Efficiencies across all iterations, given by E' + > E, where N is the total number of iterations.
notable observation reveals a potential cor- relation between the LPE and the performance improvement offered by the Prioritized Level Replay algorithm (Jiang et al. 20221b on Proc- gen games. The performance improvement is defined as Ascore scorema scoremo, where scoremo
120%
Leeper
Score Improvement
80%
Heist
scOrEpLR and scorepro are the final scores with and without PLR, respectively. The Prioritized Level Replay algorithm aims to enhance the learning process by augmenting data from sam- ples generated by high-loss levels. Interestingly, as illustrated in Figure 4 it appears that environ- ments characterized by lower LPE tend to hibit more substantial performance gains when the Prioritized Level Replay algorithm is em- ployed. This could be attributed to the algo- rithm's ability modify the sample distribution and increase LPE.
40%
Maze
Nijja
Bossfight
Climber
Coinun
40%
60%
80%
100%
Loss Production Efficiency
Figure 4: Loss Production Efficiency vs. Score Im- provement with Prioritized Level Replay in Proc- gen Games
Based the insights gained from the above findings, it becomes apparent that LPE plas a crucial role the learning speed within an environment. When designing environments, optimizing factors such as the distribution of contexts, maximum episode length, and conditions for terminating episodes can significantly impact LPE. By carefully manipulating these factors, it is possible to create environments that promote higher LPE, resulting in accelerated learning processes and improved agent performance.
5 Investigating Learning Dynamics under Context Reconfiguration
To investigate the influence of different training contexts on the reinforcement learning process, reconfigure the context setting of games using C-Procgen. Firstly, based on previous findings of loss per sample across different contexts, we grouped contexts into three categories. We then created game settings excluding these context groups, to test their necessity to learning progress. Secondly, we add new contexts into the games, aiming decrease discontinuity between existing contexts. We employed the Proximal Policy Optimization (PPO) algorithm, conducting 5 runs for each setting, consistent with the procedures Section 4
5.1 Partially Masking the Training Contexts
The experiments demonstrate the distinct roles of contexts in different games. As depicted in Figure 5 present the final average score and loss per sample heat maps for nine games when trained without certain contexts.
Overall, certain contexts appear to be crucial for the learning process in the game. For instance, in setting of Coinrun, removing key contexts causes the learning process to stall, preventing the agent from progressing through the game's various stages. Similar patterns are observed in setting 1 of Heist and setting 2 of Ninja. Morcover, generalization phenomenon was observed across multiple environments. In many cases, the agent was able to perform well even without training in certain contexts. Interestingly, the overall performance of the agent improved in some cases when certain contexts were excluded, such in setting 3 of Climber, Setting 3 of Coinrun, and both Setting 2 and Setting of Heist. One possible explanation for this improvement that the excluded contexts particularly difficult. By removing these challenging contexts, the agent may have experienced increased learning progress efficiency (LPE), which in turn enhanced its overall performance.
In summary, our observations indicate that certain contexts play crucial transitional role in the learning process. However, the benefit of having larger number or wider variety of contexts not always straightforward.
7

---

Climber
Maze
Dodge
Original
Setting
Setting
Betting
Original
Setting
Setting 2
Setting
Original
Setting1
Setting 2
Setting
Platforms num
Maze size
Enemies num
Run
Heist
Boss Fight
Original
Setting
Setting
Setting 3
Original
Setting
Switing 2
Setting
Original
Setting
Setting 2
Setting
Pit thresholl
Keys num
Round num
Section mum
Maze
Round health
Leaper
Fruit Bot
Ninja
Original
Setting
Setting
Setting 3
Original
Setting
Settingg 2
Satting
Original
Setting
Setting 2
Setting
Road lanes num
Good thinge sun
Gap range
Mord Bad things num
Section num
Water lanes num
Low
High
Low
High
Sample
Figure 5: Learning outcomes under various context settings. The leftmost panel for each game depicts the final average score and loss per sample under the original context setting after 25M training steps. Subsequent pancls show the metrics under reconfigured context settings. Regions enclosed in red represent the masked contexts, where average score testing was still performed.
5.2 Expanding the Training Contexts
In this section, we focus on the game Leaper as case study. We aim to provide a detailed exploration of how the introduction of new contexts in the original Proegen benchmark, can influence the learning progress, particularly by analyzing the changing distribution of loss during the learning process.
The game environment of Leaper requires the agent avoid road vehicles and make use of the logs on the river to reach the top of the map. Within this environment there are two environment parameters: the number of road lanes (road_lamd lanes_ __nmm and the number of water lanes (water_land lanes __nu). Under the original context setting of Leaper, only four contexts exist: (0, 0). (1, (2, 2), and (3, 3). This indicates that the game will have an equal number of water lanes and road lanes, with each context having an equal chance of being sampled at game initialization. In addition to the original context setting of Leaper, we have introduced two new context settings. Setting 1, we have new contexts to enhance the connectivity among the original context setting. In Setting 2, we permit both water _aaee__aaee___eeeeeeeeeeeeeeeeee num and road_ lanes. num to take any value within the set {0, 1,2,33. both Setting and Setting 2, each context has equal probability of being sampled.
With the original context setting, as observed from the stacked area chart in Figure o the loss during the early stages of learning is primarily focused on context 0, followed by context 1. The transition of loss focus context however, is relatively slow, and by the end of training, the loss has yet concentrated on context 3. In contrast, in Setting 1, due to the existence of more diverse contexts and stronger connectivity among them, the proportion towards context 3 as early around the 2 million steps. Comparing the results of the original setting and Setting is suggested that the expanded contexts serve as stepping stones, aiding in the agent's learning and adaptation across different contexts. When comparing Settings and 2, Setting also
loss generated by contexts
starts to shift

---

Original Setting
Expanding Setting
Expanding Setting
Lecs
road lames
road lanee
10
road
10
lanas
water
10
6
26
20
Stops (M)
Steps (M)
Steps/MI
contect
corEe011
contect
contest 3
Original contect
Expending contest
Figure 6: Loss Proportion and Score curve different expanded context settings
accelerates learning, but its effects appear less significant than those in Setting 1. This observation is consistent with the trends noted in S5.t
Moreover, upon observing the score curves and loss proportion of these three settings (see Figure D it is notable that the increase in the agent's loss proportion corresponds the time when the agent's average score in that context begins to rapidly grow. Conversely, when the agent's 's loss proportion begins to decline in context, it suggests that the agent has already become proficient in that context. Consistent with the findings in Section 4 when the loss begins to shift, the agent has indeed already mastered the respective context.
6 Conclusions
Procgen is widely used procedurally generated environment suite that has varying contexts across episodes. While previous works pay most attention to explicitly creating a curriculum of contexts, we seek to investigate the learning process itself under the multi-level training in Procgen. To this end, we build C-Procgen, which enhances Proegen with explicit access and control of context parameters. With C- C-Procgen, we analyze the learning dynamics in Procgen across contexts and reveal that an implicit curriculum happens as the learning proceeds. Our C-Procgen can also find its use other curriculum learning research. In the future, will explore if this implicit curriculum connects the number contexts and how we can take advantage of it to boost performance.
Limitation
We did not provide specific evaluation protocols for C-Procgen. Even though Procgen offers comprehensive set of games, we believe there are many other types of games that it doesn' 't cover. While conducted several experiments, notably those involving modified context distributions, we did not explore dynamic context distributions - where the context distribution evolves with the learning progress. We believe that studies in this direction hold significant value for curriculum learning. We plan to study this deeper in our future work.
Acknowledgement
This work is supported by the Advanced Research and Technology Innovation Centre (ARTIC), the National University of Singapore under Grant (project number: A-0005947-21-00 project reference: ECT-RP2), and the Singapore Ministry of Education Academic Research Fund Tier (WBS: 4-0009440-00100).

---

References
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pages 41-48, 2009,
Carolin Benjamins, Theresa Fimer, Frederik Schubert, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. CARL: A benchmark for contextual and adaptive reinforcement learning. CoRR, abs/21 10.02102, 2021. URL https:/ts://aatt http://axxvv..oo wttp://www..rgg/////.......1 abs abtpg//..//111....o/ 02100
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv: nete::8100..270 2018.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1282-1289. PMLR, 2019.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages PMLR, 2020.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. arXiv preprint arXiv: Arxiv-2012.00 02096, 2020.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. CoRR, abs/1502.00225, 2015. URL http::://aaxiv..rraaabaaa. 15020022200029
Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances Neural Information Processing Systems, 1011888891977 2021a.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pages 4940-49500 PMLR, 2021b.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural 10729, Arxi0-1806.10729, 2018.
level generation. arXiv preprint arXiv: Arxiv-1806.
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. J. Artif. Intell. Res., 76:201-264 2023. doi: N0.1613/jair...1417.
httptt
1613/jair. 1.11111......1.....1111111111111
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In International Conference Machine Learning, pages 17473-17498, PMLR, 2022.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Oudeyer. Automatic curriculum learning for deep RI: A short survey. In Christian Bessiere, editor, Proceedings of the Twenty-Ninn Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 4825. ijcai.org, 2020. doi: 2224966/jcai.2222066 URL httpttts////o.rrr//// //96/ 1jjci.2020/81. 2020/6711
Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence, 2(8):428-44) -466686,, 2020.
Clément Romac, Rémy Portelas, Katja Hofmann, and Pierre-Y Yves Oudeyer. Teachmyagent: benchmark for automatic curriculum learning deep RL. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 Proceedings of Machine Learning Research, pages 9052-9063. PMLR. 2021. http:tt///
mlr, www.ppresevviie wwteesssvv3///oacccccc http://oma2aaa.amm.
10

---

Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Kuttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https //openreview.netee wew..forrun.dddkB.lleet.....
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. arXiv preprini UrXi:::912.02973 2019.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 142-151, 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. International Conference on Machine Learning, pages 9940-9951, PMLR, 2020.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. CoRR, abs/1806.07937, 2018a. URL httphtttpp//axxi..rrg org/ abs/1806.00
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.0688 2018b.

---

A Experiment Details
We the PyTorch codebase from Raileanu and Fergus (02IT) to train PPO Schulman et al. (2017)
agents on Procgen games. The agent is parameterized by the IMPALA network architecture Espeholt
al. (018). Unless otherwise stated, we follow the hyperparameters used in Cobbe al. (0200) for
the easy mode of Procgen, as summarized in Table [
Similar to the original Procgen, our C-Procgen still enjoys high simulating speed. On server with Intel Xeon CPU cores and 56GB RAM, the FPS of Procgen is around 750 and the FPS of C-Procgen is around 710. Regarding the training cost, training PPO agent NVIDIA T4 GPU for 25M steps takes approximately 2.5 3 hours.
Table 2
provides information the episodic contexts used in experiments in Sections 4 and 5, including their descriptions and ranges values.
HYPERPARAMETER
VALUE
0,999
0.95
# timesteps per rollout
256
epochs per rollout
3
# minibatches per epoch
8
entropy bonus clip range reward normalization learning rate # workers # environments per worker
0.01
0.2
no
5e-4
1
64
total timesteps optimizer
25M
Adam
LSTM
no
frame stack
no
Table 1: Hyperparameters and their values
12

---

GAME
ENVIRONMENT PARAMETER
DESCRIPTION
VALUES
Determines the hcalth durability of each round (boss starship). Higher val- ues indicate
Numher of rounds
[1,23]]
resilient hoss that requires damage defeated.
bossfight
Specifies the number of rounds in the game. Each round represents an en- [,, 4,5]
Round health
counter with the boss starship.
Specifies the number platforms the game. Each platform represents step the player to climb.
climber
Number platforms
11,
Determines the number of sections present in each game.
Number of sections
[0, 1,23,4
coinrun
Affects the generation of pits the game. When the value smaller, there is higher probability of pits appearing game. Conversely, a larger value reduces the likelihood pits. This environment parameter interacts with several other enviromment parameters, which are further detailed code documentation under the "context de- tails" section.
threshold
[0.
Specifies the number of enemies present in the game. The player must dodge the balls thrown by these enemies and eliminate them progress.
dodgeball
Number of enemies
[1,2,3,4]
Specifies the number of fruits that the player can collect along the way. Col- lecting a piece of fruit rewards the player with a positive reward.
Number of good things
0,
fruitbot
Specifies the number of non-fruit ob- jects that the player must avoid. Mistak- enly collecting non-fruit object results in larger negative reward.
Number of bad things
0,1,
Specifies the total number of keys re- quired to unlock the locks and suc- cessfully complete the heist. The player must collect these keys scattered throughout the game.
Number of keys
[1,2,3]
heist
Determines the size of the in which the heist takes place. The is generated using Kruskal's algorithm and serves the environment for the player to navigate and find the hidden gem.
layout
Size of maze
[3.5,7.9]
Specifies the number of lanes in which cars move. The player must cross these lanes to reach the finish line and earn reward.
Number of road lanes
[0, 2,
leaper
Specifies the number of lanes with logs river.
Number of water lanes
[0, 1, 2,
maze
Size of maze
Determines the size of the maze layout.
[3,4, ... 15]
Specifies the number of narrow ledges that the player, ninja, must jump across.
Number sections
|,,
ninja
Controls the range of gaps hetween the ledges that the ninja must traverse.
Gap range
[0.1,2]
Table 2: Descriptions of the contexts used in our experiments.
13

---

References
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference machine learning, pages 2048-2056, 2056. PMLR, 2020.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukeuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In Jennifer G. Dy and Andreas Krause, editors, Proceedings the 35th International Conference Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1406-14155 PMLR, 2018. URL http' //proceedings. mlr press/v80 aspeholt18a. html
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement
learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8787-8798. PMLR, 2021. URL http .ml.r...ml......mmm.m......... //proceedings.uo mlr ppwsssvv39/eeeeepee ral.///allamm2ll.llla....
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims outlined in the abstract and introduction are well-aligned with the detailed discussions and findings presented throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper do discuss the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
There is no theoretical result in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The tools could be compiled any devices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
Yes, the code attached.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppullc//uuuii wwwtii/gguie///eeeeceoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training code is on the GitHub.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: This paper do not contain information about the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provice the sufficient information the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://:////iiss..eeee httpr//s...cpuuoo/cttaassssueeaa
Answer: [Yes]
Justification:
The answer is yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper do no discuss these.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No data model realsed in this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The answer is yes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee wwworeewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
19

---

Answer: [Yes]
Justification: Full doc provided.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No experiments and research with human subjects in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
We did not describe these.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "MZb1tpe4ad", "reviewer": "Reviewer_h4d8", "summary": "This paper presents a benchmark called C-Procgen that builds on the existing Procgen benchmark by allowing access and control of the context parameters. Furthermore, this work investigates how learning progresses for an RL agent in the absence of a curriculum given a uniform distribution over levels. The experiments demonstrate that throughout the training the agent gets better at certain contexts first, called easy contexts with high scores and losses early on, and later on it progresses in others, i.e., hard contexts, where low scores and losses occur initially. The authors consider such progress as an implicit curriculum. In addition, their evaluation of the Level Replay algorithm in C-Procgen indicates that performance gains occur in environments with lower loss production efficiency, an effectiveness measure they introduce. Finally, they investigate the effects of partially masking and expanding training contexts, and show that some contexts, especially the ones that present moderate difficulty, are more critical to the learning progress than the others.", "questions": "- Are the ranges of score, loss per sample, and number of samples consistent/fixed among training stages (Figures 2 and 3) and settings (Figure 5)?", "limitations": "-  Please, check the weaknesses section for the comment about the lack of investigation of curriculum learning approaches developed for contextual settings.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed benchmark provides a low degree of flexibility to research in the curriculum learning literature; hence, it is a very worthwhile contribution in itself.\n\n- The illustrations presented in the paper depict an easy-to-understand picture of how an RL agent learns in the contextualized Procgen environment with a uniform distribution over configurations.\n\n- Section 4 investigates masking or expanding training contexts and thus showcases interesting and valuable results for curriculum learning researchers.\n\n- The paper is well-written and makes a significant effort to describe the implicit curriculum phenomenon.", "weaknesses": "- Most of the observations on learning dynamics shared in Section 3 may be because the studied environments can terminate before reaching the maximum number of steps in an episode. It would be more informative to consider the termination conditions, the reward functions, and the dynamics of these environments when evaluating how the implicit curriculum progresses.\n\n- As most of the studied environments allow for early termination, I wonder if this causes the misinterpretation that there is an implicit curriculum. In essence, there is no curriculum being formed by the agent, but the agent progresses in certain contexts before the others, which is a valuable insight. Nevertheless, it may be misleading to say that the agent's focus shifts during training without a curriculum, as done by the authors in Section 3.2.\n\n- Section 3.2 focuses on the Level Replay algorithm and its performance in C-Procgen environments, but this study does not investigate curriculum learning approaches such as [1,2].\n\n[1] Klink, P., Yang, H., D’Eramo, C., Peters, J., & Pajarinen, J. (2022, June). Curriculum reinforcement learning via constrained optimal transport. In International Conference on Machine Learning (pp. 11341-11358). PMLR.\n\n[2] Huang, P., Xu, M., Zhu, J., Shi, L., Fang, F., & Zhao, D. (2022). Curriculum reinforcement learning using optimal transport via gradual domain adaptation. Advances in Neural Information Processing Systems, 35, 10656-10670.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the constructive feedback from the reviewer-h4d8 and will respond to the points raised as follows.\n\n---\n\n**W1. Termination Conditions**\n\nWe appreciate the insightful feedback from reviewer-h4d8. \nWhile termination conditions are indeed an important aspect to consider, they do not detract from the observations reported in our study.\n\n1. As noted, certain environments in our study indeed terminate before reaching the maximum number of steps per episode. This premature termination can shift the sample distribution across different contexts from what might be expected under uniform distribution conditions. However, as highlighted, this does not alter our finding that the sample distribution does not align with the distribution of loss signals per sample in each context, which may lead to inefficiencies in curriculum learning.\n\n2. Additionally, we concur with reviewer-h4d8 that incorporating a deeper analysis of termination conditions would enrich the information provided in Section 3! In our revision, we would love to also expand our discussion to include more details on the reward functions and dynamics of these environments, thereby providing a more comprehensive evaluation of how the implicit curriculum progresses.\n\n\n---\n\n**W2. Interpretation of Implicit Curriculum**\n\nThanks to reviewer-h4d8 for the thorough review and the insightful queries raised.\nIt appears that our phrasing may have led to a misunderstanding regarding the interpretation of the 'implicit curriculum' and the intent of Section 3.2.\n\n1. Firstly, regarding the interpretation of the 'implicit curriculum', we believe that the combination of different game contexts generated by various `level` inherently forms an implicit curriculum. \nThis does not necessarily conflict with early termination of episodes. \nAs reviewer-h4d8 noted, the agent does not deliberately form a curriculum from the environmental contexts; however, for the agents, these varying contexts of difficulty inherently serve as a curriculum, guiding their learning progression.\n\n2. Additionally, I would like to clarify a point regarding the agent's focus during training. Our discussion in the manuscript seems not assert that '*the agent's focus shifts during training without a curriculum*'.\nInstead, we examine how the distribution of loss signals and samples across contexts influences the learning process.\nWe introduce the concept of Loss Production Efficiency to assess this dynamic. \nThis metric helps us understand the effectiveness of a curriculum and the agent's interaction with it.\n\nWe appreciate the opportunity to clarify these points and will ensure that our revision reflects these clarifications to avoid any potential misinterpretations.\n\n\n---\n\n**W3. Evaluation Metrics**\n\nThank you for your observation regarding Section 3.2, which indeed focuses on utilizing the Prioritized Level Replay (PLR) to illustrate the mismatch between loss and sample distribution in the C-Procgen environments.\n\nWe are grateful to the reviewer for bringing to our attention the studies by Klink et al. (2022) and Huang et al. (2022). \nUpon a thorough review of these works, we agree that they offer valuable insights and present methodologies that are excellent and relevant to our field of study.\nThese studies provide advanced approaches to curriculum reinforcement learning, which could enrich our understanding and application of similar concepts in future research.\n\nHowever, due to the significant variability across games in Procgen, replicating similar presentations to demonstrate the learning dynamics of agents under a curriculum poses challenges.\nEach game's unique characteristics and context spaces necessitate a tailored approach when applying curriculum learning strategies, which our current focus on PLR aims to address in context-specific manners.\n\n---\n\n**Q1. Consistency of Metrics**\n\nIn our Figures 2, 3, and 5, the range of the score metric is consistent across different training stages and settings. \nFor other metrics, they are normalized on each heatmap for clarity and visibility.\nComparing these metrics, especially 'loss per sample', at different times can be challenging due to their significant scale variability during training. \nWe recognize the importance of this clarification and will address this explicitly in the versions to ensure the data's context and limitations are well understood.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5SwlT9ambr", "reviewer": "Reviewer_7pFG", "summary": "After rebuttal\n\nI think C-Procgen is a useful contribution in itself. Secondly, the author's rebuttal has persuaded me that their analysis is novel, and can be useful. I especially like the analysis of LPE and how this relates to PLR' relative performance to PPO.\n\nI am really impressed with the authors running my suggested LPE experiment on such short notice, and I think the preliminary results are promising.\n\n----------------------\n\n\n\n\nThis paper firstly introduced C-Procgen, a version of procgen where each context variable can independently and controllably be changed, instead of the default behaviour of changing everything based on only a single seed.\nNext, they analyse the distribution of contexts during training and find an implicit curriculum happens, in particular, the lengths of the episodes changes for different contexts as the agent learns. They also measure the loss per sample, and split that across contexts. They find a non-uniform split, and this also changes over training.\nFinally, they investigate different procedures of changing the contexts, and the effect this has on learning.", "questions": "- what does score improvement mean in fig 4?\n- Line 212-216, should it be LPE? If not, what does LSE mean?\n- line 224, should that not be section 3?\n- The algorithm introduced by [9] is called Prioritised Level Replay (or PLR) and not just Level Replay.\n- Figure 5 has a lot of information and is somewhat hard to parse. I think it could be made clearer by having some summary plots (e.g. average test reward) as a function of the training contexts used.", "limitations": "- There are no suggestions or (in my opinion) useful takeaways from the analysis.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- C-Procgen is very useful in itself. Being able to control different aspects separately is great. I would recommend releasing it as a standalone library.\n- The observation that due to different episode lengths, the amount of data from different contexts changes over time as the agent learns, is interesting.\n- Similarly, the correlation between PLR's performance improvement and LPE is also intriguing.", "weaknesses": "- While the results are interesting at a glance, they do not seem to provide any insights that could be used to develop better curriculum methods.\n- The analysis, and I think conclusions, are limited to procgen, and possibly similar games. For instance, if we have a domain where all episodes have an equal length, then the different samples per context results would not hold.\n- The LPE/LSE seemed somewhat tacked on, can this metric be used in some way by a method/designer?\n- It it still somewhat unclear what the loss per sample is defined as, and an equation would help. Secondly, is it all the losses (e.g. policy and value) or just some?\n- There are many figures, for all of the games, and this makes getting an overview of the results somewhat challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value the detailed feedback from the reviewer-7pFG and address each point in detail below.\n\n---\n**W1. Insights on Curriculum Design & Limitation**\n\nThanks for the reviewer-7pFG's feedback! While at first glance the implications for curriculum development might not be immediately apparent, our study does provide several insights that can be leveraged to enhance curriculum methods:\n\n1. **Dynamic Adjustment of Context Sample Distribution:**\nAs detailed in Section 3.1, by analyzing the learning dynamics across different contexts, we observed variations in the 'Loss Per Sample' —- an indicator of how efficiently an agent assimilates signals from its environment. \nThese variations suggest that agents learn different amounts of information from different contexts at different stages of training.\nHowever, the distribution of context samples mostly focuses on the contexts generating lower loss values, which may not be the most informative for learning.\nThis insight supports the idea of dynamically adjusting the context sample distribution based on the loss signal distribution to make learning more efficient.\n1. **Influence of Context Gaps on Curriculum Learning:** As discussed in Section 4, the gaps between contexts significantly impact the effectiveness of the curriculum. Section 4.1 shows that missing certain contexts can hinder an agent's learning progress, while Section 4.2 demonstrates that adding new contexts can facilitate faster transfer learning to new situations.\nIt implies that a well-designed curriculum should consider the distribution of context samples and the gaps between contexts to optimize learning efficiency.\n\nThese insights suggest that tailored curriculum designs should not only focus on adjusting to the agent's learning needs but also optimize the distribution and progression of learning contexts to improve overall training effectiveness. \nThis is precisely the purpose for which our C-Procgen was proposed: to design a system that meets these needs.\n\n---\n**W2.**\n\n> Reviewer: The analysis, and I think conclusions, are limited to procgen, and possibly similar games. For instance, if we have a domain where all episodes have an equal length, then the different samples per context results would not hold.\n\nThanks to the reviewer-7pFG for his/her insightful comments. Indeed, our experiments are based exclusively on C-Procgen, as the development and analysis of this new platform we introduced constitute the primary objective of our work.\n\nAdditionally, it is true that in domains where all episodes have equal length, the distribution of samples per context would naturally be uniform. \nHowever, this does not conflict with our findings: the distribution of loss signals across different contexts does not align with the distribution of samples, which contributes to inefficiencies in curriculum learning.\nThis misalignment is particularly noteworthy because the distribution of loss signals is dynamic, not fixed or uniformly distributed.\n\n---\n**W3.**\n\n> Reviewer: The LPE/LSE seemed somewhat tacked on, can this metric be used in some way by a method/designer?\n\nAs a post hoc evaluation metric, LPE (Loss Production Efficiency) cannot be directly used by designers or during method development.\n\nHowever, LPE can be utilized to evaluate the efficiency of loss signal production of a whole learning process, providing insights into how effectively an agent learns from its environment.\n\nIn practice, rather than using LPE directly to adjust curriculum strategies, methods can leverage monitoring of the Loss per Sample to dynamically adjust the distribution of contexts within the curriculum.\n\n---\n**W4. Clarity of Definition & Q1**\n\nThanks for the reviewer-7pFG's valuable feedback! We will enhance the clarity in our revision regarding the definitions and calculations pertinent to our methodology.\n\nRegarding to score improvement, it is utilized as a metric to quantify the performance enhancement that the PLR (Prioritized Level Replay) method brings over the standard PPO. \nThe calculation of score improvement is defined as follows:\n$\\frac{score_{\\text{PLR}} - score_{\\text{PPO}}}{score_{\\text{PPO}}}$. \nHere, $score_{\\text{PLR}}$ and $score_{\\text{PPO}}$ represent the scores achieved by the PLR and PPO algorithms respectively, after training for 25M steps under the same configuration. \n\nAdditionally, the 'loss' referred to in the paper is the PPO Loss, because all our experiments are based on the PPO.\nThe PPO Loss is in a weighted sum of three terms:\n$l_{policy}$,\n$l_{value}$,\nand $l_{entropy}$, \nand it is this weighted sum that we record.\n\n---\n**W5.**\n\n> Reviewer: There are many figures, for all of the games, and this makes getting an overview of the results somewhat challenging.\n\nThanks to the reviewer-7pFG for his/her feedback regarding the number of figures in our submission. \nWe acknowledge that the abundance of figures for each game might complicate obtaining a clear overview of the results. \nTo address this, we will attempt to use broader metrics such as Loss Production Efficiency to summarize key phenomena more succinctly in our revision. \nGiven the significant diversity in game design and context spaces within Procgen, we initially included numerous figures to substantiate our points comprehensively. \n\n---\n**Q2.**\n\n> Reviewer: Line 212-216, should it be LPE? If not, what does LSE mean?\n\nWe acknowledge this and appreciate the reviewer-7pFG for pointing it out.\nThe LSE at line 212-216 should indeed be LPE, which means the Loss Production Efficiency introduced at [*line 201*].\n\n---\n**Q3 & Q4.**\n\nThanks to the reviewer-7pFG for catching these typos. We will correct them in our revision!\n\n---\n\n\n**Q5. Clarity of Figure5**\n\nWe appreciate the reviewer-7pFG's suggestion to improve the clarity of Figure 5. \nThe primary purpose of using detailed figures was to visually highlight which contexts within the environments were masked, a detail that may be crucial for nuanced analysis across different games.\n\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hmpzmuKC1i", "reviewer": "Reviewer_2ABJ", "summary": "This paper introduces C-Procgen, an extension to the popular Procgen benchmark that includes many improvements over the original, most notably by adding \"context parameters\" that can control specific features of the procedural content generation. They thoroughly study the performance of agents in different contexts and identify an implicit curriculum in which agents spend more time and experience higher loss values in progressively harder contexts as training progresses. They also define a new metric for measuring loss signal efficiency and identify a correlation between this metric and the performance of Prioritized Level Replay, a popular curriculum learning method. Finally, the paper investigates how agent performance is affected by masking out specific contexts from the training set.", "questions": "* On line 142, how does a continuous value `bomb_prob` produce only 6 unique contexts?\n* On line 149, if `gap_range` and `bomb_prob` can be individually configured, how are they correlated? \n* Does this work use policy loss, value loss, or both?\n* Is it still possible to seed a C-Procgen environment with the original procedural content generation? This may be important for comparison to prior work.\n* How was the performance improvement for PLR calculated or evaluated?\n* Have the authors investigated whether previous work has identified higher loss values in challenging states or environments?", "limitations": "The authors list several limitations in their paper, namely that they do not define new evaluation protocols for C-Procgen, and that they do not explore curriculum learning. Curriculum learning itself seems outside the scope of this paper, though it would make a nice addition. It does seem problematic that the work provides no training curves or evaluation protocols for future work. Without them, future work may decide on different protocols, making them harder to compare and limiting reproducibility.\n\n\n\nThe paper also does not discuss whether their results could generalize to other learning algorithms aside from PPO. It is common for work in this space to only use PPO, but that limitation should be stated in the paper. The findings in this paper are also largely Procgen specific and unlikely to generalize to other RL environments.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, the paper provides several substantial contributions to reinforcement learning and curriculum learning. Procgen is one of the most popular benchmarks for evaluating generalization in RL, and this work provides valuable new features and analyses for this benchmark. The context parameters introduced by C-Procgen would allow researchers to use Procgen to study more varied curriculum learning methods. The authors seem to have this specific use case in mind as they implemented several important features for curriculum learning, such as allowing environments to be efficiently configured during a reset, and allowing each environment in the vector environment to be individually configured. \n\nThe analysis of individual procgen environments also identifies which environments and which components of those environments are challenging for RL agents to learn. It can be unintuitive to identify what RL agents may struggle to learn, so these findings may be valuable for interpreting future results using Procgen. By sorting contexts according to difficulty, it is possible to create an explicit curriculum to train agents with, as the authors allude to. Many automatic curriculum methods aim to automatically sort tasks by difficulty, so this may serve as a strong baseline for evaluating those methods.\n\nOverall, the methodology, presentation of results, and analyses are all sound. The figures are clear and easy to interpret. The claims in the paper seem to match to experimental results presented in the figures, and the authors do a good job highlighting interesting findings aside from their main claims.", "weaknesses": "This paper has several weaknesses, mostly regarding clarity and comparison to relevant literature. It would have been nice to see some comparison to other environments that are typically used in curriculum learning and generalization research, such as Minigrid [1] and Minihack [2], which provide similarly configurable gridworld environments. The authors do compare their work to CARL, but miss some popular benchmarks. A more glaring gap is that this work does not reference Unsupervised Environment Design [3], which introduces a underspecified POMDP formulation that extends POMDPs to have configurable free parameters, exactly like the context parameters used in C-Procgen.\n\n\n\nOverall the writing quality of the paper is relatively paper, particularly in the introduction. There are numerous grammatical errors, just to highlight a few:\n\n* Line 31: \"which provides explicit access and *control to* the context parameters\" should probably say *control over*.\n* Line 32:  \"The return and loss *concentrate in* easy tasks\" should say *concentrate on.*\n* Line 40: \"samples collected by the agent is from hard contexts\" should say *are from*.\n* Line 207: \"*Level Replay*\" should say *Prioritized Level Replay*.\n\nAside from these examples, the phrasing is somewhat awkward in many places. This is exacerbated by the fact that definitions of key terms do not appear until section 3 if at all. \"Context parameters\" are referenced multiple times before being defined on Line 102. The authors often reference the \"learning process\" in a vague way. For example, on Line 26 they make the claim that in previous work on curriculum learning \"The learning process itself in the face of diverse levels is less investigated.\" Clearly those works utilize and study learning processes, so it is unclear what comparison the authors are trying to make. It is possible that drawing on the terminology from unsupervised environment design may help to clarify what context and contextual parameters are.\n\n\n\nI also have a few concerns regarding soundness and novelty:\n\n* Using loss on its own as a metric is usually not well justified for policy gradient methods, but the authors show that their loss production efficiency metric is negatively correlated with score improvement from using PLR. However, the paper does not explain how score improvement in Figure 4 is calculated.\n* It's not clear from the paper whether they are using PPO's value loss, policy loss, or both.\n* The finding that agents collect more samples from harder contexts as training progresses seems obvious by definition. If we have a reward function that is correlated with survival, and episodes end upon death, then as the agent learns to maximize reward, it is learning to survive for longer and therefore collect more samples from challenging contexts. The opposite would imply that the agent is learning to minimize reward. This is corroborated by the figures in which the score for every context increases during training.\n* It is unsurprising that more challenging contexts results in higher loss values, and its likely that this has been observed in other environments in previous work.\n* Despite training agents for many diverse contexts, this work does not appear to include training curves or scores for those agents. Training curves can be a valuable reference for researchers looking to use new benchmarks. It also impossible to tell from the paper whether C-Procgen and Procgen agents perform similarly.\n\n\n\n[1] Chevalier-Boisvert, Maxime, et al. \"Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks.\" *Advances in Neural Information Processing Systems* 36 (2024).\n\n[2] Samvelyan, Mikayel, et al. \"Minihack the planet: A sandbox for open-ended reinforcement learning research.\" *arXiv preprint arXiv:2109.13202* (2021).\n\n[3] Dennis, Michael, et al. \"Emergent complexity and zero-shot transfer via unsupervised environment design.\" *Advances in neural information processing systems* 33 (2020): 13049-13061.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the feedback from the reviewer-2ABJ and will address each point in detail below.\n\n---\n**W1. Issues Regarding to Literature Comparison**\n\nWe appreciate the highlighting of Minigrid and Minihack as notable works similar to our C-Procgen, and regret the oversight of not mentioning them in our initial submission. Additionally, we acknowledge the omission of '*Unsupervised Environment Design*' in our submission, despite its relevance and prominence in the field. \n\n1. We agree that Minigrid and Minihack are widely used in curriculum learning due to their customizable environments. While Procgen offers a broader variety of challenges, it lacks the customizable features essential for detailed control, which we have incorporated into C-Procgen.\n\n2. After reviewing '*Unsupervised Environment Design*', we recognized its overlap with C-Procgen, especially in using configurable parameters for training environments. We will incorporate terms like 'underspecified environment' and 'free parameters' to clarify and deepen our discussion!\n\n---\n**W2. Typos**\n\nWe appreciate the detailed feedback on the typographical errors in our paper. We have thoroughly reviewed the manuscript and corrected these issues to provide a revision. Thanks to the reviewer for highlighting these points.\n\n---\n**W3. Lack of Clarity and Definitions**\n\nWe appreciate feedback from the reviewer on clarity. \nWe will define key terms like 'context parameters' earlier in the text and refine our descriptions of the learning process to address these concerns more directly. \nAnd, we would love to align our terminologies with those used in '*Unsupervised Environment Design*' in our revision.\n\n---\n**W4 & Q5. Lack of Metric Explanation**\n\nWe apologize for the oversight and will address this in our revision. \nThe score improvement which is defined as \n$\\frac{score_{\\text{PLR}} - score_{\\text{PPO}}}{score_{\\text{PPO}}}$. \nHere, $score_{\\text{PLR}}$ and $score_{\\text{PPO}}$ represent the scores achieved by the PLR and PPO algorithms respectively.\n\n---\n**W5 & Q3. Loss Specification**\n\nWe apologize for not specifying this clearly in the manuscript. The 'loss' referred to in the paper is the PPO Loss, which is composed in a weighted sum of three terms: \n$l_{policy}$,\n$l_{value}$,\nand $l_{entropy}$.\n\n---\n**W6. Obviousness of Training Progression Findings**\n\nAs the reviewer pointed out, it might seem unsurprising that agents collect more samples from harder contexts as they learn to prolong survival, especially when episodes end upon the agent's 'death.' \nHowever, this is not the only interpretation. \n- For example, in a maze scenario, an agent might accumulate many steps in challenging contexts not because it is effectively learning to navigate but because it cannot quickly exit the maze, potentially ending the episode with zero reward after expending many steps.\n- While initially this may not seem surprising, on reflection, the implication that an agent spends significant steps in unrewarding contexts is noteworthy.\n\n---\n**W7.**\n> Reviewer: It is unsurprising that more challenging contexts results in higher loss values.\n\nWe observed that the contexts resulting in higher loss values dynamically change throughout the learning process, rather than being consistently fixed on more challenging contexts.\nInterestingly, this phenomenon appears to be nearly universal across all environments tested.\n\n---\n**W8.**\n\nThanks for the suggestion! We have included learning curves with more detailed metrics information in the global response PDF attached. Upon comparison, we found that the performance of PPO on C-Procgen is largely consistent with its performance on Procgen.\n\n---\n**Q1:** \n\n> Reviwer: On line 142, how does a continuous value bomb_prob produce only 6 unique contexts?\n\nAlthough `bomb_prob` can be a continuous value, in the game logic, it is restricted to three discrete values: `0`, `0.25`, and `0.5`. The six unique contexts are determined by these discrete values and the number of sections (num_section). Specifically, the contexts setting of `num_section` and `bomb_prob` are as follows: [1, 0], [2, 0.25], [3, 0.25], [3, 0.5], [4, 0.5], [5, 0.5],\n\n---\n**Q2:** \n\n> Reviwer: On line 149, if gap_range and bomb_prob can be individually configured, how are they correlated?\n\nIn the game logic, as `gap_range` increases, `bomb_prob` also increases. They are correlated in the following three combinations: when `gap_range` is [0], `bomb_prob` is 0; when `gap_range` is [0, 1], `bomb_prob` is 0.25; and when `gap_range` is [0, 1, 2], `bomb_prob` is 0.5.\n\n---\n**Q4:** \n\n> Reviwer: Is it still possible to seed a C-Procgen environment with the original procedural content generation?\n\nYes, it is indeed possible to replicate the original Procgen environments in our C-Procgen framework. \n\n---\n**Q6:** \n\n> Reviwer: Have the authors investigated whether previous work has identified higher loss values in challenging states or environments?\n\nWe are aware of methods such as PLR and PER (Prioritized Experience Replay) where higher loss samples are leveraged to accelerate training. \nHowever, we may not have encountered specific studies explicitly identifying higher loss values in challenging states or environments. \nWe would appreciate any references the reviewer could provide to enrich our understanding and literature review.\n\n---\n**Limitation**\n\nThanks to reviewer for highlighting this limitation regarding the generalizability of our findings beyond PPO and their applicability to environments other than Procgen.We agree that exploring the effectiveness of our observations with value-based algorithms and in other environments would be beneficial. \nHowever, as our paper focuses on observations and discoveries within the C-Procgen environment, this is our primary area of emphasis.\n\n---\nOnce again, we thank the reviewer-2ABJ for his/her detailed feedback. We will address the mentioned limitations, clarities, and omissions in our citations thoroughly in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like express our sincere gratitude to all reviewers fortheir constructive comments!
We are particularly thankful for the following positive feedback:

- The proposed benchmark is a very worthwhile contribution. `Reviewer h4d8`, `Reviewer 7pFG`, `Reviewer 2ABJ`;
- The figures in the paper are easy-to-understand. `Reviewer h4d8`, `Reviewer 2ABJ`;
- The observations are very interesting. `Reviewer 7pFG`;
- The results are valuable for curriculum learning research. `Reviewer h4d8`, `Reviewer 2ABJ`;
- The methodology, presentation of results, and analyses are all sound. `Reviewer 2ABJ`;

We wil address the specific questions raised by the reviewers in the subsequent sections of the rebuttal.

Additionally, a new illustration of learning dynamics with learning curves and more detailed metrics information can be found in the PDF file.

Meta-review: This is in interesting analysis paper, which investigates the implicit curriculum in procgen that results simply from the agent's learning. The authors find that just due to the learning dynamics there is a shift for easy contexts to harder context, i.e. an implicit curriculum. 

The reviewers are in agreement that this analysis is useful and that even the new environment (c-procgen) is a useful contribution to the community. 

I therefor advocate for acceptance as a spotlight at the conference.
Decision: Accept (spotlight)