Implicit Curriculum in Procgen Made Explicit

Authors: zhenxiong  tan
kaixin  wang
xinchao  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 09:09:23

Abstract: Procedurally generated environments such as Procgen Benchmark provide a testbed for evaluating the agent's ability to robustly learn a relevant skill, by situating the agent in ever-changing levels. The diverse levels associated with varying contexts are naturally connected to curriculum learning. Existing works mainly focus on arranging the levels to explicitly form a curriculum. In this work, we take a close look at the learning process itself under the multi-level training in Procgen. Interestingly, the learning process exhibits a gradual shift from easy contexts to hard contexts, suggesting an implicit curriculum in multi-level training. Our analysis is made possible through C-Procgen, a benchmark we build upon Procgen that enables explicit control of the contexts. We believe our findings will foster a deeper understanding of learning in diverse contexts, and our benchmark will benefit future research in curriculum reinforcement learning.

Main paper: 
 introduction: 
 Training deep reinforcement learning agents in singleton environments is often susceptible to overfitting (Zhang et al., 2018a,b; Song et al., 2019; Cobbe et al., 2019 Cobbe et al., , 2020)) . Recently, procedurally generated environments, which take into account the agent's generalization ability, have received increasing attention (Justesen et al., 2018; Risi and Togelius, 2020; Cobbe et al., 2020) . In those environments, the agent faces an entirely new level in each episode (e.g., different maze layouts). Succeeding in such ever-changing levels demands that the agent learn robust policies. Compared to singleton environments, the diverse levels in procedurally generated environments pose new challenges for reinforcement learning training, calling for a better understanding of how learning progresses in this multi-level setting.In procedurally generated environments, each level is associated with some environment parameters (Dennis et al., 2020) that vary across levels, e.g., the agent's max speed, the size of the maze, or the number of obstacles. We refer to these various combinations of environment parameters as the contexts of the environment. The varying contexts give rise to levels of varying difficulties, suggesting a possible connection to curriculum learning (Bengio et al., 2009) . While some previous works explore this connection, they mainly focus on arranging the order of levels to form a curriculum (Jiang et al., 2021b) , or adaptively generating new levels to create a curriculum (Wang et al., 2019 (Wang et al., , 2020;; Dennis et al., 2020; Jiang et al., 2021a; Parker-Holder et al., 2022) . The learning process itself in the face of diverse levels is less investigated.To bridge this gap, we take a closer look at how reinforcement learning progresses under multi-level training in Procgen. In particular, we do not consider any form of level prioritization, i.e., the levels are sampled uniformly as in the standard setting of Procgen. To facilitate our study, we build a new benchmark, C-Procgen, which provides explicit access and control over the environment parameters. We then investigate how some metrics (e.g., the agent's reward and the loss) under different contexts evolve as the learning progresses. In addition, we inspect how learning will be affected when some contexts are masked out or additional contexts are included.We find that the learning process exhibits a gradual shift from easy contexts to hard contexts, despite the absence of explicit curriculum (e.g., level prioritization). The return and the loss concentrate on easy tasks in the early stage of training, and gradually shift towards hard tasks as the training goes on. In addition, we observe a mismatch between the loss and the number of samples across different contexts in the middle stage of training. Specifically, a large portion of samples collected by the agent are from hard contexts but the loss is concentrated in easy tasks. This implies that a considerable number of samples might be wasted and contribute little to updating the policy. Moreover, we find that this mismatch has a potential connection to how much performance gain can be achieved from modifying the sample distribution by prioritizing levels (Jiang et al., 2021b) .When the training contexts are partially masked out (easy, medium, or hard), we observe mixed results: in some games, the excluded contexts are essential for the agent to form an implicit curriculum, and masking them out will impact the performance; in other games, the remaining contexts are still able to provide useful information for the agent to learn transferrable policies. Besides, we find that expanding the set of training contexts helps expedite the learning shift from easy to hard contexts.Our C-Procgen benchmark exposes the environment parameters in the black-box context generation process in Procgen. It preserves the advantages of the Procgen benchmark, such as being comprehensive and challenging, while greatly expanding its use by adding explicit control of the contexts. C-Procgen can be useful in curriculum learning and contextual reinforcement learning.In summary, our paper makes the following contributions:• We build C-Procgen, a benchmark that enhances Procgen with accessible and controllable environment parameters, which can be of interest to curriculum learning and other areas.• Our work offers insights into the learning dynamics under the multi-context training in Procgen, and makes several interesting observations.• We reveal that an implicit curriculum occurs in the multi-context training in Procgen despite that the contexts are uniformly sampled.2 Related works 
procedurally generated environments: 
 Recently, procedural content generation has been adopted to design RL environments that consider the agent's ability to generalize as a central component of success. In procedurally generated environments, each episode begins with a new context created from a generation process, which is analogous to a "level" in video games. Across different episodes, the context is varied in certain task-relevant (e.g., number of obstacles) or task-irrelevant (e.g., background image) aspects. One popular procedurally generated environment suite is the Procgen benchmark (Cobbe et al., 2020) , which consists of 16 challenging Atari-like video games. However, in Procgen, multiple choices during the procedural generation process are governed by a single random seed. The underlying parameters that control the context generation are not accessible and visible to researchers. In other words, the context is implicitly determined by a black-box process. As pointed out by (Kirk et al., 2023) , the evaluation protocols supported by such environments are limited to varying the size of the context set. Our work empowers Procgen by making many factors of variation visible and controllable, allowing researchers to explicitly configure the environment parameters.Additionally, several works like Minigrid (Chevalier-Boisvert et al., 2018) and Minihack (Samvelyan et al., 2021) offer flexible frameworks for creating customizable reinforcement learning environments with adjustable parameters to generate different contexts. However, their environments are lightweight and lack convenient APIs for controlling environment parameters. Closely related to our work is the CARL benchmark (Benjamins et al., 2021) , which collects environments from several domains and makes their contexts configurable. While CARL mostly focuses on control tasks (including classic control environments from OpenAI Gym (Brockman et al., 2016) and physical simulations), our C-Procgen aims to extend the widely used Procgen benchmark.Curriculum learning (Bengio et al., 2009 ) is a general strategy in machine learning that organizes training examples or tasks in order to boost learning efficiency. The varying contexts in Procgen pose challenges at different levels of difficulty, which is naturally connected to curriculum learning. Jiang et al. (2021b) propose to prioritize the contexts that have higher estimated learning potential when sampling new contexts for the next training levels. This induces an emergent curriculum of increasingly difficult levels. Another line of work aims to automatically generate novel contexts that adapt to the agent's evolving ability (Wang et al., 2019 (Wang et al., , 2020;; Dennis et al., 2020; Jiang et al., 2021a; Parker-Holder et al., 2022) . While these works focus on manipulating the distribution of contexts to explicitly form a curriculum, our paper reveals that even when the contexts are uniformly sampled, the agent's learning process still implicitly follows a curriculum.The prior works mentioned above can also be viewed as a form of automatic curriculum learning (Portelas et al., 2020) . In this regard, our work is also closely related to the TeachMyAgent benchmark (Romac et al., 2021) . TeachMyAgent extends the Bipedal Walker environment from Ope-nAI Gym (Brockman et al., 2016) to parametric ones that enable controllable procedural generation, providing a testbed for benchmarking automatic curriculum learning algorithms. Similarly, our work aims to make the more challenging Procgen benchmark configurable.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper do discuss the limitations. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
c-procgen: controllable contextual procgen: 
 In this work, we first build a new benchmark that enhances the challenging Procgen suite aiming to investigate how learning progresses under the multi-context setting. Specifically, we carefully refactor the source code of Procgen and expose over 100 environment parameters that determine the context generation, giving users more control and insight into the previously opaque generation process. The resulting benchmark is C-Procgen4 , which augments the original Procgen with Controllable Contextual environment parameters and is well-suited for Curriculum reinforcement learning research. Throughout this paper, we use the term "environment parameter" (Dennis et al., 2020) to refer to any configurable factor that can vary across episodes (e.g., the agent's max speed, maze size, or number of obstacles). The term "context" refers to a specific configuration determined by the combination of all environment parameters.As illustrated in Figure 1 , the original Procgen only provides a level ID to implicitly control the generation of context. In contrast, our C-Procgen allows users to directly control the context via explicit configuration of the environment parameters. Thus, C-Procgen can serve as a highquality testbed for research in the area of Contextual Reinforcement Learning (Hallak et al., 2015) . Furthermore, C-Procgen enables users to integrate the context information into the learning algorithms, which can be useful in Automatic Curriculum Learning (Portelas et al., 2020) .In addition, C-Procgen also provides several engineering enhancements for improved usability:• Unlike the original Procgen, which assigns the same configuration to all environments during the initialization of vectorized environments, C-Procgen allows distinct contexts to be assigned to each environment. This flexibility helps expose the algorithm to more diverse game contexts during training.• Users can dynamically modify the context of each environment between episodes without creating a new instance. This feature is particularly beneficial for curriculum reinforcement learning, where environments need dynamically changing contexts throughout the learning process.• C-Procgen offers an interface that lets users track detailed context information during the learning process, enhancing transparency and facilitating better analysis.In summary, we would like to highlight that C-Procgen enhances Procgen with controllable environment parameters while the introduced overhead during simulation is negligible. In the following sections, we will take advantage of the features provided by C-Procgen to conduct a detailed analysis of how the learning dynamics evolve across contexts in Procgen.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) Justification: The answer is yes. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [No] Justification: The paper do no discuss these. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
learning dynamics in procgen: 
 In this section, we investigate the learning progress of Procgen under various contexts. For our experiments, we used C-Procgen, which faithfully simulates the same game logic and context distributions as the easy mode of the original Procgen benchmark. By leveraging the flexible context control features of C-Procgen, we recorded key metrics such as loss, entropy, episode length, average score, and the number of samples for each context. This approach provides a more detailed view of learning progress across different contexts.Specifically, we select nine environments from Procgen due to their episodic contexts, which change with each game reset, resulting in unique configurations for each playthrough. We utilize the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) in our reinforcement learning experiments. For each of these selected environments, we perform five individual runs, each encompassing 25 million steps, to ensure a comprehensive and robust analysis. Notably, the experiment with C-Procgen allows for the generation of diverse environments under all possible circumstances, following the sample efficiency experimental protocols of the original Procgen benchmark.To gain deeper insights into the learning process in C-Procgen, we first focus on the game Ninja. In the game Ninja, the agent navigates through a field of suspended platforms while avoiding bombs. This environment contains six unique contexts, each determined by different environment parameters including section_num, gap_range and bomb_prob. For instance, the number of platforms in the game, determined by the contextual factor section_num, significantly impacts the gameplay. Moreover, the environment parameter gap_range determines the distance between the two platforms. Smaller values correspond to smaller gaps, thus implying easier transitions for the agent. Furthermore, a higher bomb_prob escalates the likelihood of encountering bombs. For instance, Figure 2 presents heat maps that visualize some metrics of the learning process across different contexts within the game Ninja, where each grid indicates a unique context.5 Take, for example, the grid at coordinates (2, 1); it corresponds to a context where the game features two platforms, and the gap_range and bomb_prob are set to their respective minimum values. 2   3  1 2  4 5  3  1 2  4 5  3  1 2  4 5  3  1 2  4 5  3  1 2  4 Average Score. A notable phenomenon is observed in the Average Score for Ninja (depicted in blue heat maps, see Figure 2 ). The average score, which measures the agent's performance, varies significantly across different contexts. Notably, as early as stage T 2 , the agent demonstrated high scoring capability in the context (1, 0), while in other more challenging contexts such as (5, 2), high scores were only achieved at a later stage as T 5 . This led us to categorize the contexts into two types: "Easy contexts", where high scores were obtained early in training, and "Hard contexts", where high scores were achieved much later. This progression of mastering different contexts effectively forms a curriculum, subtly guiding the agent's learning process.Loss per Sample. Another important insight concerns the Loss per Sample of Ninja (depicted in orange color heat maps, see Figure 2 6 . This metric represents the average absolute value of the sample-wise loss7 generated at each algorithmic step within a specific context.More specifically, throughout the learning process, the focus of the loss per sample metric appears to shift progressively from simpler contexts to more challenging ones, potentially reflecting the order in which the agent learns to master different scenarios. Notably, this metric tends to concentrate on contexts that pose a moderate level of difficulty for the agent -those that are neither too easy nor too difficult. This phenomenon can be understood as follows: in contexts where the agent is either highly proficient or completely unfamiliar, it can make more accurate value judgments, resulting in fewer losses. However, in moderately challenging contexts, the agent encounters more variability in rewards, leading to errors in value estimation and, consequently, a higher number of loss signals to drive its updates. This observation aligns with the core principle of curriculum learning, which stresses the importance of guiding the agent through tasks that are neither too easy nor too difficult.Samples Distribution. An analysis of the Samples Distribution (depicted in green color heat maps, see Figure 2 reveals insights about the distribution of samples across different contexts within a single iteration. While the probability of sampling each context remains constant during environment resets and does not change over time, the length of the episode within each context is not constant, leading to variation in the samples across contexts. Aside from the earliest stage at T 1 , it is observed from the heat map that for the majority of the learning process within the game Ninja, samples tend to concentrate within the hard contexts. 8,9 0,1 2,3 4,5 6,7 8,9 0,1 2,3 4,5 6,7 8,9 0,1 2,3 4,5 6,7 8,9 0,1 2,3 4,5 6,7 The metric "Loss per sample" indicates whether the agent has learned new content in a specific context to a certain extent. As previously discussed, the agent's focus tends to gravitate toward contexts of moderate difficulty during learning. However, we observe an inconsistency: the contexts on which the agent focuses do not match the distribution of samples in the Procgen environments. This discrepancy suggests that a significant portion of the samples may not be providing sufficient loss signals to effectively guide the agent's learning. Particularly in the early stages of the process, easy contexts often promote the agent's initial progress, yet a majority of the samples the agent encounters come from hard contexts, which yield limited learning.To gain a deeper understanding of this inconsistency, we introduce a measure called "Loss Production Efficiency" that captures the effectiveness of sample distribution in generating loss signals at the i th iteration throughout the learning process in Procgen. The Loss Production Efficiency is defined as the ratio between the loss generated by the current sample distribution across different contexts and the maximum achievable loss: A notable observation reveals a potential correlation between the LPE and the performance improvement offered by the Prioritized Level Replay algorithm (Jiang et al., 2021b) on Procgen games. The performance improvement is defined as ∆score = scorePLR-scorePPO scorePPO , where score PLR and score PPO are the final scores with and without PLR, respectively. The Prioritized Level Replay algorithm aims to enhance the learning process by augmenting data from samples generated by high-loss levels. Interestingly, as illustrated in Figure 4 , it appears that environments characterized by lower LPE tend to exhibit more substantial performance gains when the Prioritized Level Replay algorithm is employed. This could be attributed to the algorithm's ability to modify the sample distribution and increase LPE.EQUATIONBased on the insights gained from the above findings, it becomes apparent that LPE plays a crucial role in the learning speed within an environment. When designing environments, optimizing factors such as the distribution of contexts, maximum episode length, and conditions for terminating episodes can significantly impact LPE. By carefully manipulating these factors, it is possible to create environments that promote higher LPE, resulting in accelerated learning processes and improved agent performance. 
investigating learning dynamics under context reconfiguration: 
 To investigate the influence of different training contexts on the reinforcement learning process, we reconfigure the context setting of games using C-Procgen. Firstly, based on previous findings of loss per sample across different contexts, we grouped contexts into three categories. We then created game settings excluding these context groups, to test their necessity to learning progress. Secondly, we add new contexts into the games, aiming to decrease discontinuity between existing contexts. We employed the Proximal Policy Optimization (PPO) algorithm, conducting 5 runs for each setting, consistent with the procedures in Section 4.The experiments demonstrate the distinct roles of contexts in different games. As depicted in Figure 5 , we present the final average score and loss per sample heat maps for nine games when trained without certain contexts.Overall, certain contexts appear to be crucial for the learning process in the game. For instance, in setting 2 of Coinrun, removing key contexts causes the learning process to stall, preventing the agent from progressing through the game's various stages. Similar patterns are observed in setting 1 of Heist and setting 2 of Ninja. Moreover, a generalization phenomenon was observed across multiple environments. In many cases, the agent was able to perform well even without training in certain contexts. Interestingly, the overall performance of the agent improved in some cases when certain contexts were excluded, such as in setting 3 of Climber, Setting 3 of Coinrun, and both Setting 2 and Setting 3 of Heist. One possible explanation for this improvement is that the excluded contexts were particularly difficult. By removing these challenging contexts, the agent may have experienced increased learning progress efficiency (LPE), which in turn enhanced its overall performance.In summary, our observations indicate that certain contexts play a crucial transitional role in the learning process. However, the benefit of having a larger number or wider variety of contexts is not always straightforward. In this section, we focus on the game Leaper as a case study. We aim to provide a detailed exploration of how the introduction of new contexts in the original Procgen benchmark, can influence the learning progress, particularly by analyzing the changing distribution of loss during the learning process.The game environment of Leaper requires the agent to avoid road vehicles and make use of the logs on the river to reach the top of the map. Within this environment, there are two environment parameters: the number of road lanes (road_lanes_num) and the number of water lanes (water_lanes_num). Under the original context setting of Leaper, only four contexts exist: (0, 0), (1, 1), (2, 2), and (3, 3). This indicates that the game will have an equal number of water lanes and road lanes, with each context having an equal chance of being sampled at game initialization. In addition to the original context setting of Leaper, we have introduced two new context settings. In Setting 1, we have six new contexts to enhance the connectivity among the original context setting. In Setting 2, we permit both water_lanes_num and road_lanes_num to take any value within the set {0, 1, 2, 3}. In both Setting 1 and Setting 2, each context has an equal probability of being sampled.With the original context setting, as observed from the stacked area chart in Figure 6 , the loss during the early stages of learning is primarily focused on context 0, followed by context 1. The transition of loss focus to context 2, however, is relatively slow, and by the end of training, the loss has not yet concentrated on context 3. In contrast, in Setting 1, due to the existence of more diverse contexts and stronger connectivity among them, the proportion of loss generated by contexts 1 and 2 starts to shift towards context 3 as early as around the 2 million steps. Comparing the results of the original setting and Setting 1, it is suggested that the expanded contexts serve as stepping stones, aiding in the agent's learning and adaptation across different contexts. When comparing Settings 1 and 2, Setting 2 also accelerates learning, but its effects appear less significant than those in Setting 1. This observation is consistent with the trends noted in Section 5.2.Moreover, upon observing the score curves and loss proportion of these three settings (see Figure 6 ), it is notable that the increase in the agent's loss proportion corresponds to the time when the agent's average score in that context begins to rapidly grow. Conversely, when the agent's loss proportion begins to decline in a context, it suggests that the agent has already become proficient in that context. Consistent with the findings in Section 4, when the loss begins to shift, the agent has indeed already mastered the respective context. 
conclusions: 
 Procgen is a widely used procedurally generated environment suite that has varying contexts across episodes. While previous works pay most attention to explicitly creating a curriculum of contexts, we seek to investigate the learning process itself under the multi-level training in Procgen. To this end, we build C-Procgen, which enhances Procgen with explicit access and control of context parameters. With C-Procgen, we analyze the learning dynamics in Procgen across contexts and reveal that an implicit curriculum happens as the learning proceeds. Our C-Procgen can also find its use in other curriculum learning research. In the future, we will explore if this implicit curriculum connects to the number of contexts and how we can take advantage of it to boost performance. 
limitation: 
 We did not provide specific evaluation protocols for C-Procgen. Even though Procgen offers a comprehensive set of games, we believe there are many other types of games that it doesn't cover. While we conducted several experiments, notably those involving modified context distributions, we did not explore dynamic context distributions -where the context distribution evolves with the learning progress. We believe that studies in this direction hold significant value for curriculum learning. We plan to study this deeper in our future work. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No data or model realsed in this paper. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The answer is yes. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: We use the PyTorch codebase from Raileanu and Fergus (2021) to train PPO Schulman et al. (2017) agents on Procgen games. The agent is parameterized by the IMPALA network architecture Espeholt et al. (2018) . Unless otherwise stated, we follow the hyperparameters used in Cobbe et al. (2020) for the easy mode of Procgen, as summarized in Table 1 .Similar to the original Procgen, our C-Procgen still enjoys high simulating speed. On a server with 2 Intel Xeon CPU cores and 56GB RAM, the FPS of Procgen is around 750 and the FPS of C-Procgen is around 710. Regarding the training cost, training a PPO agent on an NVIDIA T4 GPU for 25M steps takes approximately 2.5 3 hours. Determines the health or durability of each round (boss starship). Higher values indicate a more resilient boss that requires more damage to be defeated.[1, 2, 3]Specifies the number of rounds in the game. Each round represents an encounter with the boss starship.[1, 2, 3, 4, 5] climberSpecifies the number of platforms in the game. Each platform represents a step for the player to climb. Affects the generation of pits in the game. When the value is smaller, there is a higher probability of pits appearing in the game. Conversely, a larger value reduces the likelihood of pits. This environment parameter interacts with several other environment parameters, which are further detailed in the code documentation under the "context details" section.[0, 1, 2] dodgeballSpecifies the number of enemies present in the game. The player must dodge the balls thrown by these enemies and eliminate them to progress.[1, 2, 3, 4] fruitbotSpecifies the number of fruits that the player can collect along the way. Collecting a piece of fruit rewards the player with a positive reward.[0, 1, • • • , 9]Specifies the number of non-fruit objects that the player must avoid. Mistakenly collecting a non-fruit object results in a larger negative reward.[0, 1, • • • , 9]heistSpecifies the total number of keys required to unlock the locks and successfully complete the heist. The player must collect these keys scattered throughout the game.[1, 2, 3]Determines the size of the maze layout in which the heist takes place. The maze is generated using Kruskal's algorithm and serves as the environment for the player to navigate and find the hidden gem.[ 3, 5, 7, 9] leaperSpecifies the number of lanes in which cars move. The player must cross these lanes to reach the finish line and earn a reward.[0, 1, 2, 3]Specifies the number of lanes with logs on a river. Specifies the number of narrow ledges that the player, a ninja, must jump across.[0, 1, 2, 3, 4]Controls the range of gaps between the ledges that the ninja must traverse.[0, 1, 2]Table 2 : Descriptions of the contexts used in our experiments.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The claims outlined in the abstract and introduction are well-aligned with the detailed discussions and findings presented throughout the paper. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.The source code of C-Procgen can be found on GitHub: https://github.com/zxtan98/CProcgenDue to their direct correlation, the environment parameters gap_range and bomb_prob are combined on the same axis for simplification. Specifically, as gap_range increases, bomb_prob also increases. They are correlated in the following three combinations: when gap_range is [0], bomb_prob is 0; when gap_range is [0, 1], bomb_prob is 0.25; and when gap_range is [0, 1, 2], bomb_prob is 0.5.The value ranges in the heatmaps of Figure2and Figure3have been normalized. The range for the score is based on the maximum and minimum values achieved by the agent over the entire learning process, while the ranges for loss per sample and the number of samples are based on the maximum and minimum values at each specific time point.The loss from PPO(Schulman et al., 2017) is in a weighted sum of three terms: lpolicy,lvalue and lentropy. 


Decision: This is in interesting analysis paper, which investigates the implicit curriculum in procgen that results simply from the agent's learning. The authors find that just due to the learning dynamics there is a shift for easy contexts to harder context, i.e. an implicit curriculum. 

The reviewers are in agreement that this analysis is useful and that even the new environment (c-procgen) is a useful contribution to the community. 

I therefor advocate for acceptance as a spotlight at the conference.

Reviews: review: paper_summary: paper_summary: This paper presents a benchmark called C-Procgen that builds on the existing Procgen benchmark by allowing access and control of the context parameters. Furthermore, this work investigates how learning progresses for an RL agent in the absence of a curriculum given a uniform distribution over levels. The experiments demonstrate that throughout the training the agent gets better at certain contexts first, called easy contexts with high scores and losses early on, and later on it progresses in others, i.e., hard contexts, where low scores and losses occur initially. The authors consider such progress as an implicit curriculum. In addition, their evaluation of the Level Replay algorithm in C-Procgen indicates that performance gains occur in environments with lower loss production efficiency, an effectiveness measure they introduce. Finally, they investigate the effects of partially masking and expanding training contexts, and show that some contexts, especially the ones that present moderate difficulty, are more critical to the learning progress than the others.
strength_weakness: strength_weakness: - Most of the observations on learning dynamics shared in Section 3 may be because the studied environments can terminate before reaching the maximum number of steps in an episode. It would be more informative to consider the termination conditions, the reward functions, and the dynamics of these environments when evaluating how the implicit curriculum progresses.

- As most of the studied environments allow for early termination, I wonder if this causes the misinterpretation that there is an implicit curriculum. In essence, there is no curriculum being formed by the agent, but the agent progresses in certain contexts before the others, which is a valuable insight. Nevertheless, it may be misleading to say that the agent's focus shifts during training without a curriculum, as done by the authors in Section 3.2.

- Section 3.2 focuses on the Level Replay algorithm and its performance in C-Procgen environments, but this study does not investigate curriculum learning approaches such as [1,2].

[1] Klink, P., Yang, H., D’Eramo, C., Peters, J., & Pajarinen, J. (2022, June). Curriculum reinforcement learning via constrained optimal transport. In International Conference on Machine Learning (pp. 11341-11358). PMLR.

[2] Huang, P., Xu, M., Zhu, J., Shi, L., Fang, F., & Zhao, D. (2022). Curriculum reinforcement learning using optimal transport via gradual domain adaptation. Advances in Neural Information Processing Systems, 35, 10656-10670.
questions: questions: - Are the ranges of score, loss per sample, and number of samples consistent/fixed among training stages (Figures 2 and 3) and settings (Figure 5)?
limitations: limitations: -  Please, check the weaknesses section for the comment about the lack of investigation of curriculum learning approaches developed for contextual settings.
paper_summary: paper_summary: After rebuttal

I think C-Procgen is a useful contribution in itself. Secondly, the author's rebuttal has persuaded me that their analysis is novel, and can be useful. I especially like the analysis of LPE and how this relates to PLR' relative performance to PPO.

I am really impressed with the authors running my suggested LPE experiment on such short notice, and I think the preliminary results are promising.

----------------------




This paper firstly introduced C-Procgen, a version of procgen where each context variable can independently and controllably be changed, instead of the default behaviour of changing everything based on only a single seed.
Next, they analyse the distribution of contexts during training and find an implicit curriculum happens, in particular, the lengths of the episodes changes for different contexts as the agent learns. They also measure the loss per sample, and split that across contexts. They find a non-uniform split, and this also changes over training.
Finally, they investigate different procedures of changing the contexts, and the effect this has on learning.
strength_weakness: strength_weakness: - While the results are interesting at a glance, they do not seem to provide any insights that could be used to develop better curriculum methods.
- The analysis, and I think conclusions, are limited to procgen, and possibly similar games. For instance, if we have a domain where all episodes have an equal length, then the different samples per context results would not hold.
- The LPE/LSE seemed somewhat tacked on, can this metric be used in some way by a method/designer?
- It it still somewhat unclear what the loss per sample is defined as, and an equation would help. Secondly, is it all the losses (e.g. policy and value) or just some?
- There are many figures, for all of the games, and this makes getting an overview of the results somewhat challenging.
questions: questions: - what does score improvement mean in fig 4?
- Line 212-216, should it be LPE? If not, what does LSE mean?
- line 224, should that not be section 3?
- The algorithm introduced by [9] is called Prioritised Level Replay (or PLR) and not just Level Replay.
- Figure 5 has a lot of information and is somewhat hard to parse. I think it could be made clearer by having some summary plots (e.g. average test reward) as a function of the training contexts used.
limitations: limitations: - There are no suggestions or (in my opinion) useful takeaways from the analysis.
paper_summary: paper_summary: This paper introduces C-Procgen, an extension to the popular Procgen benchmark that includes many improvements over the original, most notably by adding "context parameters" that can control specific features of the procedural content generation. They thoroughly study the performance of agents in different contexts and identify an implicit curriculum in which agents spend more time and experience higher loss values in progressively harder contexts as training progresses. They also define a new metric for measuring loss signal efficiency and identify a correlation between this metric and the performance of Prioritized Level Replay, a popular curriculum learning method. Finally, the paper investigates how agent performance is affected by masking out specific contexts from the training set.
strength_weakness: strength_weakness: This paper has several weaknesses, mostly regarding clarity and comparison to relevant literature. It would have been nice to see some comparison to other environments that are typically used in curriculum learning and generalization research, such as Minigrid [1] and Minihack [2], which provide similarly configurable gridworld environments. The authors do compare their work to CARL, but miss some popular benchmarks. A more glaring gap is that this work does not reference Unsupervised Environment Design [3], which introduces a underspecified POMDP formulation that extends POMDPs to have configurable free parameters, exactly like the context parameters used in C-Procgen.



Overall the writing quality of the paper is relatively paper, particularly in the introduction. There are numerous grammatical errors, just to highlight a few:

* Line 31: "which provides explicit access and *control to* the context parameters" should probably say *control over*.
* Line 32:  "The return and loss *concentrate in* easy tasks" should say *concentrate on.*
* Line 40: "samples collected by the agent is from hard contexts" should say *are from*.
* Line 207: "*Level Replay*" should say *Prioritized Level Replay*.

Aside from these examples, the phrasing is somewhat awkward in many places. This is exacerbated by the fact that definitions of key terms do not appear until section 3 if at all. "Context parameters" are referenced multiple times before being defined on Line 102. The authors often reference the "learning process" in a vague way. For example, on Line 26 they make the claim that in previous work on curriculum learning "The learning process itself in the face of diverse levels is less investigated." Clearly those works utilize and study learning processes, so it is unclear what comparison the authors are trying to make. It is possible that drawing on the terminology from unsupervised environment design may help to clarify what context and contextual parameters are.



I also have a few concerns regarding soundness and novelty:

* Using loss on its own as a metric is usually not well justified for policy gradient methods, but the authors show that their loss production efficiency metric is negatively correlated with score improvement from using PLR. However, the paper does not explain how score improvement in Figure 4 is calculated.
* It's not clear from the paper whether they are using PPO's value loss, policy loss, or both.
* The finding that agents collect more samples from harder contexts as training progresses seems obvious by definition. If we have a reward function that is correlated with survival, and episodes end upon death, then as the agent learns to maximize reward, it is learning to survive for longer and therefore collect more samples from challenging contexts. The opposite would imply that the agent is learning to minimize reward. This is corroborated by the figures in which the score for every context increases during training.
* It is unsurprising that more challenging contexts results in higher loss values, and its likely that this has been observed in other environments in previous work.
* Despite training agents for many diverse contexts, this work does not appear to include training curves or scores for those agents. Training curves can be a valuable reference for researchers looking to use new benchmarks. It also impossible to tell from the paper whether C-Procgen and Procgen agents perform similarly.



[1] Chevalier-Boisvert, Maxime, et al. "Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks." *Advances in Neural Information Processing Systems* 36 (2024).

[2] Samvelyan, Mikayel, et al. "Minihack the planet: A sandbox for open-ended reinforcement learning research." *arXiv preprint arXiv:2109.13202* (2021).

[3] Dennis, Michael, et al. "Emergent complexity and zero-shot transfer via unsupervised environment design." *Advances in neural information processing systems* 33 (2020): 13049-13061.
questions: questions: * On line 142, how does a continuous value `bomb_prob` produce only 6 unique contexts?
* On line 149, if `gap_range` and `bomb_prob` can be individually configured, how are they correlated? 
* Does this work use policy loss, value loss, or both?
* Is it still possible to seed a C-Procgen environment with the original procedural content generation? This may be important for comparison to prior work.
* How was the performance improvement for PLR calculated or evaluated?
* Have the authors investigated whether previous work has identified higher loss values in challenging states or environments?
limitations: limitations: The authors list several limitations in their paper, namely that they do not define new evaluation protocols for C-Procgen, and that they do not explore curriculum learning. Curriculum learning itself seems outside the scope of this paper, though it would make a nice addition. It does seem problematic that the work provides no training curves or evaluation protocols for future work. Without them, future work may decide on different protocols, making them harder to compare and limiting reproducibility.



The paper also does not discuss whether their results could generalize to other learning algorithms aside from PPO. It is common for work in this space to only use PPO, but that limitation should be stated in the paper. The findings in this paper are also largely Procgen specific and unlikely to generalize to other RL environments.

Comments: title: Acknowledgment
Re: Rebuttal by Authors
Response to Reviewer 7pFG
Discussion
comment: Dear Reviewers and Area Chairs,

We sincerely thank the reviewers and area chairs for the time and effort they put into reviewing our submission. The reviewers' thoughtful feedback and suggestions have been especially valuable in improving our work.

We deeply appreciate the reviewers' insights and the guidance provided. Thank you once again to everyone for their support!
Dear Reviewer 7pFG,

Thank you so much for your kind words and for updating your score. We truly appreciate your encouragement and are motivated to continue refining our work.

Thank you again for your support!
I am really impressed with you running this on such short notice (sorry!) I believe the preliminary results are promising, and I am excited to see the results for more games in the updated paper.

I've updated my score again.
Dear Reviewer 7pFG2,

We woud like to share the preliminary results of the experiments you suggested. These initial findings suggest that higher LPE learning processes enhance the effectiveness of agent learning.

Due to time limit, we conducted the experiments only on the `Leaper` environment. We tested the standard PPO algorithm against a simple High-LPE prioritization strategy [1]. Each setting was run with three different seeds and the averaged results are reported.




|                    | Standard PPO | High-LPE Strategy |
| ------------------ | ------------ | ----------------- |
| LPE                | 42.67%       | 72.53%            |
| Avg Return         | 6.89         | 8.82 (+28.1%)     |



As shown in the table[1], the High-LPE strategy resulted in a 28.1% improvement in performance, suggesting that higher LPE may positively impact learning dynamics. 
We plan to expand these experiments across all environments and run additional trials to ensure the stability and reliability of our results. 
Additionally, we will explore more High-LPE strategies to identify the most effective methods for enhancing learning efficiency.

We will include more experimental results and analyses in the final revision to fully showcase our findings.

Thank you for your valuable suggestions and insights!

> [1] 
> Naive High-LPE strategy: In the High-LPE strategy, the agent interacts with two groups of environments: **reference environments** with a uniform context distribution and **interaction environments** where the contexts are dynamically adjusted based on the loss obtained from the reference environments. Only the loss from the interaction environments is used for backpropagation. To increase LPE, we adjust the context distribution based on the loss-per-sample metrics, which are recalculated every 5 iterations. Since the Leaper environment has a small contextual space (only 4 contexts), we did not apply the Top-N context strategy you suggested. Instead, we applied a soft strategy, sampling from the adjusted context distribution.
Dear Reviewer 2ABJ,

Thank you for providing further feedback! We will make sure all writing and clarity issues will be addressed in the revised version by implementing the suggested changes. We will also include a detailed discussion on how our findings connect to existing results and commonly used methods inspired by this intuition (e.g., PLR, PER).

Once again, thank you for your constructive feedback. We are happy to hear you appreciate our contributions, and we will ensure the final version incorporates all the recommended revisions.
Dear Reviewer 7pFG,


Thank you for your time evaluating our work and for providing constructive and insightful suggestions.

We will ensure that all minor points are addressed, including those raised by the other reviewers. We promise these clarifying discussions will be fully incorporated into our revised manuscript.

Regarding Section 3.2, we will add more explanation to clarify our intuition behind the LPE metric.

The experiment design you suggested is an excellent proposal; it would provide an intuitive demonstration of the LPE metric's usefulness. We started the experiments based on your suggestion, and we will share the preliminary results before the discussion period closes. We will also include the results in the final version.

Thank you again for your valuable feedback and suggestions. Your insights have been incredibly helpful in refining our work.
Thank you for responding to my review, and I apologize for the delay in my response. I hope that the authors will address all of the writing and clarity issues by implementing the changes they have suggested here in future revisions to their paper. Aside from writing concerns, I'm not convinced by the arguments in response to W6, W7, and Q6. I suspect that these findings have been recorded in literature studying loss scales in RL because it is a commonly used intuition for methods such as PLR and PER as the authors points out. I do appreciate the contribution of recording both of these observations in Procgen, and the perspective that it forms a curriculum. However, I accounted for that contribution and assumed writing issues would be addressed when giving my original rating, so I will keep my score as is.
Ok, thanks again for your discussion and answers.

I will update my score, provided that:
- All of the minor points be resolved (and that from other reviewers, e.g. 2ABJ)
- And these clarifying discussions be included in the revised manuscript.

I've come to like figure 3 a lot, but I think it can be improved if you explain the intuition behind LPE more  in section 3.2. I do like seeing that when the LPE is high, a curriculum method like PLR doesn't do much better than PPO, since the context distribution already somewhat generates optimal losses. So please add some more explanation there.



Then, relatedly. I think it would be very nice if you could run an experiment like the following:
- At every iteration, compute the LPE but do not update the agent using these losses.
- Instead, train the agent only on the top N contexts (where N can e.g. be chosen such that the new LPE is >= 80% or something). Keep the number of timesteps the same, just train on fewer (but more effective) contexts, and then update the agent using these losses.
- Compare this against standard PPO.

I think this would help demonstrate the usefulness of LPE more, by seeing if there is a causal effect.



This last experiment is not required for me to increase my score, but it would be really nice if you could add it for the camera ready version.
Dear Reviewer 2ABJ,

We hope you are well. We tried to consider and address the points you raised in your review, including literature comparison, clarity, and the evaluation metrics. If there’s anything further you’d like us to clarify or discuss, we would greatly appreciate your feedback.

Thank you again for your time and thoughtful review!
We highly appreciate the effort and thoughtful feedback provided by reviewer h4d8. We would love to briefly address the concerns raised and provide further clarification:

---
**W1: Distribution Shift**

The term "distribution shift" refers to the changes in the distribution of samples across different contexts. 
When the probability of each game context being initialized is equal, longer episode lengths result in more training samples coming from certain contexts. 
As episode lengths change, this leads to a shift in the distribution of samples across the various contexts.

---
**W3. Evaluation metrics**

We agree with the reviewer-h4d8 that the applicability of C-Procgen is crucial. 
Our C-Procgen framework offers easy and unified parameters to set up game contexts. 
We acknowledge that unifying the presentation of learning dynamics across different games is challenging due to their varying context spaces. 
We will explore new approaches to better represent these learning dynamics in future work!

---

We greatly value the reviewer-h4d8’s thoughtful evaluation and the time spent reviewing our work.
Thanks once again for his/her/their insightful comments and for helping us improve our research.
Thank you for responding to my comments and questions. I'd like to apologize for the delay caused by travel.

** W1. Termination conditions: **

> This premature termination can shift the sample distribution across different contexts from what might be expected under uniform distribution conditions.

Although I agree with the rest of the first section of the response, I'm afraid I have to disagree with this statement. As far as I understand, there is no distribution shift during training if we talk about distributions over the task space.

** W2. Interpretation of Implicit Curriculum:** 

Thank you for the clarification. Please reflect on this in the manuscript as well. 

** W3. Evaluation metrics. **

> However, due to the significant variability across games in Procgen, replicating similar presentations to demonstrate the learning dynamics of agents under a curriculum poses challenges.

Such a perspective would hurt the applicability of C-Procgen, which would be very useful for research in curriculum learning for contextual RL. 

PLR-like approaches are developed primarily for Procgen environments, where contexts are not accessible, and each context/task/level is labeled with an ID. Although C-Procgen can bring a lot of insights into PLR-like method design, its fundamental impact would be on curriculum learning for contextual RL, which you talk about in the second subsection of the related work.

** Q1. Consistency of Metrics**:

Thank you for the clarification.

I appreciate the effort put into the rebuttal and the new figures in the shared PDF. I will stick with the original score I gave.
Dear Reviewer 7pFG,

We appreciate your thoughtful comments and the time you’ve taken to review our submission. We hope our response adequately addresses your concerns, particularly regarding the detailed contributions of our work. Please feel free to share any additional feedback or suggestions, and we would be happy to address any further comments. 

Thank you once again for your kind consideration!
We sincerely thank the reviewer-7pFG for his/her/their prompt response! We believe our work primarily makes contributions to the field in two aspects:

1. We have introduced C-Procgen, which incorporates functionalities that enrich the community’s research tools, enabling more dynamic and versatile experiments in reinforcement learning.
2. Leveraging C-Procgen, we have identified an implicit curriculum in learning dynamics — a concept previously unexplored — that could inspire future studies in curriculum learning.


---
**C-Procgen**

- Curriculum and Transfer Learning: C-Procgen aims to enhances curriculum learning by allowing for dynamic adjustments to the learning context, helping to gauge and improve agent adaptability in varied scenarios.
- Context-aware Learning: C-Procgen facilitates context-aware reinforcement learning, where agents adjust their strategies based on the current context.
- Diversity and Edge Cases: C-Procgen contributes to the diversity of training environments and supports the creation of edge cases. These are vital for assessing agent resilience in non-standard conditions.
- Environment Design: C-Procgen enables precise manipulation of game mechanics and environment attributes to study their influence on agent behavior and to experiment with new game dynamics.

---
**Implicit Curriculum**

With the help of C-Procgen, we have conducted extensive experiments to analyze learning dynamics within Procgen games. 
Our findings reveal the presence of an implicit curriculum, where despite a uniform distribution of game contexts, the learning progression of agents is notably non-uniform. 
Interestingly, agents tend to learn games in a specific order, favoring those that are neither too easy nor too challenging.
Furthermore, the dynamics we observed, including the loss signals, the distribution of samples, and the performance of agents across different contexts, all exhibit notable interrelationships.


This insight diverges from previous works[1][2] that primarily focus on explicitly crafting curricula to guide agents from simpler to more complex tasks. Instead, our observations suggest that the learning process itself naturally conforms to a curriculum structure — a phenomenon that has not been discussed previously. This novel perspective enhances our understanding of how agents adapt and learn, opening a new venue for future research.  

---
We appreciate the reviewer's constructive and insightful suggestions. We are happy to address any further questions and continue the discussion.


> [1] Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). *Prioritized experience replay*. arXiv preprint arXiv:1511.05952.
> 
> [2] Dennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., & Levine, S. (2020). *Emergent complexity and zero-shot transfer via unsupervised environment design*. Advances in neural information processing systems, 33, 13049-13061.
Thank you for your response.

I am still not quite convinced that this work provides clear contribution besides the C-Procgen benchmark. 
Could you please succinctly list the useful contributions of this project, and argue for how/why they matter?