Improving Environment Novelty Quantification for Effective Unsupervised Environment Design

Authors: jayden  teoh
wenjun  li
pradeep  varakantham
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:36:22

Abstract: Unsupervised Environment Design (UED) formalizes the problem of autocurricula through interactive training between a teacher agent and a student agent. The teacher generates new training environments with high learning potential, curating an adaptive curriculum that strengthens the student's ability to handle unseen scenarios. Existing UED methods mainly rely on *regret*, a metric that measures the difference between the agent's optimal and actual performance, to guide curriculum design. Regret-driven methods generate curricula that progressively increase environment complexity for the student but overlook environment *novelty* — a critical element for enhancing an agent's generalizability. Measuring environment novelty is especially challenging due to the underspecified nature of environment parameters in UED, and existing approaches face significant limitations. To address this, this paper introduces the *Coverage-based Evaluation of Novelty In Environment* (CENIE) framework. CENIE proposes a scalable, domain-agnostic, and curriculum-aware approach to quantifying environment novelty by leveraging the student's state-action space coverage from previous curriculum experiences. We then propose an implementation of CENIE that models this coverage and measures environment novelty using Gaussian Mixture Models. By integrating both regret and novelty as complementary objectives for curriculum design, CENIE facilitates effective exploration across the state-action space while progressively increasing curriculum complexity. Empirical evaluations demonstrate that augmenting existing regret-based UED algorithms with CENIE achieves state-of-the-art performance across multiple benchmarks, underscoring the effectiveness of novelty-driven autocurricula for robust generalization.

Main paper: 
 introduction: 
 Although recent advancements in Deep Reinforcement Learning (DRL) have led to many successes, e.g., super-human performance in games [26, 9] and reliable control in robotics [2, 3] , training generally-capable agents remains a significant challenge. DRL agents often fail to generalize well to environments only slightly different from those encountered during training [21, 63] . To address this problem, there has been a surge of interest in Unsupervised Environment Design (UED; [56, 23, 57, 28, 27, 37, 33, 7] ), which formulates the autocurricula [32] generation problem as a two-player zero-sum game between a teacher and a student agent. In UED, the teacher constantly adapts training environments (e.g., mazes with varying obstacles and car-racing games with different track designs) in the curriculum to improve the student's ability to generalize across all possible levels.To design effective autocurricula, researchers have proposed various metrics to capture learning potential, enabling teacher agents to create training levels that adapt to the student's capabilities.The most popular metric, regret, measures the student's maximum improvement possible in a level. While regret-based UED algorithms [23, 27, 28] are effective in producing levels at the frontier of the student's capability, they do not guarantee diversity in the student's experiences, limiting the training of generally-capable agents especially in large environment design spaces. Another line of work in UED recognizes this limitation, leading to methods exploring the prioritization of novel levels during curriculum generation [56, 57, 37, 33] . This strategic shift empowers the teacher to introduce novel levels into the curriculum such that the student agent can actively explore the environment space and enhance its generalization capabilities.To more effectively evaluate environment novelty, we introduce the Coverage-based Evaluation of Novelty In Environment (CENIE) framework. CENIE operates on the intuition that a novel environment should induce unfamiliar experiences, pushing the student agent into unexplored regions of the state space and introducing variability in its actions. Therefore, signals about an environment's novelty can be derived by modeling and comparing its state-action space coverage with those of environments already encountered in the curriculum. We refer to this method of estimating novelty based on the agent's past experiences as curriculum-aware. By evaluating novelty in relation to the experiences induced by other environments within the curriculum, CENIE prevents redundant environments-those that elicit similar experiences as existing ones-from being classified as novel. Curriculum-aware approaches ensure that levels in the student's curriculum collectively drive the agent toward novel experiences in a sample-efficient manner.Our contributions are threefold. First, we introduce CENIE, a scalable, domain-agnostic, and curriculum-aware framework for quantifying environment novelty via the agent's state-action space coverage. CENIE addresses shortcomings in existing methods for environment novelty quantification, as discussed further in Sections 3 and 4. Second, we present implementations for CENIE using Gaussian Mixture Models (GMM) and integrated its novelty objective with PLR ⊥ [28] and ACCEL [37] , the leading UED algorithms in zero-shot transfer performance. Finally, we conduct a comprehensive evaluation of the CENIE-augmented algorithms across three distinct benchmark domains. By incorporating CENIE into these leading UED algorithms, we empirically validate that CENIE's novelty-based objective not only exposes the student agent to a broader range of scenarios in the state-action space, but also contributes to achieving state-of-the-art zero-shot generalization performance. This paper underscores the importance of novelty and the effectiveness of the CENIE framework in enhancing UED. 
background: 
 We briefly review the background of Unsupervised Environment Design (UED) in this section. UED problems are modeled as an Underspecified Partially Observable Markov Decision Process (UPOMDP) defined by the tuple: ⟨S, A, O, I, T , R, γ, Θ⟩ where S, A and O are the sets of states, actions, and observations, respectively. Θ represents a set of free parameters where each θ ∈ Θ defines a specific instantiation of an environment (also known as a level). We use the terms "environments" and "levels" interchangeably throughout this paper. The level-conditional observation and transition functions are defined as I : S × Θ → O and T : S × A × Θ → ∆(S), respectively. The student agent, with policy π, receives a reward based on the reward function R : S × A → R with a discount factor γ ∈ [0, 1]. The student seeks to maximize its expected value for each θ denoted by V θ (π) = E π [ T t=0 R(s t , a t )γ t ]. The teacher's goal is to select levels forming the curriculum by maximizing a utility function U (π, θ), which depends on π.Different UED approaches vary primarily in the teacher's utility function. Domain Randomization (DR; [53] ) uniformly randomizes environment parameters, with a constant utility U U (π, θ) = C, making it agnostic to the student's policy. Minimax training [39] adversarially generates challenging levels, with utility U M (π, θ) = -V θ (π), to minimize the student's return. However, this approach incentivizes the teacher to make the levels completely unsolvable, limiting room for learning. Recent UED methods address this by using a teacher that maximizes regret, defined as the difference between the return of the optimal policy and the current policy. Regret-based utility is defined as U R (π, θ) = REGRET θ (π, π * ) = V θ (π * )-V θ (π) where π * is the optimal policy on θ. Regret-based objectives have been shown to promote the simplest levels that the student cannot solve optimally, and benefit from the theoretical guarantee of a minimax regret robust policy upon convergence in the two-player zero-sum game. However, since π * is generally unknown, regret must be approximated. Dennis et al. [23] , the pioneer UED work, introduced a principled level generation based on the regret objective and proposed the PAIRED algorithm, where regret is estimated by the difference between the returns attained by an antagonist agent and the protagonist (student) agent. Later on, Jiang et al. [27] introduced PLR ⊥ which combines DR with regret using Positive Value Loss (PVL), an approximation of regret based on Generalized Advantage Estimation (GAE; [48] ):EQUATIONwhere λ and T are the GAE discount factor and MDP horizon, respectively. δ θ k is the TD-error at time step k for θ. The state-of-the-art UED algorithm, ACCEL [37] , improves PLR ⊥ [27] by replacing its random level generation with an editor that mutates previously curated levels to gradually introduce complexity into the curriculum.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] .Justification: We included the limitations and future directions of our paper under Section C of the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
related work: 
 It is important to note that regret-based UED approaches provide a minimax regret guarantee at Nash Equilibrium; however, they provide no explicit guarantee of convergence to such equilibrium. Beukman et al. [10] demonstrated that the minimax regret objective does not necessarily align with learnability: an agent may encounter UPOMDPs with high regret on certain levels where it already performs optimally (given the partial observability constraints), while there exist other levels with lower regret where it could still improve. Consequently, selecting levels solely based on regret can lead to regret stagnation, where learning halts prematurely. This suggests that focusing exclusively on minimax regret may inhibit the exploration of levels where overall regret is non-maximal, but opportunities for acquiring transferable skills for generalization are significant. Thus, there is a compelling need for a complementary objective, such as novelty, to explicitly guide level selection towards enhancing zero-shot generalization performance and mitigating regret stagnation.The Paired Open-Ended Trailblazer (POET; [56] ) algorithm computes novelty based on environment encodings-a vector of parameters that define level configurations. POET maintains a record of the encodings from previously generated levels and computes the novelty of a new level by measuring the average distance between the k-nearest neighbors of its encoding. However, this method for computing novelty is domain-specific and relies on human expertise in designing environment encodings, posing challenges for scalability to complex domains. Moreover, due to UED's underspecified nature, where free parameters may yield a one-to-many mapping between parameters and environments instances, each inducing distinct agent behaviors, quantifying novelty based on parameters alone is futile.Enhanced POET (EPOET; [57] ) improves upon its predecessor by introducing a domain-agnostic approach to quantify a level's novelty. EPOET is grounded in the insight that novel levels offer new insights into how the behaviors of agents within them differ. EPOET evaluates both active and archived agents' performance in each environment, converting their performance rankings into rank-normalized vectors. The level's novelty is then computed by measuring the Euclidean distance between these vectors. Despite addressing POET's domain-specific limitations, EPOET encounters its own challenges. The computation of rank-normalized vectors only works for population-based approaches as it requires evaluating multiple trained student agents and incurs substantial computational costs. Furthermore, EPOET remains curriculum-agnostic, as its novelty metric relies on the ordering of raw returns within the agent population, failing to directly assess whether the environment elicits rarely observed states and actions in the existing curriculum.Diversity Induced Prioritized Level Replay (DIPLR; [33] ), calculates novelty using the Wasserstein distance between occupancy distributions of agent trajectories from different levels. DIPLR maintains a level buffer and determines a level's novelty as the minimum distance between the agent's trajectory on the candidate level and those in the buffer. While DIPLR incorporates the agent's experiences into its novelty calculation, it faces significant scalability and robustness issues. First, relying on the Wasserstein distance is notoriously costly. Additionally, DIPLR requires pairwise distance computations between all levels in the buffer, causing computational costs to grow exponentially with more levels. Finally, although DIPLR promotes diversity within the active buffer, it fails to evaluate whether state-action pairs in the current trajectory have already been adequately explored through past curriculum experiences, making it arguably still curriculum-agnostic. Further discussions on relevant literature can be found in Appendix B.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
approach: cenie: 
 The limitations of prior approaches to quantifying environment novelty underscore the need for a more robust framework, motivating the development of CENIE. CENIE quantifies environment novelty through state-action space coverage derived from the agent's accumulated experiences across previous environments in its curriculum. In single-environment online reinforcement learning, coverage within the training distribution is often linked to sample efficiency [59] , providing inspiration for the CENIE framework. Given UED's objective to enhance a student's generalizability across a vast and often unseen (during training) environment space, quantifying novelty in terms of state-action space coverage has several benefits. By framing novelty in this way, CENIE enables a sample-efficient exploration of the environment search space by prioritizing levels that drive the agent towards unfamiliar state-action combinations. This provides a principled basis for directing the environment design towards enhancing the generalizability of the student agent. Additionally, a distinctive benefit of this approach is that it is not confined to any particular UED or DRL algorithms since it solely involves modeling the agent's state-action space coverage. This flexibility allows us to implement CENIE atop any UED algorithm. CENIE's approach to novelty quantification through state-action coverage introduces three key attributes, effectively addressing the limitations of previous methods: (1) domain-agnostic, (2) curriculum-aware, and (3) scalable. CENIE is domain-agnostic, as it quantifies novelty solely based on the state-action pairs of the student, thus eliminating any dependency on the encoding of the environment. CENIE achieves curriculum-awareness by quantifying novelty using a model of the student's past aggregated experiences, i.e., state-action space coverage, ensuring that the selection of environments throughout the curriculum is sample-efficient with regards to expanding the student's state-action coverage. Lastly, CENIE demonstrates scalability by avoiding the computational burden associated with exhaustive pairwise comparisons or costly distance metrics. To evaluate the novelty of new environments using the agent's state-action pairs, the teacher needs to first model the student's past state-action space coverage distribution. We propose to use GMMs as they are particularly effective due to their robustness in representing high-dimensional continuous distributions [14, 5] . A GMM is a probabilistic clustering model that represents the underlying distribution of data points using a weighted combination of multivariate Gaussian components. Once the state-action distribution is modeled using a GMM, we can leverage it for density estimation. Specifically, the GMM allows us to evaluate the likelihood of state-action pairs induced by new environments, where lower likelihoods indicate experiences that are less represented in the student's current state-action space. This likelihood provides a quantitative measure of dissimilarity in stateaction space coverage, enabling a direct comparison of novelty between levels. It is important to note that CENIE defines a general framework for quantifying novelty through state-action space coverage; GMMs represent just one possible method for modeling this coverage. Future research may explore alternatives to model state-action space coverage within the CENIE framework (see Section C in the appendix for more discussions).Before detailing our approach, we first define the notations used in this section. Let l θ be a particular level conditioned by an environment parameter θ. We refer to l θ as the candidate level, for which we aim to determine its novelty. The agent's trajectory on l θ is denoted as τ θ , and can be decomposed into a set of sample points, represented as X θ = {x = (s, a) ∼ τ θ }. The set of past training levels is represented by L and Γ = {x = (s, a) ∼ τ L } is a buffer containing the state-action pairs collected from levels across L. We treat Γ as the ground truth of the agent's state-action space coverage, against which we evaluate the novelty of state-action pairs from the candidate level X θ .To fit a GMM on Γ, we must find a set of Gaussian mixture parameters, denoted as λ Γ = {(α 1 , µ 1 , Σ 1 ), ..., (α K , µ K , Σ K )}, that best represents the underlying distribution. Here, K denotes the predefined number of Gaussians in the mixture, where each Gaussian component is characterized by its weight (α k ), mean vector (µ k ), and covariance matrix (Σ k ), with k ∈ {1, ..., K}. We employ the kmeans++ algorithm [12, 4] for a fast and efficient initialization of λ Γ . The likelihood of observing Γ given the initial GMM parameters λ Γ is expressed as:EQUATIONwhere x j is a state-action pair sample from Γ. N (x j | µ k , Σ k ) represents the multi-dimensional Gaussian density function for the k-th component with mean vector µ k and covariance matrix Σ k .To optimise λ Γ , we use the Expectation Maximization (EM) algorithm [22, 43] because Eq. 2 is a non-linear function of λ Γ , making direct maximization infeasible. The EM algorithm iteratively refines the initialλ Γ to estimate a new λ ′ Γ such that P (X | λ ′ Γ ) > P (X | λ Γ ).This process is repeated iteratively until some convergence, i.e., ∥ λ ′ Γ -λ Γ ∥ < ϵ, where ϵ is a small threshold. Once the GMM is fitted, we can use λ Γ to perform density estimation and calculate the novelty of the candidate level l θ . Specifically, we consider the set of state-action pairs from the agent's trajectory, X θ , and compute their posterior likelihood under the GMM. This likelihood indicates how similar the new state-action pairs are to the learned distribution of past state-action coverage. Consequently, the novelty score of l θ is represented as follows:EQUATIONwhere x t is a sample state-action pair from X θ . As shown in Eq. 3, we take the negative mean log-likelihood across all samples in X θ to attribute higher novelty scores to levels with state-action pairs that are less likely to originate from the aggregated past experiences, Γ. This novelty metric promotes candidate levels that induce more novel experiences for the agent during training. More details on fitting GMMs are explained in Appendix D.1.Design considerations for the GMM First, we specifically designate the state-action coverage buffer, i.e., Γ, as a First-In-First-Out (FIFO) buffer with a fixed window length. By focusing on a fixed window rather than the entire history of state-action pairs, our novelty metric avoids bias toward experiences that are outdated and have not appeared in recent trajectories. This design choice helps reduce the effects of catastrophic forgetting prevalent in DRL. Next, it is known that by allowing the adaptation of the number of Gaussians in the mixture, i.e., K in Eq. 2, any smooth density distribution can be approximated arbitrarily close [24] . Therefore, to optimize the GMM's representation of the agent's state-action coverage distribution, we fit multiple GMMs with varying numbers of Gaussians within a predefined range at each time step and select the best one based on the silhouette score [45] , an approach inspired by Portelas et al. [40] . The silhouette score evaluates clustering quality by measuring both intra-cluster cohesion and inter-cluster separation. This approach enables CENIE to construct a pseudo-online GMM model that dynamically adjusts its complexity to accommodate the agent's changing state-action coverage distribution. Collect trajectories τ on l θ , with stop-gradient η ⊥Compute novelty score for l θ using λ Γ (Eq.3 and Eq.4)9:Compute regret score for l ′ θ (Eq.1 and Eq.4)10:Update B with l θ if P replay (l θ ) is greater than that of any levels in B (Eq.5)11:else 12:Sample a replay level l θ ∼ B according to P replay 13:Collect trajectories τ on l θ 14:Update π η with rewards R(τ )15:Update Γ with τ and resize to W 16:for k in range K min to K max do 17:Fit a GMM k with k components on Γ and compute its silhouette score 18:end for 19:Select GMM parameters with the highest silhouette score to replace λ Γ 20:Perform edits on l θ to produce l ′ θ 21:Collect trajectories τ on l ′ θ , with stop-gradient η ⊥Compute novelty score for l ′ θ using λ Γ (Eq.3 and Eq.4)23:Compute regret score for l ′ θ (Eq.1 and Eq.4)24:Update B with l ′ θ if P replay (l ′ θ ) is greater than that of any levels in B (Eq.5)25:end if 26: end while With a scalable method to quantify the novelty of levels, we demonstrate its versatility and effectiveness by deploying it on top of the leading UED algorithms, PLR ⊥ and ACCEL. For convenience, in subsequent sections, we will refer to this CENIE-augmented methodology of PLR ⊥ and ACCEL using GMMs as PLR-CENIE and ACCEL-CENIE, respectively. Both PLR ⊥ and ACCEL utilize a replay mechanism to train their students on the highest-regret levels curated within the level buffer. To integrate CENIE within these algorithms, we use normalized outputs of a prioritization function to convert the level scores (novelty and regret) into level replay probabilities (P S ):EQUATIONwhere h is a prioritization function (e.g. rank) with a tunable temperature β that defines the prioritization of levels with regards to any arbitrary score S. Following the implementations in PLR ⊥ and ACCEL, we employ h as the rank prioritization function, i.e., h(S i ) = 1/rank(S i ), where rank(S i ) is the rank of level score S i among all scores sorted in descending order. In ACCEL-CENIE and PLR-CENIE, we use both the novelty and regret scores to determine the level replay probability:EQUATIONwhere P N and P R are the novelty-prioritized probability and regret-prioritized probability respectively, and α allows us to adjust the weightage of each probability. The complete procedures for ACCEL-CENIE are provided in Algorithm 1, and for PLR-CENIE in the appendix (see Algorithm 2) . Key steps specific to CENIE using GMMs are highlighted in blue. 
experiments: 
 In this section, we benchmark PLR-CENIE and ACCEL-CENIE against their predecessors and a set of baseline algorithms: Domain Randomization (DR), Minimax, PAIRED, and DIPLR. The technical details of each algorithm are presented in Appendix E. We empirically demonstrated the effectiveness of CENIE on three distinct domains: Minigrid, BipedalWalker, and CarRacing. Minigrid is a partially observable navigation task under discrete control with sparse rewards, while BipedalWalker and CarRacing are partially observable continuous control tasks with dense rewards. Due to the complexity of mutating racing tracks, CarRacing is the only domain where ACCEL and ACCEL-CENIE are excluded. The experiment details are provided in Appendix D. Following standard UED practices, all agents were trained using Proximal Policy Optimization (PPO; [49] ) across the domains, and we present their zero-shot performance on held-out tasks. We also conducted ablation studies to assess the isolated effectiveness of CENIE's novelty metric (see Appendix A).For reliable comparison, we employ the standardized DRL evaluation metrics [1] , presenting the aggregate inter-quartile mean (IQM) and optimality gap plots after normalizing the performance with a min-max range of solved-rate/returns. Specifically, IQM focuses on the middle 50% of combined runs, discarding the bottom and top 25%, thereby providing a robust measure of overall performance.Optimality gap captures the amount by which the algorithm fails to meet a desirable target (e.g., 95% solved rate), beyond which further improvements are considered unimportant. Higher IQM and lower optimality gap scores are better. The hyperparameters for the algorithms in each experiment are specified in the appendix.First, we validated the CENIE-augmented methods in Minigrid [23, 20] , a popular benchmark due to its ease of interpretability and customizability. Given its sparse reward signals and partial observability, navigating Minigrid requires the agent to explore multiple possible paths before successfully solving the maze and receiving rewards for policy updates. Therefore, Minigrid is an ideal domain to validate the exploration capabilities of the CENIE-augmented algorithms. Following prior UED works, we train all student agents for 30k PPO updates (approximately 250 million steps) and evaluate their generalization on eight held-out environments (see Figure 2 ). Figure 2 demonstrates that ACCEL-CENIE outperforms ACCEL in all testing environments. Moreover, PLR-CENIE shows significantly better performance in seven test environments compared to PLR ⊥ . This underscores the ability of CENIE's novelty metric to complement the UED framework, particularly in improving generalization performance beyond the conventional learning potential metric, regret. Further empirical validation in Figure 3 (a) confirms ACCEL-CENIE's superiority over ACCEL in both IQM and optimality gap. PLR-CENIE also outperforms its predecessor, PLR ⊥ , by a significant margin. Notably, PLR-CENIE's performance is able to match ACCEL's, which is significant considering PLR-CENIE uses a random generator while ACCEL uses an editing mechanism to introduce gradual complexity to environments.Beyond the normal-size testing mazes, we consider a more challenging evaluation setting. We evaluate the fully-trained student policy of PLR ⊥ , PLR-CENIE, ACCEL, and ACCEL-CENIE on PerfectMazeLarge (shown in Figure 3 (b)), an out-of-distribution environment which has 51 × 51 tiles and a episode length of 5000 timesteps -a much larger scale than training levels. We evaluate the agents for 100 episodes (per seed), using the same checkpoints in Figure 2 . ACCEL-CENIE and ACCEL achieved comparable zero-shot transfer performance. Notably, PLR-CENIE achieved close to 50% solved rate, matching ACCEL's performance. This is a significant improvement from PLR ⊥ , which had less than a 10% solved rate. We also evaluated the CENIE-augmented algorithms in the BipedalWalker domain [56, 37] , which is a partially observable continuous domain with dense rewards. We train all the algorithms for 30k PPO updates (∼1B timesteps) and then evaluate their generalization performance on six distinct test environments: BipedalWalker (default), Hardcore, Stair, PitGap, Stump, and Roughness (visualized in Figure 6 (a)). To monitor the student's generalization performance evolution, we assess the student policy every 100 PPO updates across six testing environments during the training period.In Figure 4 , ACCEL-CENIE outperforms ACCEL in five testing environments, with both achieving parity in the Roughness challenge, establishing ACCEL-CENIE as the leading UED algorithm in BipedalWalker. Similarly, PLR-CENIE consistently outperforms PLR ⊥ across all testing instances, except for the Stump challenge, where both algorithms exhibit similar performance. We present the aggregate results after min-max normalization (with range=[0, 300] on all test environments) in Figure 6 (b). Both ACCEL-CENIE and PLR-CENIE exhibit better performance compared to their predecessors in the IQM and optimality gap metrics. Notably, ACCEL-CENIE outperforms all benchmarks by a substantial margin, achieving close to 55% of optimal performance. Next, we tracked the evolution of state-action space coverage throughout training to evaluate the impact of CENIE's novelty objective on the curriculum's exploration of the state-action space. During training, state-action pairs encountered by the agent were collected for both PLR ⊥ and ACCEL, along with their CENIE-augmented versions. To visualize the distribution of these high-dimensional state-action pairs, we applied t-distributed Stochastic Neighbor Embedding (t-SNE; [54] ) to project them into a 2-D space. The resulting evolution plot and detailed implementation steps are provided in Appendix A.2. Afterwards, we quantified state-action space coverage by discretizing the 2-D scatterplot into cells and calculating the percentage of total cells occupied by each algorithm. As shown in Table 1 , CENIE drives ACCEL-CENIE and PLR-CENIE to achieve significantly broader state-action coverage by the end of 30k PPO updates compared to their predecessors. This evidence supports that the inclusion of CENIE's novelty objective for level replay prioritization contributes to broader state-action space coverage. To understand ACCEL-CENIE's improvement over ACCEL, we analyzed the difficulty composition of replayed levels at various training intervals across five seeds, as shown in Figure 5 . Level difficulty is assessed based on environment parameters such as stump height and pit gap width, using metrics adapted from Wang et al. [56] (details in Appendix A.2). It is evident that ACCEL predominantly favors "Easy" to "Moderate" difficulty levels, whereas ACCEL-CENIE progressively incorporates "Challenging" levels into its replay selection throughout training.The disparity in level difficulty distribution between ACCEL and ACCEL-CENIE is a critical factor in understanding their observed performance differences. ACCEL's training curriculum tends to remain within a comfort zone, consistently selecting a limited subset of simpler levels where the agent experiences high regret. However, this can be problematic when considering the regret stagnation problem. Specifically, in the event where the easier levels exhibit irreducible regret, it can restrict the agent's exposure to more complex scenarios, thereby constraining its generalization potential.In contrast, ACCEL-CENIE's integration of a novelty objective actively selects challenging levels, pushing the agent beyond its comfort zone into unfamiliar, complex environments. This novelty-based regularization fosters the exploration of under-explored regions in the state-action space, even if regret levels are low, thereby enhancing the agent's generalization capabilities. Furthermore, with a mutation-based approach like ACCEL, this environment selection strategy may generate or mutate new levels with high learning potential, further enriching the training curriculum. Finally, we evaluated the effectiveness of CENIE by implementing it on PLR ⊥ within the CarRacing domain [16, 27] . In this domain, the teacher manipulates the curvature of racing tracks using Bézier curves defined by a sequence of 12 control points, while the student drives on the track under continuous control with dense rewards. We train the students in each algorithm for 2.75k PPO updates (∼5.5M steps), after which we test the zero-shot transfer performance of the different algorithms on 20 levels replicating real-world Formula One (F1) tracks introduced by Jiang et al. [27] . These tracks are guaranteed to be OOD as their configuration cannot be defined by Bézier curves with only 12 control points. The middle image in Figure 6b shows a track generated by domain randomization and the rightmost image shows a bird's-eye view of the F1-USA benchmark track. The aggregate performance after min-max normalization of all algorithms is summarized in Figure 7b . Note that the min-max range varies across F1 tracks due to different specifications on the maximum episode steps (see Table 5 in the appendix for more details). Once again, the CENIE-augmented algorithm, PLR-CENIE, achieves the best generalization performance in both IQM and optimality gap scores. Table 3 in the appendix shows the zero-shot transfer returns on all 20 F1 tracks. PLR-CENIE consistently outperforms or matches the best-performing baseline on all tracks. Figure 8 presents the total regret in the level replay buffer for both PLR ⊥ and PLR-CENIE throughout the training process. Interestingly, PLR-CENIE maintains comparable, or even slightly higher, levels of regret across the training distribution, despite not directly optimizing for it. This outcome suggests that CENIE's novelty objective synergizes with the discovery of high-regret levels, providing counterintuitive evidence that optimizing solely for regret is not the only, nor necessarily the most effective, strategy for identifying levels with high learning potential (as approximated by regret). Intuitively, value predictions are inherently less reliable in regions of lower coverage density-areas characterized by higher entropy or high uncertainty regarding optimal actions-since these regions are less frequently sampled for agent's learning. These high-entropy regions are prime candidates for high-regret outcomes, especially when using a bootstrapped regret estimate, as in Eq. 1, due to the value estimation error in such states. By pursuing novel environments based on coverage, CENIE indirectly enhances the discovery of high-regret states, highlighting that noveltydriven autocurricula can effectively complement regret-based methods in uncovering diverse and challenging training scenarios. 
conclusion: 
 In this paper, we introduced Coverage-based Evaluation of Novelty In Environment (CENIE), a scalable, domain-agnostic, and curriculum-aware framework for quantifying environment novelty in UED. We then proposed an implementation of CENIE that models this coverage and measures environment novelty using Gaussian Mixture Models. By incorporating CENIE with existing UED algorithms, we validated the framework's effectiveness in enhancing agent exploration capabilities and zero-shot transfer performance across three distinct benchmark domains. This promising approach marks a significant step towards unifying novelty-driven exploration and regret-driven exploitation for training generally capable RL agents. We encourage motivated readers to refer to the appendix for further studies and discussions on CENIE.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] .Justification: We provided full implementations and hyper-parameters in the code submission and appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] .Justification: Our results are obtained across multiple independent runs, and we plotted the results with variabce.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provided the hardware details (GPUs and CPUs) in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] .Justification: We've read the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] .Justification: The main application of our paper is to train generally capable RL agents. Societal impacts are limited and thus we omitted this discussion in our paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] .Justification: Our paper does not have such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] .Justification: We've credited and cited the references and codebases properly in the paper.The resources are all open-sourced.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] .Justification: We don't release any new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] .Justification: Our research does not involve human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] .Justification: Our paper does not involve crowdsourcing nor research with human subjects Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we present extended experiment details regarding the results presented in the main body of the paper. To isolate the individual effects of regret and novelty, we conduct an ablation study in which only novelty is used to prioritize levels in the level buffer. We denote the CENIE-augmented versions of the PLR ⊥ and ACCEL, which use only novelty for level prioritization (set α = 1 in Equation 5), as PLR-CENIE † and ACCEL-CENIE † , respectively.Interestingly, we observed that in several instances, the ablation models, PLR-CENIE † and ACCEL-CENIE † , demonstrated comparable or even superior zero-shot transfer performance compared to their regret metric counterparts, PLR ⊥ and ACCEL. This finding suggests that, in specific scenarios, prioritizing training levels based on novelty alone can effectively shape curricula. This is especially notable because our GMM-based novelty metric, unlike regret, does not rely on predefined domainspecific reward structures; rather, it is derived solely from the agent's trajectory data across different levels.However, it is important to note that these ablation results do not imply that regret should be entirely replaced by novelty-based level selection. Novelty alone may encounter limitations in extremely large state-action spaces where a balance with regret is essential for effective exploration. By combining novelty and regret in CENIE to shape the training curriculum, we significantly enhance the agent's generalization capabilities beyond those of previous algorithms, as shown in our main experiments. This finding highlights the powerful synergy between CENIE's novelty metric and traditional regret-based approaches, resulting in a more robust and effective training paradigm.After training all the student agents for 30k PPO updates (∼250M steps), we evaluate their transfer capabilities on eight held-out testing environments (see the first row in Figure 9 ). We summarize all the results in Figure 9 . In addition to the zero-shot transfer evaluation, we summarize the students' aggregate zero-shot transfer performance, i.e., IQM and Optimality Gap, in Figure 10 .In Figure 9 , PLR-CENIE † outperforms PLR ⊥ in most of the testing environments (6 out of 8), indicating that the novelty metric id more effective than the regret metric in the Minigrid domain for the PLR ⊥ algorithm. In contrast, ACCEL shows a marginal performance advantage over ACCEL-CENIE † in the testing environments, with two wins, four losses, and two ties. Importantly, for both cases -PLR-CENIE and ACCEL-CENIE -the combination of both regret and novelty yields the strongest performance, outperforming their individual metric counterparts. This finding supports the assertion that the CENIE framework effectively complements the regret metric, helping UED algorithms achieve better performance. We also provide a qualitative analysis of the effect of the novelty metric on the level replay buffer of PLR-CENIE in Minigrid for the experiments detailed under Section 5.1 in the main body. Specifically, we highlight levels that feature the lowest regret (bottom 10) yet exhibit the highest novelty (top 10); these are showcased in the first row of Figure 11 . Conversely, levels that score within the lowest 10 for both regret and novelty are displayed in the second row of the same figure. Visually, we observe that levels with high novelty and low regret present complex and diverse scenarios that challenge the student. In contrast, the levels displayed in the second row, characterized by low regret and low novelty, often resemble simple, empty mazes that offer limited learning opportunities. While it is not feasible to present every example level here, the contrast between the two groups is stark. Levels selected based on low regret but high novelty are significantly more varied and intricate than those chosen for their low novelty, despite both groups having low regret scores. This demonstrates that incorporating novelty alongside regret in the selection process enhances the ability to identify levels that present more interesting trajectories (experiences) for the student to learn from. Furthermore, we quantified the occupancy of the 2-D scatterplot by each method. To achieve this, we discretized the scatterplot into cells and computed the percentage of total cells occupied by data points generated by each method. Table 1 in the main paper presents the state-action space coverage percentages for each method. Notably, both PLR-CENIE and ACCEL-CENIE exhibit significantly broader coverage of the state-action space compared to their predecessors. This evidence supports the assertion that the outperformance of CENIE-augmented algorithms is associated with the broader coverage of the state-action space. Note that although the PLR-based algorithms exhibit higher state-action space coverage, they show poorer transfer performance compared to ACCEL-based algorithms. This discrepancy is likely because ACCEL initiates the curriculum with "easy" levels, and gradually introducing complexity via minor mutations, whereas PLR relies on DR, which lacks the fine-grained control over difficulty progression that ACCEL's mutation-based method offers. As a result, while CENIE enhances state-action space coverage for both ACCEL and PLR, it is likely that ACCEL's gradual complexity introduction mechanism capitalizes on this enhancement more effectively. To plot the level difficulty composition of the replayed levels by ACCEL and ACCEL-CENIE in Figure 5 of the main paper, we adapted the difficulty thresholds originally defined in Wang et al. [56] . This is because their thresholds were designed for a smaller 5-D encoding BipedalWalker environment, whereas our setting uses an 8-D encoding, which allows for higher complexity of levels to be generated. Specifically, we introduced an additional threshold for maximum stairs height, as shown in Table 2 . A level is classified as Easy if it meets none of the thresholds, and as Moderate, Challenging, Very Challenging, or Extremely Challenging if it meets one, two, three, or four thresholds, respectively.Note that our Figure 5 differs from Figure 12 in Parker-Holder et al. [37] which shows the difficulty distribution of the levels generated and added into the buffer, but not the actual levels selected by the teacher for the student to replay/train on. Also, their figure is defined for the 5D encoding setting. On that note, this also demonstrates that CENIE remedies an inefficiency in the original ACCEL algorithm, where mutation-based generation is capable of producing high complexity levels but are not selected for student training due to solely depending on regret for level prioritization. In addition to the state-action space coverage, we also conduct the ablation study in the BipedalWalker domain. We repeat the same experiment settings as in the Minigrid domain, where both ACCEL-CENIE † and PLR-CENIE † utilize only novelty to prioritize replay levels, and ACCEL-CENIE and PLR-CENIE integrate both novelty and regret for prioritization. We assess the algorithm performance with the same evaluations as in the main paper, providing both the transfer performance during training (Figure 13 ) and IQM and Optimality Gap (Figure 14 ).Summarizing the observations from both Figure 13 and Figure 14 , we observe that novelty-driven level replay selection exhibits a similar effect as regret on PLR ⊥ but is not as effective as regret on ACCEL. PLR-CENIE † performs on par with the regret metric counterpart (i.e., PLR ⊥ ) while ACCEL-CENIE † is outperformed by ACCEL and ACCEL-CENIE in this domain. The observations differ from the ablation studies conducted in the Minigrid domain. This discrepancy is possibly due to the greater importance of exploration in Minigrid, which features a sparse reward setting, compared to the dense reward, continuous control domain of BipedalWalker.To monitor the evolution of the students' transfer performance, we evaluate the students every 100 PPO updates on four racing tracks throughout the training period and plot the results in Figure 15 . PLR-CENIE outperforms both its predecessor, PLR ⊥ , and the state-of-the-art algorithm, DIPLR, in the CarRacing domain.Since both DIPLR and PLR-CENIE achieve near-optimal performance on the four testing tracks, we conduct a more extensive and rigorous evaluation by measuring the students' transfer performance on 20 human-designed F1 racing tracks from Jiang et al. [27] . We also include the ablation model, PLR-CENIE † which uses novelty alone to prioritize replay levels, in our evaluation. The detailed results of each algorithm are listed in Table 3 . For better visualization and straightforward comparison, we plotted the IQM and Optimality Gap performances in Figure 16 . ⊥ PLR- CENIE PLR- CENIE † Australia 304±133 107±97 224±173 715±50 574±69 745±32 616±45 Austria 299±118 152±106 159±160 587±49 458±44 566±38 496±46 Bahrain 208±136 44±101 118±159 514±48 377±75 537±58 453±38 Belgium 225±104 131±87 110±100 440±31 362±36 500±41 436±35 Brazil 192±106 57±61 147±124 451±39 368±42 485±27 312±40 China -35±57 -29±80 -71±63 93±102 -23±28 278±100 281±52 France 124±111 48±129 8±126 487±75 311±98 564±65 435±97 Germany 172±105 94±100 2±97 477±59 358±35 512±80 500±82 Hungary 319±155 133±113 139±161 686±50 597±72 678±40 604±70 Italy 267±114 204±89 198±135 676±30 559±63 708±26 588±34 Malaysia 142±107 39±94 51±104 404±30 265±44 469±79 338±22 Mexico 331±199 193±123 102±169 675±24 570±76 674±51 602±57 Monaco 80±78 100±94 34±111 369±122 139±112 641±46 476±96 Netherlands 143±109 104±95 42±77 540±34 400±61 558±59 403±84 Portugal 174±118 39±94 88±153 412±22 353±27 495±66 394±43 Russia 343±151 118±105 204±163 609±60 644±31 594±58 550±60 Singapore 209±108 75±93 88±153 479±78 423±51 530±48 454±55 Spain 296±133 181±110 249±157 619±39 517±41 588±43 499±39 UK 303±127 187±101 194±156 558±49 443±45 562±26 506±36 USA 173±95 -2±84 2±161 191±110 155±90 416±143 363±61 Mean 214±115 99±92 105±132 499±20 392±28 553±32 465±42From both Table 3 and Figure 16 , we observe that the ablation model, PLR-CENIE † , outperforms PLR ⊥ by a significant margin, indicating that novelty is more important for level replay prioritization than the regret metric in PLR ⊥ for the CarRacing domain. Moreover, PLR-CENIE surpasses DIPLR and achieves state-of-the-art transfer performance in the CarRacing domain by effectively combining the strength of both novelty and regret.Curiosity-driven Approaches in RL CENIE and curiosity-driven RL [47, 50] share a conceptual similarity in leveraging novelty or unfamiliarity to guide learning. However, they differ significantly in their application and theoretical foundations. Curiosity-driven learning seeks to quantify "curiosity" as an intrinsic reward for the agent such that it learns to prioritize the exploration of interesting experiences within a static environment [38] , or across a set of predefined tasks [18] . In contrast, CENIE is an autocurricula approach that focuses on curating environments interesting or useful for the agent's learning, shaping the learning curriculum itself rather than the exploration reward signal. This distinction is analogous to the difference between Prioritized Experience Replay [46] in traditional RL and Prioritized Level Replay [28] in UED. The former is an "inner-loop" method prioritizing past experiences for training, while the latter is an "outer-loop" method using past experiences to inform the collection/generation of future experiences. Similarly, curiosity-driven learning prioritizes novel experiences for policy updates, whereas CENIE focuses on generating and curating levels that induce these novel experiences. This fundamental difference in purposes makes theoretical and empirical comparisons between curiosity-driven approaches and CENIE less direct. However, many of the previous works in curiosity-driven RL provide inspiration for the CENIE framework. Specifically, curiosity-driven RL methods often seek to represent the "visitation counts" of state-action to shape the intrinsic reward. The use of GMMs to model state-action space coverage in CENIE is motivated by successes in curiosity-driven RL approaches which have tackled the counting problem using density models. Notably, density models have been flexibly used to model state-action visitations for both large discrete state-action spaces (via pseudo-counts [8, 36] ) and continuous state-action spaces [62, 61] .Automatic Curriculum Learning UED is related to Automated Curriculum Learning (ACL; [42] ), which emcompasses a family of mechanisms that automatically adapt the distribution of training data by selecting learning situations tailored to the capabilities of DRL agents. Many ACL methods prioritize sampling of environment instances where the agent achieves high learning progress (LP).A particular relevant method in this space is ALP-GMM, introduced by Portelas et al. [40] . ALP-GMM operates by periodically fitting a Gaussian Mixture Model to a dataset of previously sampled environment parameters, each associated with an Absolute Learning Progress (ALP) score. The approach employs an EXP4 [6] bandit algorithm to select Gaussians as arms, with each Gaussian's utility defined by its ALP score. ALP-GMM's approach to fitting multiple GMMs using different number of Gaussian components and keeping the best one inspired our GMM approach. However, they evaluate the GMM's quality using the Akaike's Information Criterion [15] (AIC). AIC introduces a penalty for the number of parameters in the model (which increases with the number of Gaussian components and dimensions of the data). This penalizes GMMs with more components, which may not be ideal for accurately modeling well-separated clusters in the state-action space which is crucial for identifying sparse regions and estimating novelty. To address this, our work uses the silhouette score [45] instead, which better evaluates clustering quality by considering both intra-cluster cohesion and inter-cluster separation, making it better suited for modeling novelty in state-action spaces. Additionally, ALP-GMM uses GMMs to sample environment parameters that are likely to yield high ALP scores, aligning task difficulty with the agent's progress. This approach contrasts with how GMMs are used in the CENIE framework, where the goal is to model the novelty of an environment based on state-action coverage, independent of specific environment parameters. ACL methods like ALP-GMM generally assume a predefined target task distribution. This differs from the UED framework which only requires only an underspecified task space, i.e. θ in the UPOMDP formalism. UED seeks to directly maximize the student's robustness over any possible environments, even those that are out-of-distribution from training. Similarly, CENIE provides a general approach for quantifying novelty using state-action coverage, without relying on any predefined task distribution. Due to this generality, we believe the CENIE framework holds significant potential for crossover applications in the ACL domain, providing a robust method for assessing and prioritizing environment novelty to enhance curriculum learning. When the task space allows for unbounded complexity, autocurricula methods such as UED offer promising pathways to open-endedness by co-evolving an adaptive, infinite set of tasks for the agent.Traditionally, learning without extrinsic rewards or fitness functions has been studied in evolutionary computation where it is referred to as 'novelty search' [30, 31] . In novelty search, the novelty of an agent's behavior is typically quantified by measuring the distance between a user-defined feature or behavioral descriptor and its nearest neighbor in the population. Consistent with our findings, the open-ended learning literature has long recognized that high-performing solutions often emerge not through fitness optimization alone but through novelty-driven exploration. Despite these parallels, novelty search in environment design remains underdeveloped. Early work such as POET [56] and its successor [57] in open-ended RL have started drawing connections, linking environment design with principles of open-ended exploration. However, these approaches rely on a population of agents and distance-based novelty measures that lack curriculum-awareness; they do not adapt to the specific experiences induced by the curriculum nor improve the agent's sample efficiency in reducing uncertainty across the state-action space. More recent work by Zhang et al. [60] proposed to leverage foundation models to quantify human notions of "interestingness" (e.g. tasks that are both novel and worthwhile) in order to narrow the environment search space. It is unclear how to combine the insights from Zhang et al. [60] and this paper. Integrating these insights with our work presents an intriguing challenge. On one hand, CENIE provides a principled, general approach to quantifying novelty through state-action coverage, circumventing the need for subjective evaluations of "interestingness" using foundation models. On the other hand, Zhang et al. [60] points out critical pitfalls in novelty search, such as the potential for agents to exploit novelty measures, generating superficial variations that fail to yield genuinely meaningful insights. This highlights numerous exciting research directions for aligning novelty search with the concept of "interestingness," potentially combining the strengths of principled coverage-based novelty measures with more nuanced assessments of task value.In this paper, we demonstrated the application of GMMs to quantify the novelty of environments generated under the UED paradigm. We then validated the effectiveness of this novelty metric in prioritizing levels. Nevertheless, our work has some limitations. First, while we demonstrated the utility of the CENIE framework for novelty quantification and level prioritization, we did not explore its potential for directly generating novel environments. We anticipate that with creative manipulations, the GMM likelihood scores could directly inform level generation, either through a principled level generator (as in PAIRED) or by guiding mutations (as in ACCEL). This approach may lead to a more sample-efficient generation process, reducing the variance inherent in random generation.Second, we did not experiment with alternative weightings between regret and CENIE's novelty in level replay prioritization, as our experiments used a fixed 0.5-0.5 weighting (as in Eq.5). We hypothesize that tuning these weights based on domain characteristics, such as the required level of exploration or reward sparsity, could improve performance. Additionally, employing dynamic weighting schemes, such as linearly decaying weight adjustments or adaptive strategies based on the agent's learning progress, may further enhance curriculum optimization.Third, GMM-based clustering may encounter challenges due to the curse of dimensionality in highdimensional state-action spaces. While our current CENIE-augmented algorithms demonstrated significant improvements, future work could explore dimensionality reduction techniques, such as Principal Component Analysis (PCA; [25] ) or t-distributed Stochastic Neighbor Embedding (t-SNE; [54] ), to improve coverage representation in higher-dimensional settings. However, even with dimensionality reduction, such representations may still struggle in environments where the observations contain exogenous information irrelevant to the agent's control. Specifically, although our experiments showed strong empirical gains in simplified environments, the current approach is vulnerable to the noisy TV problem [19] , where novelty-driven level prioritization may focus on unpredictable noise elements of the environment, rather than beneficial learning experiences. This limitation highlights the importance of balancing level prioritization between novelty and regret to ensure the agent focuses on genuinely novel environments rich in learning potential.Furthermore, effective state representation is crucial for the CENIE framework. The CENIE framework is not restricted to raw state inputs; it can operate on indirect encodings, such as latent-space representations obtained from a generative model of the environment. This approach would allow CENIE to capture action-relevant information and necessary temporal dependencies between states, providing a more focused basis for novelty estimation. We expect that density-based novelty estimation could improve further by using latent representations from more expressive generative models, such as Variational Autoencoders (VAEs) [29] , which can capture richer, more informative structures in the state space.Finally, while we used GMMs for environment novelty quantification, the CENIE framework is not limited to this model, as mentioned in the main body of this paper. GMMs may face limitations in capturing more complex distributions in real-world settings, and our choice of GMMs was primarily intended to illustrate the empirical benefits of quantifying novelty using state-action coverage in simpler environment settings. It is important to point out that fitting multiple GMM on the updated state-action coverage distribution and selecting the best one every rollout can incur additional computational costs. For future work aiming to replicate our approach, exploring periodic refitting (similar to the strategy used in ALP-GMM) could be worthwhile, as it may achieve comparable effectiveness while significantly reducing computational demands. Future work could also investigate more advanced density models, such as Variational Gaussian Mixture Models [11] , Deep Gaussian Mixture Models [55] , or Normalizing Flow Models [44] . Additionally, there may be alternative approaches beyond density models for representing state-action coverage that could further enhance CENIE's effectiveness. We believe there are many promising directions for the CENIE framework, and we leave these potential extensions to future work.In this section, we provide the details about the experiments and implementations, including domain properties and additional information about CENIE and the baseline algorithms. All of our experiments are run with a single V100 GPU or GeForce 3090 GPU, using 10 Intel Xeon E5-2698 v4 CPUs. The baseline algorithms and evaluation environments are implemented using the DCD codebase provided by Jiang et al. [27] , Parker-Holder et al. [37] . The CENIE framework and our current evaluations build upon and significantly extend a preliminary version of our work [52] , where the framework was initially named "GENIE." We have since enhanced the framework and opted to rename it to CENIE, following the release of a similarly-named, related work by Bruce et al. [17] , which appeared around the same time. This change was made to distinguish our contributions clearly and avoid confusion within the research community.In this section, we provide more details about the GMM fitting process that was absent from the main body. Given an initial buffer containing past state-action pairs, Γ, and a selected number of Gaussians, K, we first use the k-means++ algorithm to perform a fast and efficient initialization of the GMM parameters [12, 4] , λ Γ = {(α 1 , µ 1 , Σ 1 ), ..., (α K , µ K , Σ K )}. We then optimize λ Γ using the Expectation Maximization (EM) algorithm [22, 43] . The EM algorithm uses the initial values λ Γ to estimate a new λ ′ Γ such that P (X|λ ′ Γ ) > P (X|λ Γ ). This process is repeated iteratively until some convergence threshold is fulfilled. Each iteration of the EM algorithm can be separated into the E-step and M-step. In E-step, the posterior probability for each component i generating the sample point x t is denoted by w t,i ,w t,i = P (i|x t ) = α i N (x t |µ i , σ i ) K i α i N (x t |µ i , σ i )where t = 1, 2, ..., N and i = 1, 2, ..., K. M-step computes the maximum likelihood estimation (MLE) using w t,i following re-estimation formulas which are derived from the partial derivatives of the log-likelihood functions and guarantee a monotonic increase in the model's likelihood value.α i = 1 N N i=1 w t,i , µ i = N i=1 w t,i N i=1 w t,i x t σ i = N i=1 w t,i N i=1 w t,i (x t -µ i )(x t -µ i ) TWe iteratively apply the E-step and M-step until the parameters converge, i.e., ||λ ′ Γ -λ Γ || < ϵ, where ϵ is a small threshold.As mentioned in the main paper, we deliberately employ a finite window for Γ to account for the effects of catastrophic forgetting. This allows levels with state-action pairs encountered in the past but subsequently forgotten by the agent's policy to regain novelty and be included back in the agent's training curriculum. Furthermore, to ensure effectiveness in clustering the state-action space, we utilized a semi-online GMM model that is able to adapt its number of Gaussians, i.e. K, to that of the highest silhouette score.We use the PyCave [13] Python library to fit the GMM using GPU acceleration, which also provides an efficient abstraction for the Expectation-Maximization (EM) algorithm. We use the PyTorch Adapt [35] Python library to calculate the silhouette scores. The hyperparameters for fitting the GMM for all domains are shown in Table 7 .Besides the algorithm for ACCEL-CENIE shown in the main paper under Algorithm 1, we also provide the algorithm for PLR-CENIE here under Algorithm 2.In the Minigrid domain, the teacher creates maze instances consisting of a 15 × 15 grid, where each empty tile can be occupied by the agent, the goal, an obstacle (i.e. block), or an empty space that can navigate through. The student is aware of its orientation and is limited by partial observability, i.e. it only has a 5 × 5 view in front of it. The student agent can only move forward and turn left/right, and will stay in place if it hits an obstacle. The student agent is implemented based on PPO [49] with an LSTM-based recurrent network structure to deal with partial observability. We use the LSTM hidden states as representations within our GMM, allowing the density model to capture temporal dependencies between states. The student agent receives a reward upon reaching the goal, where H is the episode length and H max is the maximum length (set to 250 at training) for an episode. The agent receives a reward of r = 1 -(num step /H max ) when it reaches the goal position and 0 if it fails to reach the goal. The collection of states in this domain depicts the scenarios the agent needs to navigate through.In BipedalWalker, the teacher agent generates new levels by specifying the values of the eight environment parameters (e.g., ground roughness, number of stair steps, pit gap width, etc). As for the student agent, it needs to determine the torques applied on its joints and is constrained by partial observability where it only knows its horizontal speed, vertical speed, angular speed, positions of joints, etc. The student agent receives positive rewards as it walks towards the goal position and will receive a large negative penalty if it falls down. The BipedalWalker domain is modified on top of the BipedalWalkerHardcore environment from OpenAI Gym, introduced by [56] and improved by [41, 37] . The student agent receives a 24-dimensional proprioceptive state corresponding to inputs Sample replay decision, ϵ ∼ U [0if ϵ ≥ 0.5 thenGenerate a new level l θ by G 7:Collect trajectories τ on l θ , with stop-gradient η ⊥Compute novelty score for l θ using λ Γ (Eq.3 and Eq.4)9:Compute regret score for l θ (Eq.1 and Eq.4)10:Update B with l θ if P replay (l θ ) is greater than that of any levels in B (Eq.5)11:else 12:Sample a replay level l θ ∼ B according to P replay 13:Collect trajectories τ on l θ 14:Update π η with rewards R(τ )15:Compute novelty score for l θ using λ Γ (Eq.3 and Eq.4)16:Compute regret score for l θ (Eq.1 and Eq.4)17:Update P replay with novelty and regret scores 18:Update Γ with τ and resize to W19:for k in range K min to K max do 20:Fit a GMM k with k components on Γ and compute its silhouette score 21:end forSelect GMM parameters with the highest silhouette score to replace λ ΓCollect trajectories τ on l θ , with stop-gradient η ⊥Update B with l θ if P replay (l θ ) is greater than that of any levels in B (Eq.5)25:end if 26: end while from its lidar sensors, angles, and contacts, which also form the state representation for our GMM. The partial observability here means the agent does not have access to its positional coordinates. The environment parameters and their corresponding ranges are shown in Table 4 . Note, there will be a singular value to specify Ground Roughness and the Number of Stair Steps, and a min and a max value to define the PitGap Width, Stump Height, and Stair Height, and thus we will have eight environment parameters in total. The CarRacing domain was introduced and customized by [27] . In CarRacing, the teacher creates tracks by using Bézier curves by 12 control points within a fixed radius of the center of the playfield.A track consists of a sequence of L polygons and L is fixed on the training tracks and varies on different testing tracks. While driving on the tracks, the student receives a reward equal to 1000/L. The student additionally receives a reward of -0.1 at each time step. The student observes an RBG image of size (96x96x3), where (96x96) is the (height x width) of the observed image and 3 is the number of RGB channels. Consistent with previous UED literature, we employ a CNN model to preprocess the raw image observations. The CNN extracts high-level features from the raw images, reducing their dimensionality and capturing important spatial patterns that are crucial for understanding the environment's dynamics. By feeding these feature representations into the GMM model instead of the raw image input, we ensure that the density estimation focuses on meaningful aspects of the state space rather than being overwhelmed by the complexity and noise of raw pixel data.The 20 testing F1 tracks by [27] have various track lengths and different maximum episode steps.As such, different min-max normalization ranges are used for each track to produce the IQM and Optimality Gap plots. We list the 20 test tracks and their corresponding min-max normalization ranges in Table 5 . In this section, we provide more technical details on some of the baseline algorithms used in our experiments, specifically Domain Randomization (DR), Minimax, PAIRED, PLR ⊥ , DIPLR, and ACCEL. We summarize the key differences between the baseline algorithms in Table 6 .The Domain Randomization (DR) teacher uniformly randomizes each dimension in the environment parameter space to generate various environments. The PAIRED teacher estimates regret by leverage two agents: an antagonist agent and a protagonist agent (student). In practice, PAIRED derives regret by taking the antagonist's (A) maximum performance and the protagonist's (P ) average performance over several trajectories, allowing for more accurate approximations. Let U π (τ ) denote the total reward obtained by a trajectory τ produced by policy π on the level θ. Regret is measured in PAIRED via:REGRET θ (π A , π P , θ) = max τ A U θ (τ A ) -E τ P [U θ (τ P )]where π A and π P are the antagonist's policy and the protagonist's policy, respectively. PAIRED teacher constantly creates levels that are slightly beyond the ability range of the protagonist and within the ability range of the antagonist such that the regret is maximized. The pseudocode of the PAIRED algorithm is given in Algorithm 3.Input:Randomly initialize Protagonist π P , Antagonist π A , and teacher Λ Initialize: replay buffers B 1: while not converge do Use teacher to generate environment parameters: θ ∼ Λ. Use θ to create environments, l θ 3:Collect Protagonist trajectory τ P in l θ . Compute Protagonist's average return: E θ [V (π P )]4:Collect Antagonist trajectory τ A in l θ . Compute Antagonist's average return:E θ [V (π A )] 5: Compute regret: REGRET = E θ [V (π A )] -E θ [V (π P )] 6:Train Protagonist policy π P with RL update and reward = -REGRET 7:Train Antagonist policy π A with RL update and reward = REGRET 8:Train teacher policy with RL update and reward = REGRET 9: end while However, the PAIRED algorithm faces several drawbacks [34] . Both the antagonist and protagonist policies are constantly updating, making the problem nonstationary. Furthermore, PAIRED suffers from a long-horizon credit assignment problem since the teacher must fully specify an environment before receiving a sparse reward in the form of feedback from the antagonist and protagonist agents. PLR seeks to circumvent this issue through the use of regret for prioritized selection of levels for replay rather than active generation. PLR uses Positive Value Loss (PVL), an approximation of regret based on Generalized Advantage Estimation (GAE; [48] ):PVL θ (π) = 1 T T t=0 max T k=t (γλ) k-t δ θ k , 0 ,where γ, λ and T are the MDP discount factor, GAE discount factor and MDP horizon, respectively. δ θ k is the TD-error at time step k for θ. However, the use of PVL may introduce bias due to the bootstrapped value target. An alternate heuristic score function is Maximum Monte Carlo (MaxMC), which replaces the bootstrapped value target with the highest return observed on the level during training. By using this maximal return, the regret estimates become independent of the agent's current policy:MAXMC θ (π) = 1 T T t=0 R θ max -U (τ π ) ,where R θ max is the maximal return of π on θ. We primarily focus on PVL because the original implementations of ACCEL and PLR ⊥ in Jiang et al. [27] , Parker-Holder et al. [37] found better success with the PVL scoring function for the experiments domains, i.e. Minigrid, BipedalWalker, and CarRacing, used in this paper. Future research could explore the potential of using the MaxMC scoring function to see if it yields different outcomes when combining ACCEL and PLR ⊥ with CENIE. The Diversity Induced Prioritized Level Replay (DIPLR [33] ) algorithm extends PLR ⊥ by prioritizing level replay based on both regret and diversity. Here, diversity is quantified using the Wasserstein distance between the agent's trajectories across levels in the replay buffer. The limitations of DIPLR are highlighted in the main body (see Section 3) Sample replay-decision, ϵ ∼ U [0if ϵ ≥ 0.5 thenGenerate a new level l θi by G 7:Collect trajectories τ i on l θi , with stop-gradient η ⊥ 8:Compute the regret, staleness and distance for l θi 9:else 10:Sample a replay level l θj ∈ L according to P replay 11:Collect trajectories τ j on l θj and update π η with rewards R(τ j )12:Compute the regret, staleness and distance for l j 13:end ifFlush Γ and collect trajectories on all replay levels to fill up Γ 15:Update regret, staleness, and distance for l θi or l θj 16:Update L with new level l θi if its replay probability is greater than any levels in L 17:Update replay probability P replay 18: end while Finally, the state-of-the-art UED algorithm, ACCEL, improves PLR ⊥ by replacing its random level generation with an editor that mutates previously curated levels to gradually introduce complexity into the curriculum. ACCEL makes the key assumption that regret varies smoothly with the environment parameters θ, such that the regret of a level is close to the regret of others within a small edit distance. If this is the case, then small edits to a single high-regret level should lead to the discovery of entire batches of high-regret levels -which could be an otherwise challenging task in high-dimensional design spaces. An intriguing area for future exploration is the interaction between ACCEL's editing mechanism and the novelty-driven level prioritization introduced through CENIE. Specifically, it is worth investigating whether the editing mechanism does synergize with CENIE to produce levels that simultaneously maximize both novelty and regret, further enhancing the diversity and effectiveness of the generated curriculum.In this section, we provide the hyperparameters we used for both CENIE-augmented and baseline algorithms in our experiments. We employ the same set of CENIE parameters for both ACCEL-CENIE and PLR-CENIE. We provide all the parameters for our implementations in Table 7 . Justification: We've described our claims and contributions clearly in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper introduces an algorithm for exploring environment spaces efficiently based on a new novelty metric. 
The authors provide compelling experimental results across a range of environments and were also able to address the concerns of the reviewers through the rebuttal phase. 
This resulted in a number of score increases and an strong consensus that the paper should be accepted. 
Given the relevance of the work I believe it would be great for this work to be highlighted as an oral at the conference.

Reviews: review: paper_summary: paper_summary: This paper proposes adding a domain-general metric for promoting novelty to state of the art UED methods in order to help the environment generator better explore and cover the space of environments.  The novelty bonus is based on a surprise of a (state, action) pair model which is a learned Gaussian mixture model.
strength_weakness: strength_weakness: The method would be more convincing if the generated levels for each method where visualised and compared directly.  Ideally one could demonstrate that there are a motif of level being generated which were not being generated before.  For instance, displaying the distribution over stump hight late in training of ACCEL and ACCEL-GENIE to show that ACCEL-GENIE has a more even coverage of the space, and sampling a few levels randomly from each so that this can be visually confirmed.  

Similarly, the paper would be much improved if the mechanism behind the results was dug into in more depth. If the theory is that the diversity term results in kinds of high-regret levels being presented which were left out of the buffer previously, it would be ideal to demonstrate that this happens directly.

On line 258 results are over-claimed, since ACCEL-GENIE is within the margin for error of ACCEL in all, or nearly all, of the environments. This claim should be corrected. I don't think this is essential for acceptance of the paper, as the minigrid benchmark appears pretty close to being saturated, and it serves largely as an MNIST for the field. The empirical results in the other two environments stand on their own.

#### Clarity:
It would be good to have error bars in Table 1
The plots in Figure 7 should be fixed so the colors of the same method match
There is a stray parenthesis on line 303
Equation 1 shows the PAIRED objective "optimized" for deterministic domains, the PAIRED objective as a direct comparison between the two expected returns is canonical. 

I would also suggest reconsidering the name, as another prominent environment-generating model named GENIE has been released recently. I expect that, in talking about this work, people will want to talk about how the two could be combined, which could get quite confusing.

In the abstract I think it is not clear that novelty, on it's own, is critical for agent's generalisation ability. Two situations which are different but not in a way that is relevant for the task and are processed by the network in the same way are effectively the same, and would not effect generalisation ability.

It also appears that in a few places the "underspecified" in UPOMDP is taken to mean that there is a one-to-many mapping between parameters and environments. In general this is not the case, as the minigrid environment has a one-to-one mapping between parameters and environments, the "underspecified" simply means that these parameters are not given by the designer. It is fair to want UED algorithms to work even when there is such a one-to-many mapping, but it is not required as part of the problem formulation.
questions: questions: What sort of qualitative difference do you see between ACCEL-GENIE and ACCEL levels?
limitations: limitations: See weaknesses.
paper_summary: paper_summary: This paper proposes using novelty quantification in an unsupervised environment design for training a more generalizable policy. Built on an intuition that environments with unfamiliar states are novel environments, their proposed algorithm uses Gaussian mixture models to allow an RL agent to explore novel environments. The authors of this paper compare their proposed method in various benchmarks against multiple baselines to show empirical improvement in performance.
strength_weakness: strength_weakness: - The idea of using unfamiliar states to quantify uncertainty seems similar in concept to the curiosity-driven approaches in RL. However, this paper does not address the relevant literature. It would be much better if the authors could explain how their idea fits in with the findings and theory in curiosity-based RL approaches and what the paper contributes to this avenue of thinking.

- When it comes to UED in curriculum learning to train a more generalizable policy, there is Genetic Curriculum (Song et al, 2022) which also uses UED in curriculum learning to train a generalizable policy. Since the paper was also evaluated on the BipedalWalker and BipedalWalkerHardcore environments, it would be better to compare and contrast how GENIE with Genetic Curriculum.

- GENIE uses a fixed window FIFO buffer, but would this be able to reach an equilibrium? For example, if the agent explores level set A at the expanse of forgetting level set B, and goes back to exploring level set B at the expanse of forgetting C, and so on, would agents trained by GENIE converge to a steady-state behavior?

- Finally, there are some minor typos and editorial mistakes. For example, line 63, PLR, and ACCEL are mentioned without explaining what those are.
questions: questions: The questions I hope to be addressed by the authors are the ones listed above. 1) Explanations on how this paper fits in with curiosity-based approaches and the GENIE's contributions, 2) explanations and comparisons with Genetic Curriculum, and 3) Will the agents trained by GENIE reach a steady-state equilibrium?
limitations: limitations: The authors has addressed the limitations of this paper.
paper_summary: paper_summary: This paper focus on the unsupervised environment design (UED) problem, whereby a student trains in an adaptive curriculum of environments proposed by a teacher. The authors propose GENIE, a method for assessing novelty of environments, which essentially means the teacher prioritizes environments with high exploration potential/info gain for the student. Experiments are conducted in three different relevant domains and the results are clear and show a reasonable gain. I thus favor acceptance. 

One thing to note is the general idea of choosing environments based on novelty (rather than regret) is itself a new idea, thus, the authors do not need to focus as much on the use of GMMs, which may be limiting.
strength_weakness: strength_weakness: * There is a strong assumption that all transitions are independent, which is not always true. What if there is an environment where an agent needs to conduct some initial behavior (e.g. finding a key) and then using it to act later (e.g. opening a door)? It just feels like the method is designed based on the toy environments we use in RL research, and is not actually scalable for larger more complex domains in the future.
* Hate to be that person, but the name GENIE is taken by multiple works already and recently by Bruce et al (2024) in a highly relevant paper. It would be recommended to find a more relevant acronym or simpler name.
* The arrows in figure 2 are too small.
questions: questions: What other approaches could be used to assess novelty at larger scale? Can you show any examples of levels that have high novelty but low regret, and turn out to be useful training levels for the agent?
limitations: limitations: Discussed in the Appendix.
paper_summary: paper_summary: After rebuttal

I have upped my score to 7. I think this paper is good, as it makes a small, easy to implement and simple to understand change to existing UED methods, and it delivers improved empirical results.


-----


This paper uses a GMM-based method to quantity novelty of levels in UED. It uses the state-action distribution (unordered, i.e., not trajectory distribution) induced by the agent on a level and fits a GMM to the data from previously sampled levels. A new level's novelty can be determined by computing the likelihood of its induced state-action distribution under the fitted GMM.
strength_weakness: strength_weakness: - While the focus on unordered transition tuples is mentioned as a benefit, I would think that there are certain environments where the temporal nature of trajectories are important. Could you comment on this please?
- Minor
	- Line 88, for minimax one also often cites [1]
	- The $\tau$'s beneath the max/expectation in equation (1) don't have the A and P superscripts.

[1] Pinto, Lerrel, et al. "Robust adversarial reinforcement learning." _International Conference on Machine Learning_. PMLR, 2017.
questions: questions: - Algorithm 1, please indicate what the blue text means (I assume changes to ACCEL but being clear on this would be helpful.)
- Please provide aggregate results averaged over all of the minigrid eval. levels. It is hard at a glance to see how the algorithms compare. 
- Do the GENIE generated levels actually look more diverse than those generated by e.g. ACCEL? Please show a sampling of levels.
- In table 1, PLR-GENIE has the most state-action coverage, but performs much worse than the ACCEL-based methods. This seems to contradict the claim that the increased diversity is causing better performance. Could you explain this please?
- There may be some confusion between your GENIE and [1]
- For car racing you do use images, is the image simply flattened? Could this have problems with e.g. not being translationally invariant?

[1] Bruce, Jake, et al. "Genie: Generative interactive environments." _Forty-first International Conference on Machine Learning_. 2024.
limitations: limitations: - I think putting limitations in the main paper would be better. Figure 5 feels not super necessary and can be moved to the appendix if space is an issue.
- Some other limitations I can think of, please comment/explain
	- Choosing the range of $K$ (number of GMM) kernels can be challenging?
	- Do I understand correctly that you just concatenate the observation and actions to form $x$, which is then used to fit the GMM? How can this scale to much larger observations?

Comments: title: Thank you Reviewer jT1H
Thank you Reviewer kpCs
Thank you Reviewer KtoT
Thank you Reviewer qR9x
Response to Rebuttal by Authors
No change to score
Replying to Reviewer jT1H
Replying to Reviewer jT1H
Discussion
comment: It has been a pleasure engaging with you during this discussion phase. We are pleased to have addressed your questions thoroughly and appreciate your increased confidence in our paper.
Thanks. I have updated my score to 7.
Thank you for considering our rebuttal and for your continued confidence in our paper.

> If it is accepted please include a discussion on scaling the approach in the future work/conclusion. This could even be by combining it with an environment generator like Bruce et al's Genie :)

Absolutely, we are enthusiastic about the potential for future work to further develop and broaden our novelty-driven autocurricula approach. Thank you once again for your time and support.
We sincerely appreciate your engagement in the review process. We are glad that our rebuttal has strengthened your confidence in our paper.
Thank you for your engagement and providing such technical feedback. We truly appreciate your confidence in our work.

> One test that could show this effect would be if the policies from both realities found the levels from the GENIE reality to have higher regret. I agree if this was not the case it would not be negative evidence.

> Another good measure would be to fix a policy (and freeze it so it does not train) and run both to see which finds higher regret levels.

Also, thanks for suggesting these creative experimental setups. We'll begin implementing these experiments and will incorporate the results in our revised manuscript.
Thank you for your response. The added visualisations increase my confidence in the method, so I will be increasing my score accordingly.

> As such, it is impossible to directly measure whether prioritising or not prioritising a level via GENIE would lead to discovering a higher regret level down the road because the divergence in realities would result in different policies with non-commensurable subjective regrets. 

One test that could show this effect would be if the policies from both realities found the levels from the GENIE reality to have higher regret. I agree if this was not the case it would not be negative evidence.

Another good measure would be to fix a policy (and freeze it so it does not train) and run both to see which finds higher regret levels.
Thank you for the rebuttal. I don't see much scope to increase my score, I think the paper has sufficient merit to be accepted. If it is accepted please include a discussion on scaling the approach in the future work/conclusion. This could even be by combining it with an environment generator like Bruce et al's Genie :)
Once again, thank you for your engagement in this discussion phase and the following should clarify your questions:

**Q: For this, could you foresee a problem with the CNN's representation becoming stale? I.e., as the agent learns, the representations of the same levels would change, inducing a sort of shift. So a highly novel state could come from a previously played level, the novelty being from the updated representation.**

A: You raised a very astute point and we have considered this previously. This “representational shift” issue is circumvented by storing the raw image states (instead of learned representations) in GENIE's buffer and getting the updated representations from the CNN model before GMM fitting. In retrospect, if the designer does choose to store learned representations for convenience, GENIE's choice of using a FIFO buffer would also naturally mitigate this representational shift.

**Q: Clarifying this in the updated manuscript would be helpful.**

A: Definitely, thanks for highlighting this!
Thank you.

> A: For the Car Racing domain, we are also using the CNN-preprocessed states for the GMM.

For this, could you foresee a problem with the CNN's representation becoming stale? I.e., as the agent learns, the representations of the same levels would change, inducing a sort of shift. So a highly novel state could come from a previously played level, the novelty being from the updated representation.

> A : Not quite. Our use of the word "order" in different contexts can admittedly be confusing, 

Thanks for this. Clarifying this in the updated manuscript would be helpful.
The reviewer thanks the authors for a well-detailed rebuttal. The reviewer has updated the scores accordingly.
Thank you for the very prompt response!

**Q: For car racing I meant specifically how the states are input into the GMM and not the policy. Do you use a CNN as a preprocessing step before the states are passed to the GMM/do you use the agent's CNN, or do you just flatten the obs and pass it to the GMM?**

A: For the Car Racing domain, we are also using the CNN-preprocessed states for the GMM.

**Q: Could you please expand a bit more on how you can incorporate a temporal relationship between states? Naively the thing to do would be to effectively frame stack to get an augmented state space. Are there other ways?**

A: There is active research regarding incorporating temporal information between states. Frame-stacking as you mentioned, is a simple method to incorporate temporal information for pixel environments. Other general methods would be to include recurrent or attention layers. For more specific hierarchical RL methods, temporal abstractions via decomposing sequences into simpler sub-tasks and operating over different temporal scales can be considered.

**Q: Relatedly, the policy's representation does not have to be the same as GENIE's, right? So you could framestack for one but not the other?**

A: That is correct. The representation used in GENIE is flexible and really depends on the degree of abstraction/specificity the developer is interested in regarding novelty in states.

**Q: Then regarding the increased state coverage. If I understand correctly, you are saying the order \& diversity matters, and not diversity alone? In that case I think rephrasing the discussion in l297-l300 would be beneficial, as I at least did not get that impression from reading it.**

A: Not quite. Our use of the word "order" in different contexts can admittedly be confusing, but we appreciate the opportunity to clarify. In Lines 184-187, we expressed a key strength of GENIE being its prioritization of diversity in individual induced experiences of the environment, independent of the order in which they are presented. In this context, "order" is **intra**-environment and refers to the **sequence of experiences (state-action pairs)** presented by a **single environment**. 

For our previous response regarding the differences in ACCEL and PLR, we were talking about "order" in the **inter**-environment context, more specifically how the **UED algorithm's teacher** presents the **sequence of environments** (i.e. the curriculum). GENIE enhances the diversity of individual experiences throughout all the environments in the entire curriculum but the sequence in which these diverse environments is presented in the curriculum depends on the underlying algorithm. The reason why ACCEL performs better than PLR is because the former's bootstraps its curriculum with simple levels (e.g. empty mazes) and gradually increases in complexity via minor mutations but the latter simply uses domain-randomized levels throughout. 

We hope the above clarifies the confusion. If not, we're happy to provide more specific examples and clarifications.
Thank you for your detailed response!

A few more follow ups

- For car racing I meant specifically how the states are input into the GMM and not the policy. Do you use a CNN as a preprocessing step before the states are passed to the GMM/do you use the agent's CNN, or do you just flatten the obs and pass it to the GMM?

- Could you please expand a bit more on how you can incorporate a temporal relationship between states? Naively the thing to do would be to effectively frame stack to get an augmented state space. Are there other ways?
- Relatedly, the policy's representation does not have to be the same as GENIE's, right? So you could framestack for one but not the other?


- Then regarding the increased state coverage. If I understand correctly, you are saying the order & diversity matters, and not diversity alone? In that case I think rephrasing the discussion in l297-l300 would be beneficial, as I at least did not get that impression from reading it.