Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an ML pipeline to evolve agent topologies for open ended learning by combining previous works EPOET and NEAT. Experiments are done on the 2D bipedal environment.
strength_and_weaknesses: 1. The paper lacks sufficient insights/results/illustrations to understand how the agent topologies are changing, most of the methodology and evaluation is borrowed from EPOET whose focus was more on novel environment generation rather than agent topology, so the results and discussion seem inadequate. Further EPOET already introduced the idea of using NEAT for environment generation which the authors have adapted for topology evolution.

2. Statistical significance of the results is poor, only 2 seeds have been used for the comparisons. Performance of RL based agents is typically high variance, so it is not justified to compare the agent policies/performance with so less data.

3. Currently the work comes short on novelty and lacks reusable insights that can actually be informed for progressing open ended learning. Are there any interesting patterns in the way the topologies evolve? What insights can be drawn about choosing the hyperparameters for fitness and mutation for open ended learning? 

4. To me, the empirical evaluation seems lacking as no adequate baseline for topology evolution is used.

clarity,_quality,_novelty_and_reproducibility: 1. The paper is mostly readable except for the heavy use of abbreviations, the technical details about the existing methods used are not complete and can be improved by discussion in appendix for instance.
2. The paper lacks novel ideas and quality results as discussed above and would need significant improvements before being publishable.
3. Empirical evaluation is not adequate in terms of statistical significance, also on one domain is being used for comparison with the baseline. 
summary_of_the_review: Incremental idea with poor execution.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: this paper propose to simultaneously evolve the environments and the agents by increasing complexity of the later. the proposed method extends EPOET. 
strength_and_weaknesses: 1. the paper should be reorganized in the sake of readability. 

For example "EPOET improves upon POET by adding in two algorithmic improvements" (1) ... (2). 
What section does example (1) and (2)? If those the main contributions and improvements, one would expect to see them explicitly in the text.

2. a few relevant works are not mentioned. E.g., "Task-Agnostic Morphology Evolution" and similar.

3. I am not sure that a method can be called reproducible if it takes 200000 CPU hours. A simple examples clarifying the main points one by one will be helpful. 

4. "Accumulated Number of Novel Environments Created and Solved (ANNECS), a metric for open-ended learning that, intuitively, describes the amount of interesting new...". What is the definition of "interesting"?
is Novel and interesting the same? 


clarity,_quality,_novelty_and_reproducibility: 200000 CPU hours might be hard to reproduce - simple set of experiment is required to understand the contributions.

the main definitions/contributions are not formally provided (definition of interesting, etc)


summary_of_the_review: the lack of simple experiments demonstrating the key points and the lack of explicit definitions prevents me from recommending this paper for publication in ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper provides an extension of the EPOET algorithm for co-evolving a population of RL (PPO) agents with a population of task configurations, such that the topology of agents can also be co-adapted throughout training. Specifically, this work introduces Augmentative Topology EPOET (ATEP), which uses NEAT instead of PPO to update the population of agents, and thus their topology in addition to network weights throughout training. As NEAT requires a population, ATEP effectively co-evolves populations of populations of agents, and thus requires new transfer mechanisms for transferring agents evolved to solve one environment to become the "elite" for another environment, as necessitated by EPOET. To this end, the paper proposes two such transfer mechanisms, Fitness-Based Transfer (FBT) and Species-Based Transfer (SBT). The main results show that ATEP methods result in more robust agent populations (i.e. capable of solving more task configurations) and leads to faster growth in the number of novel, meaningfully different tasks that are generated compared to baseline methods.
strength_and_weaknesses: ### Strengths
- Open-endedness is an important, underexplored topic in ML research.
- The paper provides an interesting extension of EPOET that allows agent topologies to be co-evolved with the environment.
- The paper proposes two new transfer mechanisms for EPOET that are required for applying NEAT in updating the agent population, both of which serve as novel extensions to EPOET's standard transfer mechanism.

### Weaknesses
- It seems the comparison to EPOET is unfair because **NEAT requires ATEP to effectively maintain more agents than EPOET**. It is not clear if this paper controls for the larger population size of ATEP.
- The main experimental results (e.g. Figure 2, Figure 5) are only based on two training runs.
- The ANNECS values in Figure 2 for EPOET do not seem to match that from the original paper, where at 20k updates, EPOET reaches an ANNECS score of 100, thus matching the performance of SBT-ATEP in this work. This may be due to the authors reduction of the number of active environments from 40 to 20 with respect to the original study. However, the authors should run their implementation of EPOET on the original setting of 40 environments on 3 seeds to ensure their implementation is correct, or otherwise state that they reuse the official open source implementation of EPOET.
- The agent network size seems to play a significant role in improved ANNECS and robustness. Therefore, authors should also provide more detailed discussion on how the choice of 20x20 and 40x40 compares to the original EPOET implementation, which achieved much higher ANNECS scores in the same number of updates.
- The paper is unclear about several key details, described in the Clarity section.
- The paper claims the original POET work uses BipedalWalkerHardcore, but this is inaccurate. POET and EPOET used modifications of the BipedalWalker environment, not the specific BipedalWalkerHardcore environment, which refers to a specific setting of the BipedalWalker environment parameters.
- At the top of Page 6, the authors claim SBT-ATEP does not stagnate, but this does not follow from the results, because it is unknown if SBT-ATEP will stagnate beyond the number of training steps investigated.
- The paper does not provide the correct citation for the VAE. It should be Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." 2013.
- The introduction mentions prior works are limited in open-endedness due to the finiteness of the environment. However, the authors do not state this shared weakness in their own experimental environment.
- The authors do not provide any measures of environment complexity over the course of training, e.g. as done in prior works. This would be useful for comparing the difference in kinds of environments produced by ATEP vs. EPOET.
clarity,_quality,_novelty_and_reproducibility: ### Clarity
Several important details are unclear in the paper:
- The paper, while easy to follow at a high level, assumes much prior knowledge on some key concepts. The paper can benefit from providing intuitive and technical definitions of PATA-EC, ANNECS, and the novelty metric used by EPOET directly in the background section of the paper. These are core metrics that are never clearly defined.
- Similarly, a more technical definition of how FBT and SBT are performed would benefit clarity. A diagram illustrating examples of FBT and SBT would also be useful.
- A diagram of how NEAT can update a toy network would be useful to include in the background or method section.
- Equation 1 seems unnecessary for the main paper, and can instead be added to an appendix section detailing NEAT. This would free up room for including more details on the missing definitions described above. Further, the "excess and disjoint genes" are never defined, making this passage impossible to fully understand.
- Figure 1 should include references to the equation or section numbers where each component is defined, e.g. ANNECS.
- The key design choice for why FBT replaces the entire population upon transfer, rather than just some subset of individuals in the target environment population is unclear. SBT seems to benefit from this partial transfer, so it seems the FBT comparison is unfair in this regard.
- Figure 4 can benefit from more details in the caption about what the distributions on the top and right of the figure represent.
- The paper can benefit from stronger motivation for why ML research should care about open-endedness. The current discussion feels like a recitation of existing works without building a strong, compelling case for this line of work.
- I also suggest that authors restructure their paper to bring the contents of Appendix A into the main body of the paper, while improving the legibility and sizing of the plots in the paper.

### Quality
The paper feels quite rough in presentation, using even the wrong formatting for an ICLR paper, e.g. the horizontal margins are smaller than standard. The figures and captions are not polished and the overall writing can benefit from copy editing. 

### Novelty
ATEP is a novel extension of POET and the main experimental results seem promising, despite the aforementioned issues in clarity of presentation and potential issues with reproducing the original EPOET results.

### Reproducibility
The paper provides enough details to reproduce the results in principle, but the release of their code (as promised in the paper) will greatly improve reproducibility.
summary_of_the_review: While I like the ideas in this paper, the overall evaluation is only over 2 training runs and the quality of the presentation and writing (including eliminating the existing incorrect claims previously detailed) can be much improved. Importantly, the authors do not provide reassurance that ATEP is not outperforming EPOET purely due to its use of a larger effective population size—since NEAT is a population-based approach and each agent in the population is now optimized according to a NEAT population. Further, I would like to see more evidence for the fidelity of their implementation of EPOET compared to the original, given the deviation in ANNECS score. For these reasons and those stated previously in my review, I cannot recommend this paper for acceptance in its current state. I am very open to accepting this paper if the authors can provide an improved version of this work that addresses the issues described.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to use NEAT to evolve NNs with varying structures to solve an increasingly complex collection of reinforcement learning problems under the framework of open-ended learning. It also studied two alternative transfer mechanisms to transfer evolved NNs across different environments. Experiments show that the newly proposed algorithm may achieve good performance for one benchmark of open-ended learning.
strength_and_weaknesses: Strength:It is interesting and important to study the problem of open-ended learning. It is also interesting to consider structure/topology evolution in order to improve the effectiveness of evolved NNs towards solving complex problem instances.

Weakness:This paper is motivated by the assumption that generalization can be improved by allowing evolved NNs to become increasingly complex. However, the validity of this assumption may be questionable. In fact, as the experiment results showed, the NNs evolved by NEAT tend to have small sizes, which help to maintain good generalization performance. Hence the experiment results reported do not seem to be fully compatible with the key assumption of the algorithm design.

Several key concepts were used in the paper without thorough definition. For example, it remains unclear what "completely open-ended" on page 1 means. How to measure the degree of open-endedness? In line with this question, it remains unclear how open-ended learning is defined. What does it mean by "complexity arises randomly"? I cannot find any thorough mathematical definition of this learning problem. It is also unclear how the open-ended learning problem is related to other learning paradigms such as transfer learning and multitask learning.

Furthermore, the practical significance of open-ended learning is not investigated with sufficient depth in the paper. In Section 2, the authors introduced some practical applications of open-ended learning. However, it seems that none of such applications were investigated in the paper based on the newly proposed algorithm. Hence, the practical usefulness of the new algorithm remains questionable to a certain extent.

One major contribution of this paper appears to be using NEAT instead of ES (or other reinforcement learning algorithms) to drive the evolution of NNs. The introduction of NEAT in Subsection 4.1 appears to be following exactly the original design of NEAT. It is not clear what are the technical difficulties of using NEAT for open-ended learning and how the technical difficulties were actually tackled in this paper. A direct adoption of NEAT does not seem to be sufficiently novel. Furthermore, since Subsection 4.1 introduces only an existing algorithm, it may not be appropriate to put it in the section on new methods (or new algorithms).

Another major concern is regarding the proposed transfer mechanisms. According to the paper, "transfers agents across environments can prevent stagnation and leverage experience gained on one environment as a step towards solving another". However, what kind of "stagnation" is considered in the paper? Is this a common problem for existing algorithms, such as POET and EPOET? Why can the newly introduced transfer mechanism prevent stagnation? Without clear answers to all these questions, the technical novelty and contribution of the newly developed transfer mechanisms remain doubtful. The mechanism design also appears to be heuristic in nature. It is not clear what theoretical principles are being followed while designing the transfer mechanisms.

More technical details should be provided in the paper. For example, the first criteria for environment counting on page 3 is that the environment must neither be too easy nor too hard. However, what are the precise definitions of easy and hard environments and why?

The new algorithm appears to have several new hyper-parameters to be fine-tuned, for example the threshold delta. It may not be easy to tune these hyper-parameters. Hence it may be practically difficult to use the new algorithm for some real-world open-ended learning applications.

The new algorithm is computationally costly to run (up to 200k cpu hours according to the paper). As a result, only 2 seeds have been tested in the paper. It may not be easy to draw any solid conclusions from the experiment results on 2 seeds. Furthermore, the network architectures (40x40 or 20x20) experimented appear to be much smaller than the typically used architectures among the majority of deep reinforcement learning algorithms. It is unclear whether the new algorithm can still outperform the baselines upon increasing the complexity of the network architectures. In general, as the benchmark problem studied is simple, the claim that NEAT enables the new algorithm to solve challenging or complex problems is not strongly verified in the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written overall. However, several key concepts were used in the paper without thorough definition.

The paper introduces a seemingly straightforward adoption of NEAT in an existing framework for open-ended learning. The novelty of the paper does not seem to be strong. The technical novelty and contribution of the newly developed transfer mechanisms also require more justifications.
summary_of_the_review: It is interesting and important to study the problem of open-ended learning. However, the technical novelty of the paper does not seem to be sufficiently strong. The newly proposed algorithm is also computationally costly to run. Consequently, the practical usefulness of the new algorithm requires more investigations.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies a co-evolution of agent topologies with environments. There are a number of issues raised by reviewers. 
- There are problems with experimental evaluation, relating to number of runs, size of the agents, comparison given that we have a population size and reimplementation of the original method.
- The clarity and completeness of writing should be improved. 
- The idea is too incremental without strong performance to justify it.
justification_for_why_not_higher_score: - There are problems with experimental evaluation, relating to number of runs, size of the agents, comparison given that we have a population size and reimplementation of the original method.
- The clarity and completeness of writing should be improved. 
- The idea is too incremental without strong performance to justify it.
Decision: Reject