Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a framework that allows using any reinforcement learning (RL) algorithm within a population of agents. The contribution is to use quality diversity methods to evolve populations and maintain their diversity. The most commonly used method is MAP-ELITES, but MAP-ELITES does not work well on high-dimensional search spaces. The contribution of the paper is the development of a version of MAP-ELITES, called PBT-MAP-ELITES, that does not depend on a specific RL agent, is robust to hyperparameter choices, and scales to large population sizes.  Some experimental results are included for an example problem.
strength_and_weaknesses: Strengths: 
The changes made to MAP-ELITES are simple but seem to be effective.
Weaknesses: 
- the PBT-MAP-ELITES algorithm is a variation of PGA-MAP-ELITES, which seems quite simple even though it appears to be effective. 
- There are many variations of MAP-ELITES that are used in the experiments but for which there is no explanation in the paper. 
- The experimental results included are relatively limited and are not described with enough details to be reproducible. 
- The paper assumes familiarity with all the specific robotics problems used to test the algorithm.  
- The figure with the experimental results (Fig.2) is hard to read.
clarity,_quality,_novelty_and_reproducibility: The paper is clear but not always precise and sufficiently detailed. I do not think is possible to replicate the results. Too many details are missing.
summary_of_the_review: The paper presents an algorithm that is a new variation of MAP-ELITES. It evolves a population of agents, scales well to large population sizes, and is robust to hyperparameter choices. The experimental results that are included in the paper show good performance compared to other algorithms, but they are described in a succinct way, making it very hard to replicate the results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the limitations of reinforcement learning (RL) based MAP-ELITES approaches, an approach to quality diversity (QD) optimization. The authors highlight the following four problems with existing approaches: (i) high sensibility to hyperparameters; (ii) training instability; (iii) high variability in performance; and (iv) limited parallelizability. 

To overcome these limitations, the authors propose a population-based RL-based MAP-EILTES approach. Each agent in the population encodes the policy parameters, the hyper-parameters for the training, and some other internal parameters used during the training. The proposed framework is a generic framework in that one can easily replace the RL baseline to train each agent. 

The proposed approach is compared with other MAP-Elites variants with SAC and TD3 as their RL baselines on five robotics environments that are often used in QD-RL literature. Superior performance, in particular on environments with deceptive reward signals, are observed.
strength_and_weaknesses: # Strength

Approach: The approach is relatively simple. The RL baseline can be easily replaced. And it is parallel implementation friendly. 

Evaluation: The advantages of the proposed approach over existing MAP-ELITES approaches have been demonstrated on robotics environments in terms of the best fitness, coverage, and QD score. In particular, the advantage is pronounced on environment with deceptive reward signals is reported.

# Weaknesses

Clarity: The algorithm is described thoroughly by natural language in the main text. It is not easy to precisely understand, in particular, for those who are not familiar with MAP-ELITES. The algorithm is provided in the appendix, but the details are not given there. Moreover, because of a different symbol used in the algorithm and in the main text (M vs N), it was hard to fully understand. 

Algorithm validity: The authors say that the meta-learning mechanism for the hyper-parameter is included in the algorithm. However, as far as I understand, the hyper-parameter is only randomly sampled uniformly over the domain. whenever an agent in the population is replaced. It does not look like optimizing the hyper-parameter values.

Motivation and Algorithm Design and Experimental Evaluation: The authors highlighted four difficulties (written in the summary part) and proposed the approach to address these difficulties. However, it was not clear from the paper how they are addressed. Item (i) seems to be addressed by the meta-learning of the hyper-parameter. However, as I wrote above, it was not clear why this makes sense. I couldn't find discussion related to Item (ii) and (iii). The experiments are not designed to evaluate these points. 
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow in general. However, because the algorithm is written only by the natural language, it is hard to understand precisely. I see the novelty in the proposed framework including hyper-parameter learning mechanism. However, the approach is relatively a straight-forward extension of existing MAP-ELITES approaches. The experimental details are provided in the appendix. 
summary_of_the_review: I see the advantages of the proposed approach. However, because of the lack of the discussion of the algorithm validity and (empirical) analysis of the algorithmic behavior, I think that the claims are not sufficiently supported in the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a quality-diversity algorithm for reinforcement learning that enables both the RL optimization step and population repertoire update step to make changes to agent parameters. The proposed algorithm allows updates to agent hyperparameters to reduce the sensitivity of the approach to hyperparameter choice of the underlying RL algorithm. Scaling is achieved by allowing parallel evolution of agent populations and a JAX implementation for high performance. Results show strong performance (matching competitive alternatives) on robotic control tasks, particularly in terms of deceptive environments and when evaluated not only on final performance but also coverage of strategies.
strength_and_weaknesses: ## Strengths

1. Comparable performance to existing algorithms while being able to solve hard exploration tasks (HumanoidTrap, AntTrap) fast (in terms of environment steps).
2. Generality. The approach presented can readily be coupled with many RL algorithms and directly addresses hyperparameter sensitivity issues in those algorithms. It would be interesting to see results showing how this helps compared to baselines on other RL tasks.


## Weaknesses

1. Few major performance improvements over alternatives. The results would be stronger if there were clear environments that previous algorithms simply failed to solve that PBT-MAP-Elites solves. The closest result is the HumanoidTrap result. This is not a major weakness.
2. No data on scaling. One of the main benefits of the approach is that it "3) can scale to large population sizes", but this is never tested. Adding experiments showing this scaling and how it compares to alternatives would greatly benefit the paper.
3. No data on (wallclock) speedup. Can the speedup of running experiments in JAX be quantified? The JAX implementation is claimed as being efficient, but no empirical evidence backs this claim compared to other simulators.


## Feedback & Questions
- Figure 2: The colors for ME-ES and PBT are hard to distinguish, particularly when shaded areas overlap the medians. Consider slightly different colors for readability. Also it would help to order the columns to have all "*Uni" columns and "*Trap" columns grouped together.
- Figure 3: The axis text is too small to read. Please make it larger.
- Note: At least two references are duplicated: Pierrot 2022a & 2022b, Lim 2022a & 2022b.
- What is the speedup of using the JAX implementation compared to alternatives?
- The introduction and related work are both quite lengthy and cover overlapping material. Consider condensing the introduction to highlight the main contributions and novelty, with the related work discussing the particular limitations of past efforts. This can help keep the narrative clear for readers and focus attention on the specific novelty of the work reported.
clarity,_quality,_novelty_and_reproducibility: - Clarity: Overall clear. The text is readable, but awkwardly structured: long discussions of literature occupy the first 3 pages of the text before introducing the core problem and contributions.
- Quality: Good. The results show reasonable performance and success on hard exploration tasks. The experiments demonstrate rigor for fair comparison.
- Originality: Modest. The novelty in the work lies in the design decisions made for representing policies (by including hyperparameters, replay buffers, &c. in agents) and allowing two processes to modify policies.
- Reproducibility: Modest (possibly high?). The appendix provides algorithm details and the text is clear about the major experiment and algorithm design features. No code is provided for the implementation, but the text describes open sourcing the implementation so this may be forthcoming.
summary_of_the_review: The technical novelty of the paper is modifying MAP-Elites to use a PBT-like structure for the agent population. This provides the benefit of generality to RL implementation and automatic tuning of hyperparameters (inherited from PBT). This is certainly new, though incremental in the sense that both PBT and MAP-Elites have been thoroughly explored in the past. There is substantial benefit to be had from further investigation of this kind of combination of RL meta-algorithms (like PBT and self-play) with evolutionary approaches, particularly from quality diversity.

Empirical results show some strong results in the set of tasks the algorithms are tailored to: deceptive domains. Most tasks show competitive performance with alternatives. All results are in relatively simple robotic control tasks (compared to controlling robotic manipulators or real robot locomotion), leaving plenty of room for stronger tests of the algorithm capabilities. No empirical results examine the speedup claims of the JAX implementation or the scalability claims for the algorithm, though both seem very plausible.

Taken together the work makes a step forward for quality-diversity approaches to RL in deceptive continuous control tasks. Whether this approach has wider value remains an open question.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new variation on Map-Elites, PBT-MAP-ELITES, and shows that it works well for finding sets of policies in reasonably high-dimensional spaces. The new algorithm is quite simple; the actual contributions are in the selection and update, and the algorithm is oblivious as to which RL algorithm to use as underlying algorithm. While the novelty is not overwhelming, I think this is a nice and useful enough framework. The results are also good, narrowly outperforming all competitors at several standard benchmarks (although I would have liked to see a comparison to Tjanaka's DQD-RL).
justification_for_why_not_higher_score: The algorithm is not novel enough for a spotlight.
justification_for_why_not_lower_score: Good paper, well written, simple algorithm with some novelty, good results.
Decision: Accept: poster