Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes and integrates the usage of Quality-Diversity methods for skill discovery in RL. While prior methods largely rely on maximum-information (MI) RL, this paper makes a case for QD as an alternative approach when used with fast environment simulators. The authors provide a large number of experiments which show that QD-based methods are competitive with the MI RL approach and should be researched in more dept as future work.
strength_and_weaknesses: ## Strengths

- The paper takes on an exciting direction, using QD for skill discovery. While the MI-RL line of work has shown great results in recent years, QD methods can provide an alternative, given the diversity element that is being explicitly optimized for.
- I like how the paper positions itself neutrally to both types of approaches, namely QD and MI-RL. After presenting all the experimental results, it comes to the conclusion that no method is optimal for all environments and lays down avenues for future work. It also accepts the pros and cons of both approaches, and analyses all methods without any biases, as far as I could tell.
- The paper includes extensive empirical evaluations and analysis, including experiments directly evaluating diversity, few-shot adaption and hierarchical learning. The list of baselines used in the experiments is fair, as far as I’m concerned.

## Weaknesses

- One weakness of the work is that it always provides the median of the metric of interest over the seeds (e.g. Table 1, Figure 4). This can provide unfair results if there are outliers. I suggest that the authors provide several metrics of interest, such as mean, IQM, etc. [rliable]([https://github.com/google-research/rliable](https://github.com/google-research/rliable)) is a great tool for this.
- I didn’t quite understand why haven’t the authors include rewardless DIAYN and DADS results in their experiments. Throughout Section 5, they only use DIAYN/DADS+REWARDS as baselines.
- QD-based methods are obviously more expensive computationally, given they store separate weights for each policy/skill. The authors argue that this is okay if the simulator is very fast. I have no arguments against it, but it would be great to know how much more compute-heavy QD-based methods are. The authors only mention that everything is within 2 hours of training, but there can be a big difference within this range.
- Minor: please use the correct citation styles of prior work. See the ICLR instruction template for information on where to use \citep and \citet.

Note: My score is not final and can change (in either direction) based on the authors' responses or comments of other reviewers.

# Post rebuttal update

I have increased my score based on the responses from the authors.
clarity,_quality,_novelty_and_reproducibility: Clarity: High. The paper is clear and easy to follow. The prior and related work is properly cited and helps the reader to understand the manuscript faster.

Quality: High. The paper includes extensive experiments and a thorough analysis of prior work as well as new techniques that use QD for skill discovery.

Novelty: High. While QD and skill discovery are mature fields at this point, this paper is the first attempt for using QD for skill discovery, as far as I’m concerned.

Reproducibility: High. The code is included alongside the submission. I have no worries regarding the reproducibility.
summary_of_the_review: Interesting work that combines QD and skill discovery in RL. Extensive experiments and thorough analysis of prior work are included in the manuscript. I have raised several questions and some concerns that I’d like to be answered by the authors during the rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides a comparison and benchmark of Evolutionary strategies and rl based skill discovery approaches. The authors provide a thorough literature review on the most recent approach in both lines of research and compare them in light of different metrics, how they generalise to changing environments (adaptive gravity, motor failure and different target locations), and how they can be use in hierarchical planning.
strength_and_weaknesses: Strengths:This paper provides a comprehensive and important benchmark connecting RL based skill discovery with Quality Diversity approaches. It provides three settings for comparison using metrics, using few shot generalisation on adaptive environments, and using primitives for hierarchical planning. 
Weaknesses:
The metrics provided in Table 1 and figure 2 seem to be biased towards QD methods since RL based on MI does not naturally have or optimise for QD score, can the authors provide additional metrics that are perhaps less bias?
In the hierarchical learning experiments can the authors clarify the number of seeds used? And are there other environments that can be used for this setting?

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and provides an interesting unified perspective on how both lines of research intersect.
summary_of_the_review: I recommend this paper to be accepted since it provides a comparison of approaches in MI RL and quality diversity approaches for skill discovery. They empirically analyse how different methods on each side fare in terms of metrics for diversity and in terms of few shot generalisation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper compares RL methods with information theory inspired diversity rewards and quality diversity methods for skill discovery on a large set of environments implemented in jax/brax (and thus able to take advantage of accelerators). It compares methods in terms of diversity metrics, performance on adaptation tasks and usefulness for hierarchical planning and finds that QD methods are competitive and sometimes perform better and potentially less sensitive to hyperparameters. 
strength_and_weaknesses: Strengths:
* extensive careful empirical evaluation with interesting results

Weaknesses:
* comparing with the same time budget introduces dependencies on hardware, and implementation quality that are difficult to assess. I would at least report roughly how many environment interactions each methods processes to give readers an indication
- Related to the first point I’m unclear why RL methods are less able to leverage accelerators (i.e. more env interactions). Perhaps the authors could provide their intuitions on this point. Is this because recent methods have been developed/tuned in a different regime?
- Unclear if any of the methods in this paper could scale to something that seems less toy (e.g. dexterous manipulation with a robotic hand or humanoid locomotion)
- I appreciate that the paper is has a lot results but having a limited understanding of QD methods I might have benefitted from more details.
- Consider citing Hausman et al. (https://openreview.net/forum?id=rk07ZXZRb) which is pretty similar to DIAYN + reward (and predates DIAYN).
- Figures are all a bit small and hard to read. 

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, although in parts a little more context/details would be good. Of course this is hard to balance with the page limit.

The empirical comparison seems quite thorough, and I am not aware of a similar comparison in the literature. This should be of interest to the ICLR community. 

Reproducing this work would be a substantial undertaking but should be achievable with the submitted code.
summary_of_the_review: Nice paper presenting an empirical comparison between RL-based skill discovery methods and QD methods. I think the paper can be improved in some ways but presents interesting empirical results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: At a high level, this paper is an empirical study of a collection of RL algorithms over a collection of environments.
The algorithms are reimplemented versions of:
* Multiple Quality Diversity (QD) methods, which are evolutionary and do not use exact backpropagation.
* Information Theory-Augmented RL methods, which involve adding additional mutual information terms + latent vectors inside a gradient-based RL pipeline.

The environments consist of:
* Unidirectional (Single task) 
* Omnidirectional (Multi-task)
* Adaptation
* Additional Exploration (Trap/Maze/etc)

all of which are based on Continuous Control. 


strength_and_weaknesses: # Strengths
* Comprehensive algorithm list and benchmarking. While my knowledge on the diversity methods is limited, it seems that the provided open source codebase could be a useful tool for future research, due to its extensiveness. From a glance at the Appendix, there also seem to be many ablation studies.
* The overarching goal (i.e. analyze different quality diversity methods and how useful they are) is somewhat solid. The main issue is with its execution, which I provide below.

# Weaknesses
* Lack of a **main punchline**: The current paper acts as an empirical study across previously invented algorithms, rather than inventing an method of its own. This is completely fine, but such a paper's primary focus will then be to provide deep experimental conclusions unknown previously. Currently, the experimental conclusions are not very insightful. A few examples: 
    * Section 6: "Our benchmarks however show that no single method outperforms all others, which opens the door to promising work directions" is too vague and can be said of any method in any field.
    * Section 5.2: "We observe that the performance of DADS+REWARD is very sensitive to the exact choice of hyperparameters" - Many RL algorithms are sensitive to hyperparameters; why is this an interesting observation?
    * Section 3: Are any of the methods mentioned here similar or different at a deeper, mathematical level? Currently all of the methods described use mostly words, which makes it hard to determine what components contribute to which behaviors (e.g. without needing to check references, how is MAP-ELITES different from PGA-MAP-ELITES?). It would be far more effective if the paper discussed the mathematical components which are common to certain algorithms, and show experimentally if these components contribute to certain behaviors.

* Factual/Empirical Inaccuracies
    * Section 5.1 "These [standard locomotion] tasks are challenging for evolutionary algorithms that do not leverage backpropagation through the neural policies" - The word "challenging" is too harsh here. ARS [1] has been consistently shown to beat PPO in standard locomotion tasks for a variety of reasons, such as the use of deterministic, linear policies.
    * In Table 4 of the Appendix, I see that a MLP of layer sizes (256, 256) was used. Was this used for all evolutionary methods? The issue is that the performance of evolutionary methods are greatly affected by the problem dimension (i.e. parameter count in this case). It's already been shown that linear policies suffice and are actually sometimes the best [1] for many continuous control tasks. Since the difference in parameter count between a linear policy (order of 100's) and a MLP (20K+) is huge, this can be a very big confounding factor.


[1] Simple random search provides a competitive approach to reinforcement learning (https://arxiv.org/abs/1803.07055)
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper could definitely improve on its formalism of the compared algorithms, with e.g. more mathematical notation. Currently the descriptions of certain methods (e.g. MAP-ELITES) is purely with words and references, which makes it very difficult to know the differences between such algorithms (please see "Weaknesses" above as well).

Quality: The paper appears to be quite rigorous in its experimental study and has multiple ablation studies, both in the main body and ablations.

Novelty: As mentioned above in "Weaknesses", the core issue is that the paper lacks a "punchline" or a compelling purpose, due to its unconvincing analysis.

Reproducibility: The code appears to be quite clean and organized, no issues here.
summary_of_the_review: I currently lean reject, because of the lack of a compelling purpose or analysis. Currently as written, from a devil's advocate point of view, the paper can easily appear to be a collection of somewhat random algorithms applied on a collection of random benchmarks - the paper needs to have stronger analysis and focus on a compelling purpose for providing these experimental results.

EDIT: I have raised my score to a 5 in light of the authors' rebuttals. I think the writing currently is still not convincing enough to a non-expert such as myself, and I urge the authors to improve on this front.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers agreed that this comprehensive empirical study comparing two classes of methods (information-theory-inspired skill discovery in RL and quality diversity from the neuroevolution community) is very valuable to the field. This is relevant not only in terms of performance itself but also in terms of defining a common ground, cross-evaluation across environments designed for both types of methods, and so on.

There is already enough empirical data in this paper, so no additional experiments are needed. That being said, reviewers pointed out the writing could be improved. Multiple recommendations were given, such as more concisely summarizing the take-home message of the paper. I urge the authors to go over the reviewers’ recommendations and incorporate as much as you can in the camera ready version of the paper.

justification_for_why_not_higher_score: The paper does not necessarily have a groundbreaking result nor it has a clear take-home message. I would expect that for a higher score. The main virtue of the paper is a careful empirical analysis of two types of methods.
justification_for_why_not_lower_score: All reviewers agreed upon the value of the paper, even the reviewer who gave it weak reject acknowledged that the paper makes interesting contributions but the writing should be improved, which can be done in the camera-ready version of the paper.

I'm recommending a spotlight talk because it bridges different fields in ML and the paper will benefit from additional visibility.
Decision: Accept: notable-top-25%