Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a variant of an already existing algorithm CMA-ME for Qualty-Diversity (QD) problems by introducing a learning rate based annealing function.
strength_and_weaknesses: Strength:

1) The paper presents a systematic study with adequate empirical experiments.

Weakness:

1) Significance and relevance of QD problem in machine learning along with some warm-up examples should have been given in the introduction.

2) The simple algorithmic change in CMA-ME is seemingly not significant enough to warrant a publication in ICLR. The theoretical proofs seem quite standard and straightforward.

3) For Figure 3, why are you plotting the QD score of different algorithms against number of iterations? WOn't it be such that these different algorithms will perform different amounts of jobs in their inner loops and hence, their "iterations" will not consume same amount of time?

4) The proposal was not adequatey validated through comparison against evolutionary algorithms of different genre than CMA-ES, like DE for QD problems. Also the multi-objective optimization approaches like the following one should have been compared:

Thomas Pierrot, Guillaume Richard, Karim Beguir, and Antoine Cully. 2022. Multi-objective quality diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '22). Association for Computing Machinery, New York, NY, USA, 139–147. https://doi.org/10.1145/3512290.3528823

5) Why the parametric ANOVA test was used instead of non-parametric statistical tests?

6) Despite focusing on derivative-free QD algorithms, did you really handle a non-dfferentiable function like Weirstrass? Or I am missing something here?


clarity,_quality,_novelty_and_reproducibility: Given the earlier works on CMA-ME and QD problems, the novelty seems less than expected for ICLR. The linguistic quality of the paper needs very significant improvement. 

The reproducibility aspect is appreciable, the authors released detailed source codes. 
summary_of_the_review: The paper presents an interesting study with appreciable set of experiments. However, the novelty and innovative content seem somewhat less for a conference like ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims at improving CMA-ME, a quality-diversity (QD) algorithm. The authors focus on the way that the CMA-ME ranks candidate solutions. The authors hypothesize that the rapid change of the fitness threshold value, denoted by t_e, is the cause of three limitation highlighted in the QD community: prematurely abandoning the objective, struggling to explore flat objectives, and having poor performance for low-resolution archives. The proposed approach changes only the way to update the threshold: introducing a damping factor. 

The proposed approach, CMA-MAE, is compared with CMA-ME, MAP-Elites (line) and MAP-Elites on 5 test problems. Higher QD-score and coverage are observed for all test problems.


strength_and_weaknesses: # Strength

* A minimal change to the existing approach, CMA-ME, has been proposed, but a statistically significant improvement has been observed in the benchmark problems.

# Weaknesses

Technical contribution: A simple approach is nice. However, at the same time, the proposal is incremental and its difference is minimal. 

Test problem choice: It has not been discussed why these 5 test problems are selected. Because the selection of test problems is a very important part of the experimental design, this must be carefully explained. In this work, why are these 5 test problems are sufficient to reveal the goodness and limitation of the proposed approaches? 

Discussion on the significance: I understand that the results are statistically significant. However, because I am not very familiar with QD community, it is hard to see whether this improvement is meaningful. A discussion on how meaningful these improvements are would improve the readers' understanding, especially for those who are not familiar with QD, which I guess the most audiences in this conference are.

Significance of this topic: Because the test problems are selected from the existing work where differentiability is assumed, a possible application of the proposed approach is not clear.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow. Because of the proposal is incremental and the change is minimal, its novelty is limited. The experimental details are provided.
summary_of_the_review: It is hard to judge this paper. The technical contribution is minimal. However, this minimal change leads to a statistically significant improvement over the existing approaches. It is not clear how it contributes to ML communities. Probably because of the lack of my knowledge in QD, the significance of the proposed approach may be underestimated. I would currently suggest weak reject, but it may be increased if the authors response convinces me.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces a new quality diversity algorithm, CMA-MAE, that try to address some limitations of the baseline algorithm CMA-ME. The authors introduce a learning rate $\alpha$ into the acceptance threshold, which encourages CMA-MAE to spend more time on the promising zones before transitioning to exploration. The performance of the proposed algorithm is evaluated in three QD benchmark domains and the results show that the proposed algorithm outperforms state-of-the-art QD algorithms.
strength_and_weaknesses: Strength

- The paper is overall well written and easy to understand.
- The experimental results show that the proposed method is robust and superior to other QD algorithms.

Weaknesses

- The proposed method is very incremental. Compared with CMA-ME, it just adds the smoothing factor to update the acceptance threshold. 
- The experiments are not very convincing. The authors claimed that the modification allows CMA-MAE to spend more time on the promising zones before transitioning to exploration, thus can make CMA-MAE do better on domains with objectives that are hard to optimize as well as domains with flat objective functions. However, in the experiments, in addition to the common benchmarks, the authors only tested the methods on a simple function with the flat property, which makes the results not very convincing. I suggest the authors to conduct experiments on more problems with the mentioned properties.
- More ablation studies are needed. The performance of CMA-MAE depends on not only the setting of $\alpha$ but also that of $min_f$. What is the impact of the setting of $min_f$ on the performance?
- There are some minor problems, for example, in the section on limitations and future work, the order of the two paragraphs may be wrong.
clarity,_quality,_novelty_and_reproducibility: The clarity, quality and reproducibility are good, but the novelty is very limited.
summary_of_the_review: The paper is overall well written and easy to understand. However, the proposed method is very incremental compared with the baseline algorithm, and more convincing experiments are needed to demonstrate the advantages of the method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The main contribution of this work lies in proposing a variant of the known CMA-ME algorithm for quality diversity problems by introducing a learning rate into the acceptance threshold, so as to overcome the limitations of CMA-ME.

(A) While the empirical results presented in this paper show the better performance of their proposed modification, the reviewers all agree that the contribution (in particular, the algorithmic innovation and proofs of theoretical results) is incremental, even after reading the authors' rebuttal.

Besides the reviewers' suggestions for improvement, the authors can consider the following additional suggestions: 

(1) If the authors think that the challenge lies in proving the theoretical results, they are advised to reorganize their paper to focus on explicitly explaining and highlighting the nontrivial challenges with proving the results. For example, are there important intermediate results that are not straightforward to be proven? 

(2) The authors can consider whether the more complex algorithm with the same performance benefits leads to further insights not gleaned through the simple modification in this paper. If so, the authors can prove their equivalence and show the additional insights entailed by the more complex algorithm. 
justification_for_why_not_higher_score: See the above-mentioned major concern (A).
justification_for_why_not_lower_score: N/A.
Decision: Reject