Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a self-attention based meta learning framework for the learned Gaussian ESs and tests the performance on neuroevolution tasks. The metaBBO module includes steps like Meta-sampling, Inner loop search, meta normalize, and Meta updating. Limited evaluation results on standard datasets were provided. 

strength_and_weaknesses: Strength: The paper addresses a timely and ambitious research topic. 

Weakness: 

Statements like "We show that metaevolving this system on a small set of representative low-dimensional analytic optimization problems is sufficient to discover new evolution strategies capable of generalizing to unseen optimization problems, population sizes and optimization horizons"  - are very strong and have not been at all supported through any theoretical treatment. The generalization of LES to unseen neuroevolution tasks even in disparately high dimensions is not explained via theory of insightful empirical evidences. 

Related works ignored:

Vishnu TV, Pankaj Malhotra, Jyoti Narwariya, Lovekesh Vig, and Gautam Shroff. 2019. Meta-Learning for Black-Box Optimization. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16–20, 2019, Proceedings, Part II. Springer-Verlag, Berlin, Heidelberg, 366–381. https://doi.org/10.1007/978-3-030-46147-8_22

Choice of the functions in table A.1 for training is not adequately justified by analysis of the funcional landscape features.

It is not clear how the method achieves diversity in meta-training task distributions, no theoretical treatment was included. 

The results were not compared against the best known algorithms in the pertinent areas from other genre. 

Why were the results not validated through appropriate non-parametric statistical tests?
clarity,_quality,_novelty_and_reproducibility: The paper is more or less clearly written.

The technical quality and originality (including absence of theoretical treatments) are not up to the expected level of a conference like ICLR. 
summary_of_the_review: A meta learning framework is extended for Gaussian ESs without very exhaustive set of experiments and any theoretical results. In my view, the paper does not meet the high standards expected for ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a method to learn the parametrization of an Evolution Strategy through the meta-training of a self-attention-based architecture. They demonstrate the generalization of their meta-training procedure to test tasks that are different from the meta-training tasks. They further analyze the limitations of their proposed algorithm and deduce an interpretable parametrization of the ES.

The approach uses a self-attention mechanism to learn the parameters of an ES as well as an LSTM architecture to learn the learning rate of the ES parameter updates from one generation to another. Their approach is by construction invariant to reordering. It also allows them to meta-train on multiple dimensions, which can then be leveraged at test time.
strength_and_weaknesses: Strengths: 
The paper is easy to follow, which is excellent for a meta-training paper, with limited jargon and many implementation details. The strengths of the paper don’t lie in the idea itself but rather in the study of its extensions and limitations. This is a well-studied idea, and the experiments illustrate well the points the authors try to make, should it be about interpreting the LES or self-referencing of the LES. In particular, the study of the meta-training task distribution is very insightful and has not been done in many meta-learning papers. Noticeably, the authors can interpret the LES to devise a Discovered ES that works in some of their experiments.

Weaknesses: 

Some weaknesses are outlined by the paper itself, such as scaling the self-attention mechanism or that this algorithm works only for diagonal Gaussian ES.  It is worth pointing out also that the features used in the self-attention mechanism F_t are manually created, i.e. it is a choice made in the algorithm that the self-attention would consider these only three statistics derived from fitness. This works in practice, as shown in experiments but looks like a significant limitation in terms of generalization. The authors mention this as a limit to their approach, but the manual engineering of these features is another.

Questions: 
- Would it be possible to replace the manually engineered features F_t with a learned feature map? 
- Would it be possible to relax the diagonal assumption of the Gaussian covariance matrix but use, for instance, a sparsity regulator to keep it maybe sparser than full? 
- In the experiments, the authors show that the method can adapt to scenarios unseen in the meta-training task distribution and generalize to higher dimensions and different loss surfaces. However, there are still, of course, tasks in which it performs poorly. Of course, one cannot expect a meta-model to perform perfectly on all tasks, and any such pretrained model will suffer from distributional shift, and these cases call for a fine-tuning procedure. How could this approach be fine-tuned, and how long would it take to adapt it for instance, to a task on which it performs poorly straight out of meta-training?
- How do you explain the difference in Brax Performance in Figure 3 bottom middle and right between the medium budget and large budget on tasks like hopper and ur5e or walker2d? Otherwise, the experiments are well-designed and conclusive. The ablation studies are particularly insightful.
clarity,_quality,_novelty_and_reproducibility: Please see above. 
summary_of_the_review: Please see above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The update rules of Black-Box Optimization (BBO) methods are traditionally formalized by hand-craft heuristics, which is inflexible. To tackle this problem, this paper proposes to parameterize the update rules of a typical BBO algorithm, Evolution Strategy (ES), by a transformer, and use meta-learning together with BBO techniques to optimize it, obtaining the Learned Evolution Strategy (LES). Experiments on BBOB benchmark, classification on MNIST and continuous control tasks generally show the effectiveness of the proposed framework. Furthermore, this paper shows that the proposed LES can be optimized through a self-referential style.
strength_and_weaknesses: Strength:

This paper is well-written and easy to follow. Experiments show the effectiveness of the proposed method from various views. 

Weakness:

1. Even that the learning of update rules in ES has not been studied in a targeted way, the general idea of using learning techniques to replace the heuristic components in optimization methods is not new [1], which limits the novelty of this paper. I did not find much new insight from this work. There have been many works to learn hyper-parameters of some complex optimization methods.

2. The authors should clearly explain and show the advantage of the proposed method (i.e., using a transformer to represent the update rule of ES and applying again BBO (e.g., CMA-ES) to optimize the parameters of the transformer) over other general ways of learning the heuristic rules. Can you show the benefit of using transformer empirically? How about the cost of the proposed method, i.e., training the transformer? As mentioned in the paper, the self-attention mechanism scales quadratically in the number of population members (in section 8). Meanwhile, it seems that a relatively large population size is important (in figure 4), which may limit the application of the proposed method.

3. More advanced ES methods besides OpenES and SepCMA like [2-4] are expected for comparison to show that the learning of the update rules can obtain a competitive performance w.r.t. SOTA ES methods.

[1] Marcin Andrychowicz et al, Learning to Learn by Gradient Descent by Gradient Descent, NIPS’16.
[2] Guided evolutionary strategies: Augmenting random search with surrogate gradients. ICML’19
[3] From complexity to simplicity: Adaptive ES-active subspaces for blackbox optimization. NeurIPS’19
[4] Self-guided evolution strategy with historical estimated gradients, IJCAI’20.

clarity,_quality,_novelty_and_reproducibility: This is a well-organized paper with high quality. However, as the general idea of this paper is using learning techniques to replace the heuristic components in optimization methods, which is not new, the novelty of this paper is limited. The experiments in this work are well-motivated with good introductions, figures and explanations, but not convincing (please see my comment in the above “weakness” part). The reproducibility looks good to me.
summary_of_the_review: This paper proposes a new method to learn the update rules of ES, with relatively various experiments to show its effectiveness. However, the general idea of this paper, using learning techniques to replace the heuristic components in optimization methods, is not new, but another standard implementation (here the authors use transformer to represent the rules and optimize the parameters of transformer by CMA-ES). As listed in the weakness, the lack of comparison with advanced ES methods and the scalability issues also limit the contribution of this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a method for Evolutionary Search (ES) that works by sampling hyperparameters for Learned Evolutionary strategies (LES) over different tasks and then uses self-attention to compute the recombination weights. The authors compare this approach to Handcrafted ES baselines (openES, SNES and sep-CMA-ES), and ablate their method to study the impact of the different components of Neural Networks. Based on this ablation, they derive a heuristic which is incorporated into an algorithm called Discovered Evolutionary Strategy (DES). Finally, the authors bootstrap LES using LES (argued as self-reference).
strength_and_weaknesses: ### Strengths:
- The approach of using self-referential mechanisms for evolutionary optimization is interesting and seems to lend itself naturally to the ways in which evolutionary methods perform search and exploration.
- The experiments have gone in interesting directions, and have been shown n a clear and concise manner
- The literature has been covered sufficiently

### Weaknesses:
- Set transformers seem to be the differentiating factor here. It is not clear how the proposed method explicitly relates to Set Transformers, except for the invariance property being leveraged here. Is that a sufficient condition for any method that uses attention to be considered a set transformer? I would propose explicating this in a more self-contained manner 
- The central question seems to be: Can an end-end method perform better than some other methods, static and dynamic, in optimizing black box functions? The comparison has been performed only on static baselines (NES replaces points with distributions, openES uses ES for control problems and RL, and sep-CMA-ES uses diagonalization). None of the baselines are dynamically tuning hyperparameters of the inner loop (something that the BB method does). An example of potential baseline in this direction could be Dynamic Algorithm Configuration (DAC) (https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/20-PPSN-LTO-CMA.pdf). In general, having a dynamic baseline would make the experimentsstronger.
- The rationale for using the LSTM for adjusting learning rates is not clearly explained, Since we see that  fixed LR ends up performing as well as the dynamic ones. Additionally, since we see that learning rates learned by an LSTM are not having a significant impact when compared with fixed ones, what I am not sure of is whether a learning rate schedule policy would perform better or not (https://arxiv.org/pdf/2007.04223.pdf).   
- The selection of hyperparameters : T=50, N=16, D, z , e.t.c - is not explained. How did the authors decide these to be the relevant ones, and is the method sensitive to these values? Considering the argument for an end-end method, I think this question becomes more relevant

clarity,_quality,_novelty_and_reproducibility: ### Clarity

The paper has been written in a relatively clear manner. However, I feel the writing can be a bit more self-contained. While the referenced publications do contain the material, I feel that briefly mentioning the core idea behind key concepts: ‘set transformers, pegasus trick, baselines, e.t.c would help a broader set of readers in grasping and critically understanding the approach

### Novelty

The work seems to be novel and the studies done through DES and other ablations provide clarity over what works and what doesn’t.

### Quality

The limitations have been briefly discussed and addressed in the conclusion as well

### Reproducibility

The compute resources on which the experiments have been run are not mentioned in the main paper or the appendix. Considering this is an ES-based strategy, I would highly recommend mentioning this. 

summary_of_the_review: The idea is novel and interesting and I definitely see the relevance to the meta-learning community. Evolutionary strategies have always been an interesting direction to gradient-based optimization, and so, I feel the research direction of end-end meta-optimization in a self-referential manner is interesting, to say the least. However, I feel the work is not yet ready for publication. This is primarily due to missing stronger baselines. I think the rationale for using a self-referential loop i.e. why should we use implicit models to begin with needs to be sufficiently justified with comparisons against static and dynamic baselines and that is currently lacking from the work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the black-box optimization and presents a method for discovering effective update rules for evolution strategy via meta-learning. The main idea is to parameterize a search strategy by self-attention-based architecture such that the update rule is invariant to the ordering of the candidate solutions. The heuristic components in the Gaussian ES is replaced by a meta-learning technique. This might be a general idea of meta-learning but the current work formulates it in a proper way and demonstrates its validity. Initial reviews pointed out several weaknesses and limitations, where details can be found in individual comments. Authors did an excellent job to resolve most of concerns in their rebuttal. During the discussion period, most of reviewers were satisfied with the author response, increasing the overall score. Some weaknesses, which are also outlined in the paper, includes the scaling self-attention mechanism, manually engineered features F_t, diagonal Gaussian ES. Nonetheless, most of reviewers agree that the paper contains interesting contributions, which deserve to be published. Last minute, I had a personal discussion with the reviewer who pointed out critical comments on lack of theoretical treatments and achieving diversity in meta-task distributions. With these extensive discussions with reviewers and thanks to the authors’ effort in responding to each comment, all reviewers finally agree to champion this paper. 

justification_for_why_not_higher_score: This is a nice use of meta-learning where the heuristic components in the optimization are replaced by a learning technique. However, there are a few inherent limitations, which are also outlined in the paper. Thus, the poster presentation is a right place.
justification_for_why_not_lower_score: The authors did an excellent job in responding to individual comments. Most of reviewers were satisfied with the author response, increasing the overall score.
Decision: Accept: poster