Discovering Evolution Strategies via Meta-Black-Box Optimization
Authors: Robert Tjarko Lange
Tom Schaul
Yutian Chen
Tom Zahavy
Valentin Dalibard
Chris Lu
Satinder Singh
Sebastian Flennerhag
Venue: ICLR 2023 poster (2023)
Keywords: Meta-Learning
Evolution Strategies
Gradient-Free Optimization

Abstract: Optimizing functions without access to gradients is the remit of black-box meth- ods such as evolution strategies. While highly general, their learning dynamics are often times heuristic and inflexible — exactly the limitations that meta-learning can address. Hence, we propose to discover effective update rules for evolution strategies via meta-learning. Concretely, our approach employs a search strategy parametrized by a self-attention-based architecture, which guarantees the update rule is invariant to the ordering of the candidate solutions. We show that meta-evolving this system on a small set of representative low-dimensional analytic optimization problems is sufficient to discover new evolution strategies capable of generalizing to unseen optimization problems, population sizes and optimization horizons. Furthermore, the same learned evolution strategy can outperform established neuroevolution baselines on supervised and continuous control tasks. As additional contributions, we ablate the individual neural network components of our method; reverse engineer the learned strategy into an explicit heuristic form, which remains highly competitive; and show that it is possible to self-referentially train an evolution strategy from scratch, with the learned update rule used to drive the outer meta-learning loop.

Published as a conference paper at ICLR 2023
DISCOVERING EVOLUTION STRATEGIES VIA META-BLACK-BOL OPTIMIZATION
Robert Tjarko Lange Technical University Berlin
Tom Schaul & Yutian Chen & Tom Zahavy DeepMind
Valentin Dallibard DeepMind
Chris Lu" University of Oxford
Satinder Singh & Sebastian Flennerhag DeepMind
ABSTRACT
Optimizing functions without access gradients is the remit of black-box meth- ods such as evolution strategies. While highly general, their learning dynamics are often times heuristic and inflexible exactly the limitations that meta-learning can address. Hence, we propose to discover effective update rules for evolution strategies via meta-learning. Concretely, our approach employs a search strategy architecture, which guarantees the update
parametrized by sulf-attention-batt rule is invariant to the ordering of the candidate solutions. We show that meta- evolving this system on small set of representative low-dimensional analytic optimization problems is sufficient discover new evolution strategies capable of generalizing to unseen optimization problems, population sizes and optimization horizons. Furthermore, the same learned evolution strategy can outperform estab- lished neuroevolution baselines supervised and continuous control tasks. As additional contributions, we ablate the individual neural network components method; reverse engineer the learned strategy into explicit heuristic form, which remains highly competitive; and show that it is possible to self-referentially train evolution strategy from scratch, with the learned update rule used drive the outer meta-learning loop.
INTRODUCTION
Black-box optimization (BBO) methods are those general enough for the optimization of functions without access to gradient evaluations. Recently, BBO methods have shown competitive perfor- mance to gradient-based optimization, namely of control policies (Salimans al. 2017 Such al., 2017 Lee et al.] 2022). Evolution Strategies (ES) are a class of BBO that iteratively refines the sufficient statistics of (typically Gaussian) sampling distribution, based on the function evalua- tions (or fitness) of sampled candidates (population members). Their update rule is traditionally formalized by equations based on first principles (Wierstra et al. 2014, Ollivier et al., 2017), but the resulting specification is inflexible. On the other hand, the evolutionary algorithms community has proposed numerous variants of BBO, derived from very different metaphors, some of which have been shown to be equivalent (Weyland 2010). One way to attain flexibility without having to hand-craft heuristics is to learn the update rules of BBO algorithms from data, in a way that makes take: We meta-learn neural network
them more adaptive and scalable. This is the approach parametrization of BBO update rule, on a set of representative task families, while leveraging modern accelerators, building recent devel-
evaluation parallelism of different BBO instances opments in learned optimization (e.g. Metz et al. (022). This procedure discovers novel black-box optimization methods meta-black-bo optimization, and is abbreviated by MetaBBO. Here, we investigate one particular instance of MetaBBO and leverage it to discover a learned evolution strat- egy (LES) The conerete LES architecture can be viewed as a minimal Set Transformer (Lee al., 2019), which naturally enforces an update rule that is invariant to the ordering of candidate solutions within batch of black-box evaluations. After meta- meta-training, LES has learned to flexibly interpo- late between copying the best-performing candidate solution (hill-climbing) and successive moving average updating (finite difference gradients). Our contributions are summarized as follows:
Work done during an internship at DeepMind. Contact: robert. .t. Ceaerr....aaaee wan.eentnnbeerrreeee de.
Pronounced 'meta-boh' and 'less".

---

Published as a conference paper at ICLR 2023
Performance vs. Baselines
Sample tasks
Sample & evaluate ~ M(m, r)
Rel. Brax Performance
1.25
1.00
0.75
Fitness properties
0.50
Sample LES parameters
0.25
0.00
Meta-learned attf-tttenionn
-0.25
ASEBC
Generations
0
0
500
1000
1500
2000
Q, K, V (F)
Generations (x256 Evais)
Generalization Across Populations
QKT
Run search
softmax
/Dx
Norm. Performance
0.8
softmax (8) RV
00.4
Normalize
Search dist. update
meta-update
Am Ao
0.2
Meta-Train Population
Outer Loop: MetaBBO
Inner Loop: LES
100
200
300
400
500
Population Size
Figure 1: Overview diagram of MetaBBO, LES & performance on continuous control tasks. Left:: MetaBBO. The outer loop samples set of inner loop tasks (uniformly), and set of candidate LES parameters {0, ) from a meta-ES (here: CMA-ES). After obtaining a normalized meta- fitness score of each LES instance, the meta-ES is updated and iterate the meta-optimization process. Middle: LES. In the inner loop and for each task, we iteratively sample candidate solu- tions from Gaussian search distribution and evaluate their fitness. Afterwards, (meta-learned) Set mensformer-inspiree search strategy processes tokens of fitness transformations corresponding the population member performance. It outputs set of recombination weights, which used to update the mean and standard deviation of the diagonal Gaussian search distribution. An additional MLP module (not shown) computes per-dimension learning rates by processing momentum like statistics and timestamp. Right: Neuroevolution of control policies. Average normalized perfor- mance across continuous control Brax environments (Freeman et al. (021). The LES discovered by MetaBBO on small set of analytic meta-training problems generalizes far beyond its meta- training distribution, in terms of problem type, population size, search dimension and optimization horizon. In fact, LES outperforms diagonal ES baselines (normalized against OpenES) and scales well with an increased population size (normalized by min/max performance across all strategies). Results averaged over 10 independent runs (#:1.96 standard errors). Each task-specific learning curve is normalized by the largest and smallest fitness across all considered ES. Afterwards, the normalized learning curves are averaged across tasks. More training details and results can be found in Section 5 and in the supplementary information (SI,C and Figure T6).
1. We propose novel self- self-attention-baseee ES parametrization, and demonstrate that it is possible to meta-learn black-box optimization algorithms that outperform existing hand- crafted ES algorithms neuroevolution tasks (Figure [ right). The learned strategy gen- eralizes across optimization problems, compute resources and search space dimensions.
2. We investigate the importance of the meta-task distribution and meta-tran training protocol. We find that order to meta-evolve a well- cell-performing ES, only a handful of core optimization classes are needed meta-training time. These include separable, multi-modal and high conditioning functions (Section 55).
3. We reverse-engineer the learned search strategy. More specifically, we ablate the black-box components recovering an interpretable strategy and show that all neural network compo- nents have a positive effect the early performance of the search strategy (Section (6) The discovered evolution strategy provides simple to implement yet very competitive new ES.
4. We demonstrate how generate new LES starting from blank-slate LES: A randomly initialized LES can bootstrap its own learning progress and self relereeffrntt meta-leam its weights (Section 7).
2

---

Published as a conference paper at ICLR 2023
2
RELATED WORK
Evolution Strategies & Neuroevolution. Unlike gradient-based optimization, ES (Rechenberg, 1973 Schwefel 1977 Beyer & Schwefel, 2202) rely only on black-box function evaluations with- out the necessity of access to gradient evaluations. Instead, they iterate between the sampling of can- didate solutions, their evaluation and the updating of the sampling distribution. ES can be grouped into estimation of distribution algorithms, which refine the sufficient statistics of sampling distri- bution order to increase the likelihood of high fitness solutions (Hansen & Ostermeier, (2001) and finite-difference gradient-based methods (Wierstra et al. 20144; Nesterov & SSokoiin] 2017). They can address non-differentiable problem settings and yield competitive wall-clock times on neural network tasks due to parallel fitness evaluation (Salimans et al. 2017: Such al.) 2017).
Discovering Algorithm Components from Data. The history of machine learning has been char- acterized by the successive replacement of manually designed components by modules that are in- formed by data. This includes the use of convolutional neural networks that learn filters (Fukushima) 1975 LeCun al.| 1998), neural architecture search (Elsken al. 2019) or initilization schemes Dauphin & Schoenholz, 2019). Recently, these efforts have been extended to the end-to-end dis- covery of objective functions Reinforcement Learning (Kirsch al., 2019 Oh et al., 2020; Xu
ctt al. 2020 Lu ct al. 2022), the online refinement of hyperparameter schedules (Xu ct al. 2018
Zahavy et al., 2020: lennerhag et al.,
202T Parker-Holder al.) 2022) and the meta-learming of entire learning algorithms (Wang et al.
2016
Kirsch & Schmidhuber] 202 Kirsch et al. 2022).
Meta-Learned Gradient- Optimization. An exciting direction aims to meta-leam gradient descent-based learning rules (Bengio et al. 1992; Andrychowicz et al., [016]] Metz et al. 2019) to update the weights of a neural network. Unlike our proposed method, these approaches require access to gradient calculations using the backpropagation algorithm implemented by automatic dif- ferentiation software. The gradient and several summary statistics are processed by a small neural network, whose weights have been meta-learned on a distribution of tasks (Metz et al., 20200.. The resulting outputs are calculated on a per parameeeeeer basis and used to characterize the magnitude and direction of the weight change. It has been shown that this approach biases for domain-specific gradient-base can struggle to broadly generalize to task domains beyond the meta-training distribution.
extract useful inductive
optimization (Merchant et al.) 2021 Metz et al.] 2022) but
Meta-Learned Black-Box Optimization. Shala et al. (2020) meta-learn a policy that controls the scalar search scale of CMA-ES (Hansen & Ostermeier, (001). Chen al. (2017); TV et al. (2019): Gomes et al.] (2021) explored meta-learning entire algorithms for din-dimeniional BBO. They parametrize the algorithm by recurrent network processing the raw solution vectors and/or their associated fitness. Their applicability is constrained to narrow optimization domains, fixed population sizes or search dimensions. The proposed LES, on the other hand, learns to recombine solution candidates by leveraging the invariance properties of dot-product self-attention (Lee et al.) 2019 TTagg& & Ha. 2021 I. After meta-training the learned ES generalizes to unseen populations and large number of dimensions (>> 5000). To the best of our knowledge we are the first to demonstrate that a meta-learned ES generalizes to neuroevolution tasks. Our meta-evolution approach does not rely on access to teacher ES (Shala et al. 2020) or knowledge of task optima (TV et al. 2019).
Self-Referential Meta-Learning, Our current learning systems are not capable of flexibly refining their own learning behavior. [Schmidhuber] (1987) first articulated the vision of self-referentially updating a genetic algorithm to improve its own performance. Kirsch & Schmidhuber (2022) further formulated simple heuristic to allocate compute resources for population- self-referential learning. Metz et al. (2021a) showed that the recursive meta-training of gradient-based optimizers feasible using an improvement mechanism based on population- 2017). Here, we introduce a simple self reffrrefeeetiaal MetaBBO loop, which successfully meta-evolves LES capable of generalization continuous control tasks.
training (Jaderberg et al.)
3 PRELIMINARIES: GAUSSIAN EVOLUTION STRATEGIES
Many common ES use Gaussian search distributions, with the most scalable ones using isotropic or diagonal adiieeali Gaussian approximations. They have shown to effectively scale to large search space dimensions, specifically in the context of neuroevolution (Salimans al. 2017). and can outperform full covariance search methods (Ros & Hansen 2008). We therefore chose them as the starting point for our irenttoo-based LES architecture, and use four common ones baselines:
3

---

Published as a conference paper at ICLR 2023
OpenES (Salimans et al. 2017). sep-CMA-ES (Ros & Hansen 2008), PGPE (Sehnke et al. 2010
and SNES (Schaul et al. 2011p. We also compare with ASEBO (Choromanski et al. 2019), which dynamically adapts the hitness geometry based on the iterative estimation of gradient subspaces.
Diagonal ES maintain mean m E RD and standard deviation vectors o E RD, where D denotes the number of search space dimensions. At each generation one samples population of candidate solutions X; E P ~ (m, 01p) for all j = 1. 1..... N population members. Each solution vector evaluated, producing fitness estimates fj R. which are used construct recombination weights W. These weights define how to update the search distribution (mean and standard deviations). Most often, the weights are assigned based on the rank of the fitness rank(j) E [V. among the current population, for example:
1 F. if rank(j) S E,
Wt, ((aa) Wrank(f)
(1)
0, otherwise,
where (rank(j) denotes the weight assigned to the rank of member j. and E denotes the number of high-fitness candidates that contribute to the update. The search distribution is then updated using an exponentially moving fitness- average:
m (1
Om.t
mm
+
mm
W
Xj+
(2)
N
0+ (1
Cxat
0 +
OCott
Wts
(x) m,)
(3)
which can be interpreted learning rates am, O₂ and the weighting scheme wt, highlighted in red, are commonly fixed across the update generations t 1, ..... T, which is obvious restriction to the flexibility of the ES. The main aim of our proposed meta-learned neural network parametrization is to overcome the implied limitations by meta-learning these.
finite-difference gradient update the expected fitness. Both the
META-EVOLVING EVOLUTION STRATEGIES
In this section, first introduce the learned evolution strategy architecture (Figure 1 middle). self-attention-basee generalization the diagonal Gaussian ES in equations 2 and 3 Section 4.2
then outlines the MetaBBO procedure (Figure left) used to meta-evolve the weights of L.E..
4.1 Learned Evolution Strategies (LES): POPULATION ORDER-INVARIANE VIA ATTENTION
A fundamental property that any (learned) ES has to fulfill is invariance in the ordering of popu- lation members within a generation. Intuitively, the order of the population members is arbitrary and should therefore not affect the search distribution update. A natural inductive bias for an appro- priate neural network- -bsed parametrization is given by the dot-product self-attention mechanism. Our proposed learned evolution strategy processes matrix F, of population member-specific to- kens (Figure I middle). F. E N consists of (1) the z-scored population fitness scores, (2) centered rank transformation (lying within [-0.5, 0.5)), and (3) Boolean indicating whether the fitness score exceeds the previously best score. This fitness matrix is embedded into queries (Q N ), keys (K R NxDx and values (V (x11 using learned linear transforms with weights WK. Wo, Wv. Recombination weights are computed as attention scores over the members:
QKT (vDx)
F. (.WoWFFF F. VD
we = softmax
softmax
vv
softmax (oftmax
F(WY) E ERR.
The resulting recombination weights are shared across the search space dimensions, but vary across the generations f. Instead of using set of fixed learning rates, we additionally modulate Om,t, Qr,,1 E R by a .LP parametrized by ó and shared across D. It processes a timestamp em- bedding and parameter-speciff information provided by momentum-like statistics (see AI The chosen parametrization can in principle smoothly interpolate between integrating information from many population members and copying over single best performing solution by setting a single
4

---

Published as a conference paper at ICLR 2023
recombination weight and am,, to for all dimensions. The set of LES parameters given by 9 = [WK, Wq. Wv.dh o). whose total number is usually small (<500) given a sufficient key embed- ding size (Dx = 8). The complexity of the proposed LES scales quadratically in the population size, which for common application budgets (N < 10, 000) introduces a negligible runtime cost.
4.2 MetaBBO: META-BLACK-BEL OPTIMIZATION OF BLACK-BOX OPTIMIZERS
How do we learn the LES parameters so that they provide useful inductive biases for black- box optimization? Meta-learning via evolutionary optimization has previously proven successful in the context of learned gradient-based hased optimization (Vicol et al. (022) and RI. objective discovery (Lu et al. 2022). It avoids exploding meta-gradients (Metz et al. 20011b) myopic meta-solutions Flennerhag et al. 2021 Lange & Sprekeler] 2022). and allows to optimize 0 through long inner loop ES executions involving stochastic sampling. We adopt this meta-evolution approach in MetaBBO and iterate the following steps (Figure I left; SI Algorithm
Meta-sampling. At each meta-generation we sample a population of LES network parametrizations 8, for i = ...... M meta-populle popullapllt members using standard off-the- shelf ES algorithm with meta-mean μ and covariance E.
2. Inner loop search. Next, we estimate the performance of different LES parametrizations on a set of tasks. We sample set of inner loop optimization problems. For each task we initialize a mean ma E RD,, standard deviation do E R.. Each LES instance is then executed on a batch of inner loop tasks (in parallel) with N population members and for fixed set of inner-loop generations = I.....
3. mirmalzor In order ensure stable meta-optimizztion with optimization problems that can exhibit very different fitness scales, we normalize (z-score) the fitness scores within tasks and across meta- popplaopoottion members. We average the scores over inner-loop tasks, and maximize over inner-loop population members and over generations.
4. Meta-updating. Given the meta- meta-perffrrmace estimate for the different LES parametriza- tions, we update the meta-ES search distribution μ², >' and iterate.
The choice of the meta-task distribution is crucial ensure generalization of the meta-learned evo- lution strategy. We select a small set of classic black-box optimization functions, to characterize a space of representative optimization problems. The problem space includes smooth functions with and without high curvature and multi- modal surfaces with global and local structure. Furthermore, we apply Gaussian additive noise the fitness evaluations to mimic unreliable estimation and sam- ple optima offsets. MetaBBO differs from standard meta-learning in a couple aspects: First, we use BBO in both the inner and outer loop. This allows for ficxible LES design, stochastic sampling and long inner loop ES execution. We also share the same task parameters, mo and fixed randomness for all meta meta-population members O,. The fixed stochasticity and task-based normalization enhances stable oeta-oppimizztiinn ( ("Pegasus-trick Ng & Jordan 2000).
EXPERIMENTS
In this section we discuss the results of meta-learning LES parameters via MetaBBO. We answer the following questions: Does MetaBBO discover an ES capable of generalization beyond the meta- training distribution and can the LES meta-trained only limited set of functions even generalize to unseen neural network tasks with different fitness functions & network architectures?
5.1 META- CEETA-TRIIING ON THE BLACK- -ARKK--O BOx OPTIMIZATION BENCHMARK
Throughout, meta-train on functions from the BBOB (Hnnsen al. 2010 benchmark, which
comprise a set of challenging functions for optimization (Table l). From BBOB, we construct
small, medium, and large meta-training set (detailed in Section 5.2 To create a task in- stance, we randomly sample: (1) a function f from the meta-training set, (2) dimensionality
D 3,,,3.... 10) for f DDD -> IR, (3) a noise-level ~ UO,, 0.1). and (4) an optimum offset 2 ~ 5[-5, 5] 5) which together creates a stochastic objective f(x) f(x 2) =N 1). Finally, (5) we sample an initialization mo [-5, 5)_ and starting generation counter E [0, 2000].
5

---

Published as a conference paper at ICLR 2023
Sphere: Final Fct. Value
10-Dim. BBOR Functions
Meta- Perf. Final Mean
Number of Dimensions
30
-12
14
Rell Fanrtion Valae
2.5
20
SeCCKA
SVES
LES
Meta-Peri Perf.
20
10
-5
10
-
OpenES SNES CMA-ES
1.0
8.
-10
500
1000
1500
Romenbreck
Schvelal
Grimeant
Lussentt
BereCiger
Sharptidg
Gallopher L01 Gollggerz2
Meta- Geta-Generations
8
35
32
128
Population Size
Figure 2: BBOB evaluation of LES. Left. MetaBBO fitness objective over meta- generations. The optimization process is robust to the choice of meta-ES. Middle. Minimization performance on Sphere task across dimensions and population size (T 25). Boxed cells indicate meta-training tasks. Right. Relative performance on different BBOB test functions (D = 10, N = =16,7 = 100), lower is better. Despite LES only being meta-trained on population size of N 16 and low- dimensional (D) < 10) search spaces, it is capable of generalizing to different functions, problem settings and resources. Detailed meta-training curves can be found in SI (Figures
In the MetaBBO outer loop we optimize 0 using CMA-ES (Hansen & Ostermeier, 2001 for 1500 meta-generations with a meta-population size of M = 256. We sample 128 tasks and train each sampled LES on each task independently. Each LES is unrolled for T 50 generations, with N = 16 population members each iteration Each LES-task pair is assigned a raw fitness score according the highest fitness (lowest loss) obtained in the rollout. These raw scores are meta- normalized per task before we aggregate them into a meta-fitness score for each sampled LES. During MetaBBO training, we find average meta-training improves rapidly (Figure 2 top left) and is invariant to the choice of the meta-ES optimizer. We assess the performance of LES obtained through the MetaBBO outer loop, on the BBOB functions (Figure right). LES generalizes well to larger dimensions and larger evaluation budgets on both meta-training and meta-testing function classes. While the task complexity increases with and performance drops for a fixed population size. LES is capable of making use of the additional evaluations provided by an increased population.
5.2
META-GENNEAALIZATTOON OF LES TO UNSEEN NEUROEVOLUTION TASKS
Our results on BBOB suggests that LES have strong generalizing properties, thus prompting the question, what are the limits of LES' generalization capabilities? In this section, we evaluate an LES trained on BBOB on set of CNN-based classification tasks and continuous control tasks (Figure 3 using MLP policies. This requires LES to generalize beyond the 10-dimensional problems it was trained on to thousands of dimensions. Not only that, but also to completely different loss surfaces, and even learning paradigms (in the case of continuous control).
Figure 3 shows that LES is indeed capable of successfully evolving CNN classifiers on different MNIST variants. We also find that it is capable of evolving 4-layer MLP policies for robotics tasks Freemann et al. 202I). We consider three budgets for test-time optimisation (that vary in inner loop length and population size N, see Figure 3) LES generally outperforms ES baseline across test- time budgets. It enjoys substantial advantage in small budgets, where it outperforms baselines 5 out of tasks. LES still performs best on the majority of tasks for the medium and large adaptation budgets, which requires evaluate the impact of the meta-training distribution on the generalization of LES. The small meta- training set consists of BBOB function and restricts dimenionality to D with inner loop length T = 25. medium is restricted to 5 functions., with D E 2 5} and T = 25; large is given by 10 functions, E [1, ...., 10) and 7 50. F shows that training on broader meta-training distributions has positive effect on the generalization capability of the trained LES,, as measured on continuous control tasks. particular, we find that meta-training with larger populations in the inner loop especially benefits from scaling up the meta-training set.
generalize larger populations and longer unroll lengths. Next, we
we demonstrate that the MetaBBO is largely robust the choice of D, T, N, Dg. For all considered ES we grid-searched over different do and tuned learning rates were additionally tuned on the Brax ant environment. More details can be found in the SI D We provide more results on 4 MinAtar Young & Tian 2019) tasks with CNN policics in the SI (Figure
Pn ST Figures 10

---

Published as a conference paper at ICLR 2023
Small Budget
Medium Budget
Large Budget
Generations: 100 & Population: 32 fashion_ minst
Generations: 200 & Population: 64 fashioo_ mmrist
Generations: 500 & Population: 128 fashion,
Test Accuracy
mnia
onnist
mnis
omnist
omnist
OpenES SNES
ASEBO
mnist corrupted
miist corrupted
mnist_corrupted
Generations: 500 & Population: 128 halfchretah
Generations: 1000 & Population: 256
Generations: 2000 & Population: 512 halfcheetah
halfchectan
Brax Performancd
hopogr
apacher
reacher
seacher
fetc
fetci
fetc
urSe
walker2d
walker2d
urSe
walker26
humaanid
humarod
humanoid
Figure 3: Evaluation of LES on classification and continuous control tasks. Top. BBOB meta- evolved LES generalizes to classification tasks including MNISTTPasion.MNNSTKKMMMIM
clas-
sification using a simple Convolutional Neural Network. Bottom. The learned strategy can also evolve control policies based different control environments. Results are averaged over (vision) and 10 (control) independent runs and max/min normalized for Brax for better comparison. Details: SI (Figures [6 20
hidden layer Tanh MLP architecture with 32 hidden units in
REVERSE ENGINEERING LEARNED EVOLUTION STRATEGIES
In order to investigate which algorithmic com- ponents are discovered by the MetaBBO pro- cedure, we start by ablating the different meta- learned neural network modules. Fixing both recombination weights and the learning rate modulation, per the simple Gaussian ES in Section 3 degrades performance significantly (Figure top left). Fixing the recombination weights (as in equation 1, E = 0.5N) also decreases performance, but fixing the learning rates to Om = a, = 0.1 does not hurt per- formance significantly. We observed that the learning rates are flexibly adapted early in train- ing, but settle quickly constant values (Figure 5) right). Next, visualized the recombina- tion weights for a LES run on the Walker2D Brax task (Figure top middle). LES per- forms variable amount of selection and either applies hill-climbing integrates information from multiple well-performing solutions. We plot the weights as a function of the centered rank (for N 256) for sampled Gaussian fitness scores (Figure 5 bottom middle). The recombination weights can be well approximated by a softmax parametrization of the centered ranks:
Impact of Meta-Task Distribution
Rel. Perf. vs. OpenES
OpenES (256)
Pop.: 128
Pop.: 256
Pop.:
1.0
0.9
1 BBOB Function D2-T25
5 BBOB Funtions D5-T25
10 BBOB Functions D10-T50
Figure 4: The performance of LES increases as is meta-trained on richer task distributions. We plot the relative performance of LES on Brax tasks against OpenES (Salimans et al. 2017).
Wt, softmax (20 x [1 sigmoid (3 (rank(j)/N 0.5)))) Vj 1,.., N.
with a temperature parameter 3 2 12.5. Intuitively, the temperature regulates the amount of lection pressure applied by the recombination weights. Based on these insights, we implement an interpretable ES that uses the discovered recombination weight approximation. We call the resulting

---

Published as a conference paper at ICLR 2023
Ablation Study: LES Components
LFS Weights Ranking
Learning Rates: am,t
Norm. Brox Performance
0.6
1.000
Learning Rate Mean
0.8
Filees Rank
10
0.999
0.6
15
0.998
Fixed w.
20
-0.2
0.2
Foedd Iralles
Fixeet weights
25
0.997
0.0
0.0
0
10
20
Generation
0
20
40
Norm. Performance
Reverse-Enginenn LES DES
15
Centered Rank Logit
DES: 12.5
Learning Rates: Co,t
0.8
Weight Logit
10
L85: improved
LES: Improved
LES: Improved
Learning Rate Sigma
0.3
0.2
0.4
- 12.5
0.1
DES 9 Tuned
Parameter 1
0.0
Parameter
0.0
1000
2000
Generations (x256 Evaluations)
Centered Rank
Generation
Figure 5: Reverse engineering of LES. Left. Ablation study of LES and performance of reverse- engineered DES on continuous control tasks. The self-attention- recombination weights con- tribute most to the performance of LES. Furthermore, DES provides competitive ES when suffi- ciently tuning 3. Results are averaged over independent runs (#1.96 standard errors). Middle. Recombination weights the first 20 LES generations on the Walker2D task. LES can perform both hill climbing by assigning high recombination weight to the best performing population mem- ber and distribute the recombination weight across multiple recombination weights can be approximated well by equation (4). Right. Learning rate modulation of LES on subset dimensions. LES learning rates quickly converge to fixed values after initial adaption. Detailed learning curves can be found in SIE (Figures 13
pell-orffommgg candidates. The LES
algorithm Discovered Evolution Strategy (DES, SI Algorithm I DES 12.5) performs simi- larly to LES on the Brax benchmark tasks and can be further improved by tuning the temperature parameter (Figure 5 bottom left). This result highlights that MetaBBO can be used not only to discover black-box ES strategies, but also to discover new interpretable inductive biases.
7 SEL.F-EFERENTIIG META-EVOLUTION OF LES
Given a choice of meta-ES, MetaBBO can discover LES that performs competitively with baseline ES. It is therefore natural to ask, can we use an LES to discover new LES? And can this be purely self-referential loop, starting from a random initialization? To address this question we consider a two-step self-referential metaBBO paradigm, which starts from random 8meta (Figure 6 left):
1. We evaluate set of candidate LES parametrizations 0g [(,, Σ) with Σ = o1. After obtaining the meta-fitness scores f(,)). we update the strategy statistics using the meta- LES update rule Jr E' <- LES( M Ie, Σ, Ometa).
2. If the meta-fitness has improved, we replace 0meta + arg maxe, f(0.) using a hill- climbing step that copies the best performing inner-loop mean n Ometa- Addition- ally, we reset the meta-search variance to induce additional exploration around the newly improved meta-search mean: Σ +- do Dmett
Note that this procedure starts out as simple random search with hill-climbing selection. As the meta-parameters are updated, the meta-search is improved by the newly learned inductive biases. Thereby, LES is capable of improving its own optimization behavior. Figure compares different meta-training procedures for LES, including the usage of CMA-ES, previously meta-trained and fixed LES instance, and our proposed self-referential training paradigm. We find it is indeed fea- sible to effectively self-evolve LES starting from a random initialization. While the meta meta-fitnee fitness is improved fastest by the pre-trained LES instance, observe that the resulting meta-trained LES
8

---

Published as a conference paper at ICLR 2023
Median Pert. Final Mean
Circles Classifier (Population 32)
0.9
Sample tasks
p. LES LES((A, (6, (omtts 14 σ)
Meta Performance
-20
-40
0.7
-60
Sample LES
0.6
Pretrained
-80
Seff LES
CMA-ES (Nodium)
-100
500
1000
1500
500
1000
1500
Geaa-eerrttoons
Meta-Generations
Run search
Acrobot-v1 (Population 32)
Self-Referential MetaBBOB
-80
Performance
Meto Performance
1.0
Meta-update
-85
0.8
0.6
0.4
Meta- M Copy best & set mean/std
0.2
CMA-55
500
1000
1500
2000
Meta- SettaGenerations
Generations (X256 Evaluations)
Figure 6: Self-Referential MetaBBO of LES. Left. We use LES to meta-evolve its own parame- ters I starting from random initialization and using hill climbing replacement. Middle. Meta- performance over training. is possible to meta-train LES using both a previously trained and frozen instance and seff-referentiallyy improved random initialization. Right, Top. Performance on 2D Circles MLP classifier and Acrobot control tasks throughout meta-training. Right, Bottom. While the LES discovered by frozen pre-trained version does not generalize Brax meta-testing, the self-referentially meta-trained one does. Results are averaged over independent runs (+/- 1.96
standard errors). Detailed learning curves can be found in SI E (Figures 18 D.
does not generalize well to the downstream Brax evaluation (Figure 6 bottom right). Both the CMA-ES optimizer and self-referential meta-training procedure, on the other hand, are capable of meta-evolving stable and performant LES, that do not overfit the meta-training task distribution. We speculate that self-referential meta-training performs implicit regularization: LES forced to simultaneously perform well the inner loop tasks and to progress its continual improvement.
CONCLUSION & DISCUSSION
Summary. We introduced self-attention-bassee ES architecture and a procedure effectively meta-learn its parametrization. The resulting learned evolution strategy is flexible and outperforms established baseline strategies both vision and continuous control tasks. We reverse engineered the discovered search update rule into a simple and competitive DES. Finally, we showed that it is possible to successfully self-referentially evolve LES starting from a random initialization of itself.
Limitations. The LES considered in this paper only applies to diagonal covariance strategies. This limits its effectiveness on non-separable problems. We observed that the learning rate module can struggle to generalize to long time horizons indicating degree of temporal meta cever-overiitiing. Finally, the self-attention mechanism scales quadratically in the number population members. This may be addressed by improvements in Transformer efficiency (Wang et al. 2020 Jaegle et al. (02T).
Future work. Our LES architecture meta-learns to modulate the search distribution update solely based on fitness transformations. One potential extension is to provide additional information (agent trajectory or multiple evaluations) to the attention module. Thereby, the MetaBBO could discover behavioural descriptors in the context of quality-diversity optimization (Pugh et al. 2016 Cully & Demiris 2017) or application specific denoising procedures. Furthermore, one may want to attend over archive of well-performing solutions (e.g. as in genetic algorithms) using cross-attens cross-attention that implements a cross-over operation. Finally, our experiments indicate the potential of further scaling of the MetaBBO procedure broader meta-training task distributions.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
Work funded by DeepMind. We thank Nemanja Rakićević for valuable feedback on a first draft of this manuscript. Furthermore, the authors grateful to Antonio Orvieto, Denizalp Goktas, Evgenii Nikishin, Junbyok Oh, Greg Farquhar, Alexander Neitz, Himanshu Sahni, Ted Moskovitz and other colleagues the Discovery team and DeepMind for stimulating discussions over the course of the project.
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gescei. On the optimization of a synaptic learning rule. In Optimality in Biological and Artificial Networks?, pp. 281-303. Routledge, 1992.
Hans-Georg Beyer and Hans-Paul Schwefel. Evolution strategies-a a comprehensive introduction.
Natural computing, 1(1):3-52, 2002.
James Bradbury. Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- MMllne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com gotpo///oogll///aa.//a
Yutian Chen, Matthew W. Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy Lilli- crap, Matt Botvinick, and Nando de Freitas. Learning learn without gradient descent by gra- dient descent. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, PMLR, 06-11 Aug 2017. URL https:s/pproceediggg..iicse wtt.:sssss...r..//100. chen17e. html
pp.
Krzysztof M Choromanski, Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, and Vikas Sind- hwani. From complexity to simplicity: Adaptive es-active subspaces for blackbox optimization. Advances in Neural Information Processing Systems, 32, 2019.
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular frame- work. IEEE Transactions on Evolutionary Computation, 22(2):245-259) 2017.
Yann N Dauphin and Samuel Schoenholz. Metainit: Initializing learning by learning to initialize. Advances in Neural Information Processing Systems, 32, 2019.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1): 1997-07777 2019.
Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy. Hado van Hasselt, David Silver, and Satin- der Singh. Bootstrapped meta-learning. arXiv preprint wxXiv:2109004504 2021.
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physies engine for large scale rigid body simulation. arXiv preprint arXiv:2/06. 13281, 2021.
Kunihiko Fukushima. Cognitron: A self-organizing multilayered neural network. Biological cyber- netics, 2013)12 1975.
Hugo Siqueira Gomes, Benjamin Léger, and Christian Gagné. Meta learning black-box population- based optimizers. arXiv preprint arXiv:2103.03522 2021.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2):199-195, 2001.
Nikolaus Hansen, Anne Auger, Steffen Finck, and Raymond Ros. Real-parameter black-box opti- mization benchmarking 2010: Experimental setup. PhD thesis, INRIA, 2010.
10

---

Published as a conference paper at ICLR 2023
Charles R Harris, K. Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor. Sebastian Berg, Nathaniel J Smith, et al. Array program- ming with numpy. Nature, 585(7825):357-366) 2020.
John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, (03):90-95, 2007.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czamncki,, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et Population based train- ing of neural networks. arXiv preprint arXiv:1711.09866 2017.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pp. 6651-4664 PMLR, 2021.
Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving it. Advances in Neural Information Processing Systems, 34:14122-14134, 2021.
Louis Kirsch and Jürgen Schmidhuber. Self-referential meta learning. In Decision Awareness in Reinforcement Learning Workshop ICML 2022, 2022.
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. arXiv preprint 1xXiv:1910:04098, 2019.
Louis Kirsch, Sebastian Flennerhag. Hado van Hasselt, Abram Friesen, Junhyuk Oh, and Yutian Chen, Introducing symmetries to black box meta reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 7202-7210, 2022.
Robert Tjarko Lange. evosax: JAX-based evolution strategies, 2022a. URL http:////thbb.
Com/ wow.eetttaaggeeeeooaoaaaa
Robert Tjarko Lange. gymnax: A
reinforcement learning environment library, 2022b.
URL http::
com Cat.m///obrerraaggge...... gymnax
Robert Tjarko Lange and Henning Sprekeler. Learning not to learn: Nature versus nurture silico. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7290-7299, 2022.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):228-2322, 1998.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-basee permutation-invariiat neural networks. In Interna- tional conference on machine learning, pp. 3744-3755 PMLR, 2019.
Kuang-Huei Lee, Ofir Nachum, Tingnan Zhang, Sergio Guadarrama, Jie Tan, and Wenhao Yu. Pi- ars: Accelerating evolution-learmed visual-locomotion with predictive information representa- tions. arXiv preprint 2221v:2207.132244 2022.
Chris Lu, Jakub Grudzien Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Nicolaus Foerster. Discovered policy optimisation. In Decision Awareness in Reinforce- ment Learning Workshop at (C22, 2022.
Amil Merchant, Luke Metz, Samuel S Schoenholz, and Ekin D Cubuk. Learn2hop: Learned opti- mization on rough landscapes. International Conference on Machine Learning, pp. 7643-7653. PMLR, 2021.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training learned optimizers. In International Conference Machine Learning, pp. 4556-4565. PMLR, 2019.
Luke Metz, Niru Maheswaranathan. C Daniel Freeman, Ben Poole. and Jascha Sohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint arXiv:2009. 1243, 2020.

---

Published as a conference paper at ICLR 2023
Luke Metz, C Daniel Freeman, Niru Maheswaramathan, and Jascha Sohl-Dickstein. Training learned optimizers with randomly initialized learned optimizers. arXiv preprint arXivv2101.07337, 2021a.
Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman. Gradients are not all you need, arXiv preprint urXiv:2111.05800 11.0880, 2021
Luke Metz, C Daniel Freeman, James Harrison, Niru Maheswaranathan, and Jascha Sohl-Dickstein. Practical tradeoff's between memory, compute, and performance in learned optimizers. arXiv preprint arXiv:2203.1 2tt.::220031118800 18860 2022.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-566, 2017.
Andrew Y Ng and Michael I Jordan. Pegasus: A policy search method for large mdps and pomdps. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence (UAI), pp. 406-415, 2000.
Junhyuk Oh, Matteo Hessel, Wojeiech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 33:1060-1070, 2020.
Yann Ollivier, Ludovic Arold,, Anne Auger, and Nikolaus Hansen. Information-geomeete opti-
mization algorithms: A unifying picture via invariance principles. Journal of Machine Learning Research, 18(18):1-65, 2017.
Jack Parker-Holder, Raghu Rajan, Xingyou Song, André Biedenkapp. Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, et al. Automated reinforcement learning (autorl): A survey and open problems. Journal of Artificial Intelligence Research, 74: 517-568, 2022.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley, Quality diversity: A new frontier for evolu- tionary computation. Frontiers Robotics and AI, pp. 40, 2016.
Ingo Rechenberg. Exolutionsstrategic. Optimierung technischer Systeme nach Prinzipien derbiolo-
gischen Evolution, 1973.
Raymond Ros and Nikolaus Hansen. A simple modification cma-es achieving linear time and space complexity. In International conference parallel problem solving from nature, pp. 296- 305. Springer, 2008
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03866 2017.
Tom Schaul, Tobias Glasmachers, and Jürgen Schmidhuber. High dimensions and heavy tails for natural evolution strategies. Proceedings of the 13th annual conference on Genetic and evolu- tionary computation, pp. 645-852,, 2011.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how to learn: the meta-meta-.. hook. PhD thesis, Technische Universität München, 1987.
Hans-Paul Schwefel. Evolutionssttattgien für die numerische optimierung. In Numerische Opti- mierung von Computer- Modeeleen mittels der Evolutionsstrategie, pp. 123-176. Springer, 1977.
Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, and Jürgen Schmidhuber. Parameter-exploring policy gradients. Neural Networks, 23(4):551-53 2010.
Gresa Shala, André Biedenkapp, Noor Awad, Steven Adriaensen, Marius Lindauer, and Frank Hut- Learning step-size adaptation in ema-es. In International Conference Parallel Problem Solving from Nature, pp. 691-706. Springer, 2020.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neurocvolution: Genetic algorithms are a competitive alternative for training 2017.
deep neural networks for reinforcement learning. arXiv preprint
12

---

Published as a conference paper at ICLR 2023
Yujin Tang and David Ha. The sensory neuron transformer: Permutation iivioonnnninanintt neural networks for reinforcement learning. Advances in Neural Information Processing Systems, 34: 22574-22587, 2021.
Yujin Tang, Yingtao Tian, and David Ha. Evojax: Hardware-accellrateed neuroevolution. arXiv preprint wrXiv:2202.05000 2022.
Vishnu TV, Pankaj Malhotra, Jyoti Narwariya, Lovekesh Vig, and Gautam Shroff. Meta-learning for black-box optimization. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 366-381. Springer, 2019.
Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein. Unbiased gradient estimation in unrolled com- putation graphs with persistent evolution strategies. In International Conference on Machine Learning, pp. 10553-10563 PMLR, 2021.
Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint /61 1,05763, 2016.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.
Michael L Waskom. Seaborn: statistical data visualization. Journal of Open Source Software, (60):3021, 2021.
Dennis Weyland. A rigorous analysis of the harmony search algorithm: How the research commu- nity can be misled by a "novel" methodology. International Journal of Applied Metaheuristic Computing (IJAMC), 1(2):50-60, 2010.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. The Journal of Machine Learning Research, 15(1)949-988, 2014.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. Ad- vances in neural information processing systems, 31, 2018.
Zhongwen Xu, Hado van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. Meta-gradient reinforcement learning with an objective discovered online. Advances in Neural Information Processing Systems, 33:1525 152644, 2020.
Kenny Young and Tian Tian. Minatar: atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.0311 0317790,..1.,,. 2019.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. self-tuning actor-critic algorithm. Advances in Neural Information Processing Systems, 33:20913 2020.
13

---

Published as a conference paper at ICLR 2023
Appendix
Table of Contents
Meta- Training Details
14
B Discovered Evolution Strategy
15
C MetaBBO Algorithm for Meta-Evolving Evolution Strategies
16
D
Hyperparameter Settings
16
D.1
MetaBBBO Hyperparameters
16
D.2
Neuroevolution Evaluation Hyperparametery
16
D.3
Baseline Settings & Tuning
17
D.4
Software & Compute Requirements
17
E
Additional Results
18
E,
Meta- Evolution Training Curves & Ablations
18
Detailed DESILES Neurocvolution Results
19
F
Pseudo-Code Implementations for DES & LES
22
F1
JAX-Based Pseudo-Code for Discovered Evolution Strategy
22
F2
IAX-Based Pseudo-Code for Learned Evolution Strategy
24
A META-TRAINING DETAILS
Function Name
Reference
Property
Meta-
Distribution
Sphere
Hansen al.
(p. 2010)
Separable (Independent)
Small
Rosenbrock
Hansen et al.
(p. 40. 2010
Moderate Condition
Medium
Discus
Hansen al.
(p. 55,
2010
High Condition
Medium
Rastrigin
Hansen al.
(p. 75, 2010
Multi-Modal (Local)
Medium
Schwefel
Hansen et al.
(p. 100,
2010
Multi-Modal (Global)
Medium
Baschiea
Hansen
(p. 20,
2010)
Separable (Independent)
Large
AttractiveSector
Hansen et al,
(p. 30,
2010
Moderate Condition
Large
Weierstrass
Hansen
(p. 80,
2010
Multi-Modal (Global)
Large
SchaffersF7
Hansen et al.
(p. 85,
2010
Multi-Modal (Global)
Large
GriewankRosenbrock
Hansen al.
(p. 95, 2010
Multi-Modal (Global)
Large
Table 1: Meta-training black-box optimization tasks with different structural properties.
During meta-training we uniformly sample task parameters & for h: = 1..., K different tasks, characterizing the inner fitness f IR - namely the function identifier and the number of dimensions D E [2, 10). In addition, we sample a constant offset to the optimum x² and a noise level E, [f(x) + e] ((x) with e ~ (0, E). The initial mean is sampled m, E [-5, 5] order to ensure solutions that are robust to their starting initialization. We unroll the LES for T = 50 generations with N = 16 population members/function evaluations each. The meta-objective is computed based on the collected inner loop fitness scores:
K
M
1
E NxTXKXM
1=1
1=1
14

---

Published as a conference paper at ICLR 2023
For each candidate LES 8, minimize over time and population members. Afterwards, 2-score the task-specific results over all population members and compute the mean across all tasks:
K [1OO k=1 = min min [IN
0g
k=1 i=1
(f00)) = Median 12-Score (15(0.)E:)
In practice we found that can be helpful meta-optimize the median across tasks the z-scored fitness. Inspired by Shala et al. (020)) the learning rate modulation of Om Ot is performed using a small MIPP that outputs two sigmoid activations corresponding the per-parameter learning rates (mean/std) using tanh timestamp embedding as in (Metz et al. 2022 and set of evolution paths, which mimic momentum terms at different timescales:
Om,,,d Carttd Pe.,t+,, Pa,t,d p(())
Pc,4-1 (1 ap. Pe.t + Apo E, wy(2z m.) -Pc,t) E ERO Dx3
Po,+1 (1 Opp Part Opp E, Wy(#1 m,)/o: Pa.) E R Dx3
with three time-scales Cp. Op Pa = 0.5, 0.9] and p(t) = tanh(t// 1) with [1,3, 10, 30, 50, 100, 250, 500, 750, 1000, 1250, 1500, 2000]. The 'medium' of BBOB meta- training functions for D and the time embedding features are visualized in figure 7
Sphere Contoer
Resembrrck Corbour
Disus Centour
Bastrigt Contour
Scheefll Contour
Tanh Time Embedding 0-50
Tanh Time Embedding 1000-1050
Feature
5
10
10
40
10
20
30
40
50
Time
Time
Figure Visualization of 2D 'medium' BBOB functions and tanh timestamp embedding features.
B DISCOVERED EVOLUTION STRATEGY
Algorithm Discovered Evolution Strategy (DES)
Inputs:
Population members search dimensions D, generations T. Initial scale do R.".
Mean and std learning rates = 1, ag = 0.1, temperature parameter / z 12.5.
2: Initialize the search mean m 0n R D
3: for = 0, ... T do
4: Sample candidates: I ~ NM(,, 0,1p)
5: Evaluate all candidates: f,, Vj 1, I,.., N
6: Compute weights using: Wt, = softmax (20 x sigmoid (3 x (rank(j)/N -0.5)) Vj
1. 1,.., N
7: Update mean using equation (2) + m
8: Update std using equation (3) 0++1
9: end for
15

---

Published as a conference paper at ICLR 2023
C METABBO ALGORITHM FOR META-EVOLVING EVOLUTION STRATEGIES
Algorithm 2 MetaBBO Training of Learned Evolution Strategies
1: Inputs: Meta-populattee population members M. meta-task size K, inner loop population size N, inner loop generations T, MetaES (c.g. CMA-ES)
2: Initialize the meta-search mean and covariance J4, E.
3: while not done do
4: Sample K tasks with parameters Ek. Vk 1, I,.... K
5: Sample LES candidates: 0g ~ (M, Σ), Vi = 1, M
6: Evaluate all M LES candidates (in parallel) on K tasks: for k = I,... K do
7:
8:
for i
1, L.... M do
9:
Initialize inner loop search means/std mo,k.i E R ER Bo.kii RR
for 0, ..., 7 do
11:
Sample inner loop candidates: Zj,t NM ot, Vj N..., Evaluate all inner loop candidates: f(zj (e), Vjj L..., N
12:
13:
Update LES mean/std using equation (2) and (3) - m 0411,k with 8;
14:
end for end for
15:
16:
end for
M 10
17:
Collect all inner loop fitness scores
[(, k=1
RRNTTKKM
18:
Compute the population normalized and task aggregated meta-fitness scores (f(0)))
19:
Update meta-mean, meta-std p², E' MetaEs((O,, f end while
20:
D HYPERPARAMETER SETTINGS
D.1 METABBO HYPERPARAMETERS
MetaBBO Hyperparameters for LES Discovery
Hyperparameter
Value
Hyperparameter
Value
Meta-Generations
1500
Meta-Population
256
Meta-Tasks
128
Meta-ES
CMA-ES
CMA-ES du
0.1
Amerr-Ppuulation
16
Inner-Gent Generatnnss
50
my range
[-5,
Timestamp Range
[0, 2000]
do range
[1, 1]
D₁ Attention keys
MI.P Hidden dim.
8
For the self-referential meta-trainin results (Section we replace the meta-LES parameters every 5 meta-generations we observe an improvement in the meta-objective by one the inner LES candidates. Furthermore, reset the meta-variance Σ Tn 0.17.
D.2 NEUROEVOLUTION EVALUATION HYPERPARAMETERS
Brax Task Evaluation Hyperparameters
Hyperparameter
Value
Hyperparameter
Value
Generations
2000
Population
256
MC Evaluations
16
MI.P Layers
4
Hidden Units
32
Activation
Tanh
The Brax experiments all used observation normalization.
16

---

Published as a conference paper at ICLR 2023
MNIST Task Evaluation Hyperparameters
Hyperparameter
Value
Hyperparameter
Value
Generations
4000
Population
128
MC Evaluations
CNN Layers
1
Batchsize
1024
Activation
ReLU
MinAtar Task Evaluation Hyperparameters
Hyperparameter
Value
Hyperparameter
Value
Generations
5000
Population
256
MC Evaluations
64
CNN Layers
1
Linear Layers
Activation
ReLU
Hidden Units
32
Episode Steps
500
The MinAtar experiments do not use observation normalization.
D.3 BASELINE SETTINGS & TUNING
Throughout the Brax control neuroevolution experiments, we consider the following baselines:
1. OpenES (Salimans al.
2017) using Adam and centered rank fitness transformation.
2. PGPE (Shnnke et al.
2010 using Adam and centered rank fitness transformation.
3. ASEBO (Chrromankii et al.
2019) using Adam and z-scored fitness transformation.
4. SNES (Wierstra al.
2014 using the default rank-based fitness transformation.
5. sep-CMA-ES (Ros & Hansen 2008 using elite ratio of 0.5.
We tuned the initial search scale for all baselines and LES by running a grid search over dg [0.05, 0.2). We repeat this exercise for all individual environments. Furthermore, we tuned the learning rate for OpenES (0.05), PGPE (0.02) and ASEBO (0.01) on the Brax ant environment and using a population size of 256. For OpenES we exponentially decay a to a minimal value 0.01 using decay constant of 0.999. For ASEBO the decay rate is set to y 0.99, better comparability we sample full populations instead of a dynamically adjusting the population size. For PGPE the learning rate for is 0.1 and maximal relative change of is constrained to 20 percent.
150 and for
For the computer vision tasks again tune the initial scale by running grid search du [0.05, 0.2] for each task. The learning rates are fixed as follows: PGPE (0.01), OpenES (0.01), ASEBO (0.01). Finally, for the MinAtar experiments (Figure 21 we only consider PGPE and OpenES as baselines and tune both their learning rate (a E 10.005, 0.0075, 0.01, 0.02, 0.03, 0.04)) and initial scale (oo E [0.025, 0.05, 0.075, 0.1, 0.125, 0.15)) by running grid search.
Note that LES only has a single hyperparameter aro and is fairly robust to its choice (see SI E.2.
D.4 SOFTWARE & COMPUTE REQUIREMENTS
This project has been made possible by the usage of freely available Open Source software. This
includes the following: NumPy: Harris et al. (2020), Matplotlib: Hunter (2007). Seaborn: Waskom
(2021) JAX: Braddbrr et al.] (2018). Evosax: Lange (002a)), Gymnax: Lange (2022b). Evojax:
Tang et al. al. (0022). Brax: Freeman et al. (2021). All experiments (both meta-training and evaluation on Brax) were implemented using JAX for parallel rollout evaluations. The simulations were run on individual NVIDIA V100 GPUs. Each MetaBBO meta-training run for 1500 meta- mnea-geneeations takes ca. hour. The Brax evaluations require between 30 minutes (hopper, reacher, walker, fetch, ur5e) and hours (ant, humanoid, halfcheetah). The computer vision evaluation experiments take 10 minutes and the MinAtar experiments last for 35 minutes V100.
17

---

Published as a conference paper at ICLR 2023
E ADDITIONAL RESULTS
E.1 META-BVOLUTION TRAINING CURVES & ABLATIONS
0.0
final,ial..aa median
60
RosembrockRotated d0_
20
Welerstrass_de pirrres_______0____
200
CartPolent1 d16
Meta-Pert
-2.5
40
15
199
-5.0
10
-7.5
modium 05-722
500
1000
1500
500
1000
1500
100
1000
1500
300
1000
1500
Mota- Generatiesss
Meta: Generations
Meta- Generations
Meta Generations
mean, finel, median
RosenbrockRotated d8 p64
Welerstrass_ p8.
CartPolew1 d16 932
Meto Part.
Eeci + 2-ccom
10
-40
197
Generations
Meta: Generctions
Mota Generations
Generations
Figure 8: Comparing different meta-evolution training distributions (top) and input ablations to LES (bottom). Results are averaged over 5 independent runs (+/- 1.96 standard errors).
moan finafinnanna median
40
RosenbrackRotateed d8, p64
Welerstrass_de Secrrtrass_dB__t_e d8_p16
200
CartPolow3 OLE p32
Meto-Pent
30
199
OpentS SMCS
20
198
10
500
1000
1500
500
1000
1500
500
1000
1500
500
1000
1500
Meta Genii-GGeeratiors
Meta- Generations
Meta- Generations
Meta- Getta-Gererationss
medion
RosenbraccRRtttted
p64
Welerstrass, d8,
CartPole
Prshiored
Mettr Pert
-20
75
15
199
30
198
25
500
1000
1500
500
1000
1500
500
1000
1500
500
1000
1500
Mela--Gnneratione
Mela- Generations
Mela- Satte-Geneeeltons
Mela-GGnnerations
Figure 9: Comparing meta-evolution optimizers (top) and training setups (bottom, MetaBBO & self-referential MetaBBO). Results are averaged over 5 independent runs (+/- 1.96 standard errors).
0.0
mean final median
60
RosenbrockRatated p64
20
Weierstrass_dB d8_p16
200
CartPolo-v1 d16 p32
Meta-Pert.
-2.5
40
15
199
10
20
-10.0
500
1000
1500
0
500
1000
1500
500
1000
1500
500
1000
1500
Geta-Gonnrations
Meta- Generationn
Meta: Generations
Mota- Cetta-Genaratoon
mean mean_final wwn.fllmaiiiann median
RosenbrockRottteed Batreocc-oRtttted_______.__.
Weierstrass, d8 pll_p66
CartPole d16 p32
0.0
60
20
200
Meta-Perf
-2.5
40
15
-5.0
10
-7.5
0V-1--22 1-200-13
500
1000
1500
500
1000
1500
1000
1500
500
1000
1500
Meta: Generrainns
Meta: Gera-Gennssssons
Meta: Generations
Meta: Generations
Figure 10: Comparing different meta-cvolution training problem dimensions (top) and population sizes N (bottom). Results averaged over 5 independent runs (+/- 1.96 standard errors).
18

---

Published as a conference paper at ICLR 2023
mean cea._faaaameleee median
60
RosenbrackRotatedd.oo
20
Welerstrass_ _e__.dp_ppp_____
CartPole vi d16 p32
Meto- Pee--Prtt
-2.5
15
-5.0
10
20
-7.5
75.0--3-------00 30.0
-10.0
19/
Mete-Generations.
Meta Gentrrttnesso
Mete-Generations
Meta-Generations
median
60
RosenbrackRotatedd.oo Bactiateed______________
Weierstrass_
_d__p16
CartPole-v1 p37
Meta-Pert
-2.5
15
-5.0
10
20
198
-7.5
-10.0
197
1500
3000
1500
Meta Generations
Meta-Generations
Generations
Figure 11: Comparing different meta-evolution training inner loop time horizons T (top) and LES key embedding sizes Dx (bottom). Results are averaged over independent runs (+/- 1.96 std err.).
E.2 DETAILED DES/LES NEUROEVOLUTION RESULTS
E.2.1 DETAILED ABLATIOW/ROBUSTESS PERFORMANCE ON BRAX TASKS
ant
halfcheetah
hopper
humanoid
Mean Performance
7030
7000
7500
6500
6000
7000
6000
Pops pe: 128
5000
Popeite:
1950
5500
Popsic:
5000
reacher
wallker2d
fetch
urSe
Mean Performence
-20
-25
3500
8
3000
0
2500
shall
medium
lorge
small
medium
large D10-T50
small
modium
large D10.750
small
medium
largo
D10-150
Figure 12: Impact of Meta-Tase Task Distribution Brax Performance. Results averaged over
independent runs (+/- 1.96 standard errors).
ant
helfcheetah
hopper
humanoid
Meen Perforenne
6200
8000
2000
8000
Rover freies
6000
6000
4000
1000
4000
2000
2000
500
2000
1000
2000
1000
2000
1000
2000
1000
2000
reacher
wallier2d
tetch
Mean Performance
-100
10.0
-300
2000
7.5
0
2.5
1000
2000
1000
2000
2000 Generations (x256 Evals)
1000
0
1000
2000
Gonerations (K256 Evais)
Gonerations (x256 Evals)
Gonerations (x256 Evals)
Figure 13: Brax LES Neural Network Components Ablation Study. Results are averaged over independent runs (+/- 1.96 standard errors).
19

---

Published as a conference paper at ICLR 2023
halfcheetah
hopper
Tetch
urSe
Mean Performance
12
6000
2000
4000
1500
5.0
2000
1000
2.5
ACESO
humanoid
walker2d
Moan Perforance
4000
8000
6000
3000
4000
4000
2000
2000
400
0.05 0:10 0:15
0.05
0.10 Initill Scale:
0.05
0:10
0.15
Initial Scale:
Initial Scale:
Initial Scale:
Figure 14: Robustness to initial scale of standard deviations Results are averaged over 5 inde- pendent runs (+/- 1.96 standard errors).
halfcheetah
hopper
fetch
u1Se
8..20
0.20
0.20
0.20
0.15
0.15
Opentis
0.15
0.15
SepK
A9F60
Best
0.05
0.05
0.05
0.05
humanoid
walker2d
ant
reacher
8..20
0.20
0.20
0.20
0.15
0.15
0.15
0.15
Best Initial
0.05
0.05
0.05
0.05
200
400
200 Population Size
400
200 Population Size
400
200
400
Populetion
Size
Population Size
Figure 15: Best initial scale of standard deviations across population sizes. over 5 independent runs (+/- 1.96 standard errors).
Results are averaged
E.2.2 DETAILED NEUROBVOLUTION LEARNING CURVES
helfchestah
hopper
humanoid
Mean Performence
8000
8000
6000
6000
4000
1000
4000
2000
2000
1000
2000
1000
2000
1000
2000
1000
2000
reacher
walleer2d
fetch
urSe
Mean Performance
-200
3000
10
-300
-400
SeCCMA
tho
1000
ASEBO
Generations (x256
Figure 16: Detailed Brax Learning Curves with population size of 256. PPO scores taken from Lu et al. (2022) using default settings provided by the Brax implementation (Freeman et al, 2021 Results are averaged over 5 independent runs (+/- 1.96 standard errors).
20

---

Published as a conference paper at ICLR 2023
ant
halfcheetah
hopper
humanoid
Mean Performance
6000
8000
2000
8000
1500
6000
4000
4000
1000
4000
2000
2000
500
2000
055
1e5 c=
--33--333
0
1000
2000
1000
2000
0
1000
2000
0
1000
2000
reecher
welkeer2d
fetch
Moan Performance
4000
12.5
10
-200
10.0
2000
7.5
-400
-600
0
2.5
1000
2000
2000 Generations (x256 Evals]
1000
2000 Generations (x256 Evals)
1000
1000
2000
Generations (x256 Evals)
Generations (x256 Evals)
Figure 17: Detailed Brax Learning Curves with DES(3) variants and a population size of 256. Results are averaged over 5 independent runs (+/- 1.96 standard errors).
ant
halfchestah
hopper
humanold
Mean Performance
6000
2000
6000
6000
1500
4000
4000
1000
4000
2000
500
2000
10000
2000
1000
2000
1000
2000
1000
2000
reacher
walleer2d
fetch
Mean Perforance
12.5
10
-100
3000
10.0
-200
-300
LES: CHA-E5 (Madumm
LES
Rofrentil
1000
5.0
2.5
1000
2000
1000
2000
1000
2000
0
1000
2000
Generations (x256 Evais)
Generations (X256 Evals)
Generations (x256 Evals)
Generations (x256 Evals)
Figure 18: Detailed Brax Learning Curves with Relf-Referentially Meta- Trained LES and a popula- tion size of 256. Results averaged over 5 independent runs (+/- 1.96 standard errors).
ant
helfcheetah
hopper
humanoid
Meen Performene
8000
6200
2000
8000
4000
1000
4000
2000
2000
2000
0
1000
2000
1000
2000
1000
2000
1000
2000
Mean Performance
reacher
4000
walkeer2d
12.5
tetch
urSe
-100
10.0
-200
LS5 (Sed
155 (Sees
2000
-300
L25 (Soed
(Seed
1000
5.0
0
2.5
1000
2000
1000
2000
2000 Generations (x256 Evals)
1000
0
1000
2000
Gonerations (x256 Evais)
Gonerations (x256 Evals)
Gonerations (x256 Evals)
Figure 19: Detailed Brax Learning Curves with 5 separately Trained LES and a population size of 256. Results averaged over 5 independent runs (+/- 1.96 standard errors).
21

---

Published as a conference paper at ICLR 2023
Mean Performance
Tashion mnistt
mnist
mnist comupted
kmnist
0.5
0.75
0.75
0.50
0.50
0.50
0.4
0.25
0.25
SMES
Oparts
AD250
RGPE
0.00
Evals)
(x128 Evals)
Generations Evals)
Generations (x128 Evals)
Figure 20: Detailed Learning Curves (Test Accuracy) MNIST variants with population size of 128. Results are averaged over independent runs (+/- 1.96 standard errors).
Meenn Performance
MinBreakout-v00
75
Minssacelnvadeers-00
20
MinAsterix--v
MinFreeway-v0
Opones PGPE
50
10
25
2000
4000
2000
4000
4000 Generations (x256 Evals)
2000
4000 Generations (x256 Evals)
2000
Generations
Generations
Figure 21: Detailed MinAtar Learning Curves with population size of 256. Results are averaged over independent runs (+/- 1.96 standard errors).
F PSEUDO-CODE IMPLEMENTATIONS FOR DES & LES
F.1 JAX-BASED PSEUDO-CODE FOR DISCOVERED EVOLUTION STRATEGY
from typing import Tuple
import chex
import jax
import jux.numpy as inp
def get_aes aes wew__aassswweeeeeeee weights (popsize: int, temperature: float 12.5): ***Compute discovered recombination weights. **n
ranks imp.arange (popsize)
ranks /- ranks.size
ranks ranks 0.5
sigout nn (temperrattttratuut * ranks)
weights = nn. nn.softmax (-20 * sigout)
return weights
class DES (object))
def _init_
self, popsize: int, num dins: int, temperature: float 12.5,
): """iiscovered Evolution Strategy""* self.strategy_name "DES"
self.popsize popsize self,num num dims num_dims
self.temperature = temperature
@property
def Sarame_strategy (self) -> EvoParams:
"""Return default parameters of evolution strategy. "". return EvoParams (temperature-sell rtmmarrt
22

---

Published as a conference paper at ICLR 2023
def initialize(
self, rng: chex. chex.eRN parans: EvoParams
EvoState: initiialle the evolutien strategy. "**
Get DES discovered recombination weights.
weights get_des_ _e__d___wwihhsseeeeeeess_sssssee (self, popsiie, params, temperature) initialization jax. random, www.rrrddmmunfoooe fnom rng,
(self.num_dimn)
minval=params init min,
maxval-params ini init_max, max,
state EvoState(
mean-initislization,
sigma-params sigma init inp. (self. num _nm__dnss) weights-weights. reshapp reshape (-1, 1),
return state
def ask{
self, rng: chex. PRNGKey, state: EvoState, parama: EvoParams RvoState]:
) -> [chex,
"ask" for new proposed candidates to evaluate next.
jax. wax.random.nnor normal (rng, uself.popsize self num, dims))
x state. mean
sigta.sigga, reshape 1, self. nim_dims
return 8, state
def tell(
self, x: chex,Array, fitness: Arra,,
state: Evostate,
params:
EvoParams,
EveState:
"""'tell" update to RS state. "m
weights stath.weights
x xliitness. aratnosssaagsortt)
# Reighted updates
weighted mean (weights x) sum (axis=0)
weighted_sigm jnp.sqrt( (weights
state state.mean)
sum (axis=0) 1e-06
mean = state.mean params. wwwatlraateeaaaeeeeeeeeeeeeeaeee mean (weighted_ mean - state.
mean)
sigma state. sigma params.. trate_sigma seighted_signt state. sigma
return state. replace (mean-mean, sigma=sigma)
Listing 1: Pseudo-Code Discovered Evolution Strategy.
23

---

Published as a conference paper at ICLR 2023
F.2
PSEUDO-CODE FOR LEARNED EVOLUTION STRATEGY
From typing import Tuple
import chex
import jax
import numpy as jnp
from flax import linen as nn
class ittentionneien (nn. Module)
*==sef--attentae layer for recombination weigats." """
att hidden dins: int =
compact def _aall_la_______ (self, X: chex.Array) -> chex.Array:
keys = nn n..enns att_iiddttttitnnttteeeeeee
(X)
queries = nn.De
htw.ii.att_iidde_ddtteee (X)
values Dense (1) (X)
A nn softmax (jnp.matmul matmul (queries, keys.T) / jqprtqrt (X. shape
weights = nn. n softmax tinp.matmu (A, values) squeeze squeeze )))
return weights None]
class kvoPathMLP (nn. Monulll
"""MLP layer for learning rate modulation based evopaths. "**
mip hidden dins:
enn.compact def _call_
self, path_e: chex.Array, path_sigma chex.Array, sime_embedd
chex.Array
) >> Tuple[chex.Array. chex. Chex.Array]:
timestamps jnp. repeat (jnp. expand (time_embed, axis=0),
zepeats-path_
axis=0)
X = jnp. concatenate (lpath_c, path sigma, timestampsl, axis=i)
hidden jax. vmap
(sel hwwp_hiidenndiiiieee
in_axes- (0) (x)
hidden nn relu (hidden)
Irates siigssssgaa... sigmoid (nn. (1) (hidden) sqqueeze
Irates_mean nn.
sigmoid (nn. Dense (1) (hidden) squeezee
return prates_mean, prates_signt
class LES (cbjJct)):
"""oopulation Invariant Black-Bex Evolution strategy. "".
def ininii isel__ popsize: int, num_dims: int, les_net_ppraan, chex.
ArrayTres):
seli.strategy seli.popsize popsize
"LES"
self.num dims num dims
self.evopath - EvolutionPath ( dind timescales-
jnp (rra 0.5, 0.91))
self.weight_layer Attentionweights (8)
self.1rate_layer = EvoPathMLP (8)
self.fitness_feett features FitnessFeatures ()
self.les_net_parce = jes_nee__paaee parmms
@property
def params_strategy -> EvoParams:
"""Return default paraneters of evolution strategy. ""n
return EvoParams (ret_params-sell 1es_net_ppaamms
24

---

Published as a conference paper at ICLR 2023
def initialize (self, rng: chex. PRNCKe,,, params: EvoParams) Evoštate:
" the evoluticn strategy. "**
init_mean = jax. random. uniferm(
rng,
(self. #self..um_dims,)
minval-params, iniiittttnn -nit_min, maxval=params init_max,
init_sigma params. sinitmimmmiiiti........iiiii.iia inp. ones (self. dims)
init_path_s self.evopath. initializei
hit_path_sigma waw.f.evcpath.initteee
return EvoState (mean-init_mean) sigma-init_sigme path_s- init_path__e path_ aeth__sima=iiit_pae pathiintttpttthh__ee sigma)
def ask
sclf, rng: chex. PRNCKCC, state: EvoState, params: EvoParams
-> Tuple/chox, Arpay Evoštate]:
*** 'ask" for parameter candidates to evaluate next. "**
noise jax. raadommnormen (rng, (self.popsi ze, self.num dims))
x state. noise sigma. reshape (1, self, celffnnmmddaaa x - inp.clippp (x, params.clip_n params. clmmmaxxll_x pp_max)
return x, state
def tell self, x: chex.Array. fitness: chex. Array, state: EvoState, params EvoParams,
EveState:
"""'tell" performance data for strategy state update. """
fit_re hew.fffttmsssfeauuee..ppp. (x, fitness, state.
best_ fast_fitness)
(state. gon_counter) time_embed welf.weight_layet.roo
weights self
params srrass.net_param. ("r.caammsrrrcbb..... reccemb fit_re
weight_diff (weights (x - state. sum (axis=0)
weight_noiss (weights * state.mean) / state.sigma) sum
ax)s=0)
path c wel..
weight_
= www..evepath. ste sigh_ weight_noisee
Irates_mest mean, 1rates_ sigma self.lrace layer.
params. net__praas____pppee [ratt__modulation") path_c, path_sigma, time_embed
weighted_ mean (weights x) sum (axis=0)
weighted jnp.sqrt(
(weights
mean)
(axis=0) 1e-10
mean_change = Irates_mean (weighted - state.meat
si digma__hannge change = crates_signt + Berighted_sigmn state.
mean mcate.mean mean_change.
si simma state. sigma sigma_change
mean jnp. (mean, params.clip_me
sigma jnp. clip 0, params. max)
return state. replace
mean-mean, sigma=sigma,
path_c-path_c, path_sigme-path_ske gen_counter=state wcw.rttttttcccccccttereeeeeeeeee 1,
Listing 2: Pseudo-Code Learned Evolution Strategy.
25


Reviews:
[{"review_id": "ddmmhYUr76", "reviewer": "Reviewer_RKfF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The update rules of Black-Box Optimization (BBO) methods are traditionally formalized by hand-craft heuristics, which is inflexible. To tackle this problem, this paper proposes to parameterize the update rules of a typical BBO algorithm, Evolution Strategy (ES), by a transformer, and use meta-learning together with BBO techniques to optimize it, obtaining the Learned Evolution Strategy (LES). Experiments on BBOB benchmark, classification on MNIST and continuous control tasks generally show the effectiveness of the proposed framework. Furthermore, this paper shows that the proposed LES can be optimized through a self-referential style.", "summary_of_the_review": "This paper proposes a new method to learn the update rules of ES, with relatively various experiments to show its effectiveness. However, the general idea of this paper, using learning techniques to replace the heuristic components in optimization methods, is not new, but another standard implementation (here the authors use transformer to represent the rules and optimize the parameters of transformer by CMA-ES). As listed in the weakness, the lack of comparison with advanced ES methods and the scalability issues also limit the contribution of this paper. ", "strength_and_weaknesses": "Strength:\n\nThis paper is well-written and easy to follow. Experiments show the effectiveness of the proposed method from various views. \n\nWeakness:\n\n1. Even that the learning of update rules in ES has not been studied in a targeted way, the general idea of using learning techniques to replace the heuristic components in optimization methods is not new [1], which limits the novelty of this paper. I did not find much new insight from this work. There have been many works to learn hyper-parameters of some complex optimization methods.\n\n2. The authors should clearly explain and show the advantage of the proposed method (i.e., using a transformer to represent the update rule of ES and applying again BBO (e.g., CMA-ES) to optimize the parameters of the transformer) over other general ways of learning the heuristic rules. Can you show the benefit of using transformer empirically? How about the cost of the proposed method, i.e., training the transformer? As mentioned in the paper, the self-attention mechanism scales quadratically in the number of population members (in section 8). Meanwhile, it seems that a relatively large population size is important (in figure 4), which may limit the application of the proposed method.\n\n3. More advanced ES methods besides OpenES and SepCMA like [2-4] are expected for comparison to show that the learning of the update rules can obtain a competitive performance w.r.t. SOTA ES methods.\n\n[1] Marcin Andrychowicz et al, Learning to Learn by Gradient Descent by Gradient Descent, NIPS’16.\n[2] Guided evolutionary strategies: Augmenting random search with surrogate gradients. ICML’19\n[3] From complexity to simplicity: Adaptive ES-active subspaces for blackbox optimization. NeurIPS’19\n[4] Self-guided evolution strategy with historical estimated gradients, IJCAI’20.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cio7EikFYlP", "reviewer": "Reviewer_cHGq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a method to learn the parametrization of an Evolution Strategy through the meta-training of a self-attention-based architecture. They demonstrate the generalization of their meta-training procedure to test tasks that are different from the meta-training tasks. They further analyze the limitations of their proposed algorithm and deduce an interpretable parametrization of the ES.\n\nThe approach uses a self-attention mechanism to learn the parameters of an ES as well as an LSTM architecture to learn the learning rate of the ES parameter updates from one generation to another. Their approach is by construction invariant to reordering. It also allows them to meta-train on multiple dimensions, which can then be leveraged at test time.", "summary_of_the_review": "Please see above.", "strength_and_weaknesses": "Strengths: \nThe paper is easy to follow, which is excellent for a meta-training paper, with limited jargon and many implementation details. The strengths of the paper don’t lie in the idea itself but rather in the study of its extensions and limitations. This is a well-studied idea, and the experiments illustrate well the points the authors try to make, should it be about interpreting the LES or self-referencing of the LES. In particular, the study of the meta-training task distribution is very insightful and has not been done in many meta-learning papers. Noticeably, the authors can interpret the LES to devise a Discovered ES that works in some of their experiments.\n\nWeaknesses: \n\nSome weaknesses are outlined by the paper itself, such as scaling the self-attention mechanism or that this algorithm works only for diagonal Gaussian ES.  It is worth pointing out also that the features used in the self-attention mechanism F_t are manually created, i.e. it is a choice made in the algorithm that the self-attention would consider these only three statistics derived from fitness. This works in practice, as shown in experiments but looks like a significant limitation in terms of generalization. The authors mention this as a limit to their approach, but the manual engineering of these features is another.\n\nQuestions: \n- Would it be possible to replace the manually engineered features F_t with a learned feature map? \n- Would it be possible to relax the diagonal assumption of the Gaussian covariance matrix but use, for instance, a sparsity regulator to keep it maybe sparser than full? \n- In the experiments, the authors show that the method can adapt to scenarios unseen in the meta-training task distribution and generalize to higher dimensions and different loss surfaces. However, there are still, of course, tasks in which it performs poorly. Of course, one cannot expect a meta-model to perform perfectly on all tasks, and any such pretrained model will suffer from distributional shift, and these cases call for a fine-tuning procedure. How could this approach be fine-tuned, and how long would it take to adapt it for instance, to a task on which it performs poorly straight out of meta-training?\n- How do you explain the difference in Brax Performance in Figure 3 bottom middle and right between the medium budget and large budget on tasks like hopper and ur5e or walker2d? Otherwise, the experiments are well-designed and conclusive. The ablation studies are particularly insightful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X5PxaDhGpEj", "reviewer": "Reviewer_Fevx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a self-attention based meta learning framework for the learned Gaussian ESs and tests the performance on neuroevolution tasks. The metaBBO module includes steps like Meta-sampling, Inner loop search, meta normalize, and Meta updating. Limited evaluation results on standard datasets were provided. \n", "summary_of_the_review": "A meta learning framework is extended for Gaussian ESs without very exhaustive set of experiments and any theoretical results. In my view, the paper does not meet the high standards expected for ICLR. ", "strength_and_weaknesses": "Strength: The paper addresses a timely and ambitious research topic. \n\nWeakness: \n\nStatements like \"We show that metaevolving this system on a small set of representative low-dimensional analytic optimization problems is sufficient to discover new evolution strategies capable of generalizing to unseen optimization problems, population sizes and optimization horizons\"  - are very strong and have not been at all supported through any theoretical treatment. The generalization of LES to unseen neuroevolution tasks even in disparately high dimensions is not explained via theory of insightful empirical evidences. \n\nRelated works ignored:\n\nVishnu TV, Pankaj Malhotra, Jyoti Narwariya, Lovekesh Vig, and Gautam Shroff. 2019. Meta-Learning for Black-Box Optimization. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16–20, 2019, Proceedings, Part II. Springer-Verlag, Berlin, Heidelberg, 366–381. https://doi.org/10.1007/978-3-030-46147-8_22\n\nChoice of the functions in table A.1 for training is not adequately justified by analysis of the funcional landscape features.\n\nIt is not clear how the method achieves diversity in meta-training task distributions, no theoretical treatment was included. \n\nThe results were not compared against the best known algorithms in the pertinent areas from other genre. \n\nWhy were the results not validated through appropriate non-parametric statistical tests?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1_zhEuWP3NF", "reviewer": "Reviewer_Pfze", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method for Evolutionary Search (ES) that works by sampling hyperparameters for Learned Evolutionary strategies (LES) over different tasks and then uses self-attention to compute the recombination weights. The authors compare this approach to Handcrafted ES baselines (openES, SNES and sep-CMA-ES), and ablate their method to study the impact of the different components of Neural Networks. Based on this ablation, they derive a heuristic which is incorporated into an algorithm called Discovered Evolutionary Strategy (DES). Finally, the authors bootstrap LES using LES (argued as self-reference).", "summary_of_the_review": "The idea is novel and interesting and I definitely see the relevance to the meta-learning community. Evolutionary strategies have always been an interesting direction to gradient-based optimization, and so, I feel the research direction of end-end meta-optimization in a self-referential manner is interesting, to say the least. However, I feel the work is not yet ready for publication. This is primarily due to missing stronger baselines. I think the rationale for using a self-referential loop i.e. why should we use implicit models to begin with needs to be sufficiently justified with comparisons against static and dynamic baselines and that is currently lacking from the work.", "strength_and_weaknesses": "### Strengths:\n- The approach of using self-referential mechanisms for evolutionary optimization is interesting and seems to lend itself naturally to the ways in which evolutionary methods perform search and exploration.\n- The experiments have gone in interesting directions, and have been shown n a clear and concise manner\n- The literature has been covered sufficiently\n\n### Weaknesses:\n- Set transformers seem to be the differentiating factor here. It is not clear how the proposed method explicitly relates to Set Transformers, except for the invariance property being leveraged here. Is that a sufficient condition for any method that uses attention to be considered a set transformer? I would propose explicating this in a more self-contained manner \n- The central question seems to be: Can an end-end method perform better than some other methods, static and dynamic, in optimizing black box functions? The comparison has been performed only on static baselines (NES replaces points with distributions, openES uses ES for control problems and RL, and sep-CMA-ES uses diagonalization). None of the baselines are dynamically tuning hyperparameters of the inner loop (something that the BB method does). An example of potential baseline in this direction could be Dynamic Algorithm Configuration (DAC) (https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/20-PPSN-LTO-CMA.pdf). In general, having a dynamic baseline would make the experimentsstronger.\n- The rationale for using the LSTM for adjusting learning rates is not clearly explained, Since we see that  fixed LR ends up performing as well as the dynamic ones. Additionally, since we see that learning rates learned by an LSTM are not having a significant impact when compared with fixed ones, what I am not sure of is whether a learning rate schedule policy would perform better or not (https://arxiv.org/pdf/2007.04223.pdf).   \n- The selection of hyperparameters : T=50, N=16, D, z , e.t.c - is not explained. How did the authors decide these to be the relevant ones, and is the method sensitive to these values? Considering the argument for an end-end method, I think this question becomes more relevant\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster