NeuroEvoBench:  Benchmarking Evolutionary Optimizers for Deep Learning Applications
Authors: Robert Tjarko Lange
Yujin Tang
Yingtao Tian
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Evolution
Evolution Strategies
Evolutionary Optimization
Benchmark
Neuroevolution
Black-Box Optimization
TL;DR: We introduce a benchmark of Evolutionary Optimization methods tailored to machine learning applications.

Abstract: Recently, the Deep Learning community has become interested in evolutionary optimization (EO) as a means to address hard optimization problems, e.g. meta-learning through long inner loop unrolls or optimizing non-differentiable operators. One core reason for this trend has been the recent innovation in hardware acceleration and compatible software -- making distributed population evaluations much easier than before. Unlike for gradient descent-based methods though, there is a lack of hyperparameter understanding and best practices for EO – arguably due to severely less `graduate student descent' and benchmarking being performed for EO methods. Additionally, classical benchmarks from the evolutionary community provide few practical insights for Deep Learning applications. This poses challenges for newcomers to hardware-accelerated EO and hinders significant adoption. Hence, we establish a new benchmark of EO methods (NEB) tailored toward Deep Learning applications and exhaustively evaluate traditional and meta-learned EO. We investigate core scientific questions including resource allocation, fitness shaping, normalization, regularization & scalability of EO. The benchmark is open-sourced at https://github.com/neuroevobench/neuroevobench under Apache-2.0 license.

NeuroEvoBench: Benchmarking Evolutionary Optimizers for Deep Learning Applications
Robert Tjarko Lange' Technical University Berlin Science of Intelligence Cluster of Excellence
Yujin Tang Google DeepMind
Yingtao Tian Google DeepMind
Abstract
Recently, the Deep Learning community has become interested in evolutionary opti- mization (EO) as means to address hard optimization problems, e.g. meta-learning through long inner loop unrolls or optimizing non-differentiable operators. One core reason for this trend has been the recent innovation in hardware acceleration and compatible software - making distributed population evaluations much easier than before. Unlike for gradient descent- -aseed methods though, there is lack of hyperparameter understanding and best practices for EO arguably due to severely less "graduate student descent" and benchmarking being performed for EO meth- ods. Additionally, classical benchmarks from the evolutionary community provide few practical insights for Deep Learning applications. This poses challenges for newcomers to hardware-acceleratee EO and hinders significant adoption. Hence, establish new benchmark of EO methods (NourcEv)Beench) tailored toward Deep Learning applications and exhaustively evaluate traditional and meta-learndd EO. We investigate core scientific questions including resource allocation, fitness shaping, normalization, regularization & scalability of EO. The benchmark is open-sourced at httpt:://// httppp//gggg.p.cc./////////////p der Apache-2.0 license.
un-
Introduction
The Deep Learning revolution has been largely enabled by the (arguably) "unreasonable effective- ness" of gradient descent (GD))based optimization in high-dimensional search spaces of parame- ters (Kleinberg et al.) 22118) Hardt et al., 2016 Ge et al. 2015). However, a plethora of challenging high-dimensional optimization problems where GD methods are inadequate exist, including not only hyperparameter search but also the optimization of con-differentiable operators (e.g. objective or
architecture, Tian and Ha 2022, He et al.) (02T). the computation of ill-behaved gradients through long computational graphs (Peyré et al. (017), and applications requiring black-box optimization (Metz et al. 2021). Such challenges have sparked recent resurgence of interest in modern scalable EO methods by the machine learning community. Still, compared to widely-studied GD optimizers, there remains lack of intuition for EO methods. We argue that this is due to three reasons:
Hardware lottery phenomenon (Hooker 2021): Until recently neuroevolution experimentation required a significant amount of engineering overhead caused by the orchestration of parallelized population rollouts (e.g. using Dask or Ray, Moritz et al.) 2018). This has obstructed replicability and hindered EO from leveraging the recent advances in hardware acceleration.
Lack of 'graduate-student' descent (Gencoglu et al. 2019): The last decade has seen strong increase in GD optimization practitioners. This facted the discovery of robust hyperparameter basins that perform well standard research tasks. EO, the other hand, lacks behind in human
Work was partially done while being at Google DeepMind. Contact: robert. langeêtu- berl in de
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Interquertil Meen (IQM)
Evolutionary optimizer
Experiment type
Problem/task
GESMR-GA
SAMP-GA
BBO
Evolution strategies
Grid eval of hyperparams
Sep CMAES
Control
Genetic algorithms
Multi-sead tuned eval
LEB
Vision
Learned EvoOpt
Random search tuning
LGA
Sequence
SimpleG4
Tuning protocal
SNES
Default
Small
Medium
Large
ARS
Default
1-20
21-40
41-50
PGPE
Configuration
trials
Constrain search space
OpenES Performance
Figure I: Proposed NeuroEvoBench benchmark. Left. of problems, different classes of EO and experiment types including a random search tuning protocol with different budgets (default, small, medium, large). Right. Aggregated normalized median performance across EO algorithms on nine neuroevolution tasks, sorted by their median performances and across seeds. Genetic Algorithms are generally outperformed by Evolution Strategies.
The benchmark is composed of four classes
capital investment and sufficient hyperparameter intuition. This has been partly caused by engineering challenges and the lack of adequate hardware acceleration and the required software.
Missing ML--relvan EO benchmarks: The majority of existing benchmarks focus on synthetic functions and standard black-box optimization tasks (e-g. BBOB, [ansenn 2009). But neural network fitness landscapes can fundamentally differ from these settings, limiting the meaningfulness of any drawn conclusions for relevant practical cases of neurocvolution. This is due differences in the nature of considered problems (e.g. robot policy parameters) and the size of the search space.
However, recent advances have shifted the balance. For example, have seen tremendous advances in both hardware acceleration (e.g. GPU/TPU) well as general- purpose frameworks/nfrasttucce that facilitate smooth development of distributed programs (e.g. PyTorch and JAX, Bradbury et al.) 2018). These developments enable not only data- and poreel-paallelism for GD-based training pipelines but also large-scale EO with auto-vectorized and device-parallel population evaluations, e.g. EvoJAX (Tang et al.] 2022) and evosax (Lange) 2022a). This makes it the right time to reassess the uncertainty about HO use cases, problem-dependent HO choices, and optimal hyperparameter settings. We. therefore, introduce NeuroEvoBench (NEB), benchmark for EO methods tailored to relevant Deep Learning applications and leveraging such advances in the hardware and software stack. This allows for effective scientific validation of new algorithmic improvements low iteration times. As shown in Figure I our proposed benchmark contains four classes of different, selected black-box optimization tasks, each with three experiment types including random search tuning protocol with different budgets. Using NEB, we can investigate several core scientific questions, such the performance of different EO, and the impact of different decisions such as population size, and dimensionality, to give the right picture for future reference. Our contributions are summarized as:
1. We construct hardware-acceelrated EO benchmark providing relevant scientific insight for the ML community. It consists of selected black-box optimization tasks and a standardized experiment protocol. We execute the benchmark for 10 EO methods including both Evolution Strategies (ES) and Genetic Algorithms (GA) and analyze their robustness.
2. We open-source the benchmark for a broader adoption, to provide evidence for future analysis and make benchmarking new EO method easier. Furthermore, we make a quick overview for reference available online: oo

---

3. We investigate several core scientific questions with regard to resource allocation, gradient optimizer choice and regularization techniques. We further investigate the scaling of EO with respect to parameters, population size and the number fitness evaluations per candidate.
Related Work & Background
Gradient-Basd Base Optimization Benchmarks. The success of Deep Learning methods has arguably been enabled by the "unreasonable" success of GD- -bsedd optimization (Kleinberg et al. 2018 Hardt
al. 2016 Ge al. 2015) and the adoption of standardized benchmark tasks (e-g. ImageNet, Deng
et al. 2009). Furthermore, there exist several benchmarks for gradient- gradient-based descent optimization,
e.g. Schaul et al. (2013) Schneider al. 2019 Metz et al. (2020) which vary in the runtime and number of optimization tasks. However, these benchmarks consider only subspace of imaginable solutions due relying on explicit well-behaved gradient evaluations and not natively supporting distributed population rollouts required for EO methods. EO, the other hand, deals with problems with fewer restrictions, e.g., long-horizon tasks or non-differentiable dynamics.
Black-Box Optimization Benchmarks. For BBO methods, on the other hand, the benchmarks have largely focused on synthetic functions or problems. e.g. the BBOB (Hansen 2009). HPO-B (Arango et al. 2021), nevergrad (Bennet et al. 2021) and Vizier (Golovin et al. 2017) benchmarks. This limits their relevance for large-scale BBO problems and Deep coariig--ppccinnn practitioners. Here we aim to overcome this limitation by introducing NeuroEvoBench which explicitly targets neuroevolution tasks of varying scale (parameter count, problem domain, resource constraints, etc.). Finally, Mousavirad et al. (2020) introduce a benchmark evaluating meta-heuristic algorithms on neural network tasks. Unfortunately, they do not consider ES/GA and the code closed-source.
Evolutionary Optimization. EO constitute a set of random search algorithms based on the principles of biological evolution. In this benchmark, we focus on two classes of EO methods:
Evolution Strategies (ES): ES adapt a parameterized distribution (e.g. multivariate normal) to iteratively search for well performing solutions. After sampling a population of candidates, their fitness is estimated using Monte Carlo (MC) evaluations. The scores are used to update the search distribution in order maximize the likelihood of well well-performing candidates.
2. Genetic Algorithms (GA): Unlike ES, GA does not rely on a uni-modal search distribution. Instead, they keep an archive of 'parent' solutions from which new 'children' candidates are sampled and adapted via mutation. After evaluation, well cell-performng children are selected to replace parents. GAs differ in their replacement and mutation rate adaptation strategy.
Accelerated Neurvevolution. Traditionally, EO has required substantial amount of software engineering to orchestrate parallel evaluations of fitnesses polulation. We argue that this additional overhead has largely held back EO progress. NeuroEvoBench aims leverage set of software improvements that facilitate auto-vectorized and device parallel evaluations using the JAX (Bradbury accelerated fitness evaluations implemented by EvoJAX (Tag et al. 2022) and EO algorithms from evosax (Lange, 2022a). These advances in turn significantly reduce the benchmark runtime and allow for seamless GPU/TPU acceleration.
al. 2018) ecosystem. More specifically,
3 NeuroEvoBench: Problems, Optimizers, Fitnesses and Experiment Types
We introduce the NeuroEvoBench benchmark (see Figure to provide insights into the performance of EO algorithms for Deep Learning problems. Our design strongly based the choices made by Schmidt et al. (202T] GD case). At its core a NeuroEvoBench experiment combines a problem with an Evolutionary Optimization algorithm and a fitness shaping operation:
Problems
Optimizers
Fitnesses raw
Experiment Types
Pl
OpenAI-ES
Random Search
P2
PGPE
z-score X
Multi-Seed Eval
ranks
Grid search
PI1
CMP-CMA-ES
[[-1, 1J norm
3

---

More specifically, they include different optimization substrates, fitness functions, and optimization resource budgets, i.c. population sizes and number of stochastic evaluations We detail each component of NeuroEvoBench below (see also Table Tb.
3.1 Problems
NEB implements set of 11 problems from core classes chosen cover wide range of settings. All the tasks are implemented JAX (Bradbury et al. 2018) and thereby can be directly evaluated accelerators, speeding up the parallel population fitness computation significantly.
BBO For comparability, we host a set of BBO benchmark tasks including the noiseless BBOB (aaseen 2009) and the HPO-B (Arango et al. 202 I continuous) sweep. These are concerned with optimization of analytical functions with different characteristies (multi-modal, conditioning, etc.) and surrogate functions extracted from hyperparameter settings of small machine learning models.
set of control Reinforcement Learning tasks including continuous control (Brax, Freeman et al., 2021] 1) tasks using small Tanh MIP policies and MinAtar (Young and Tian 2019 Lange 2022b) visual control tasks using CNN for policies. The fitness score is computed as the cumulative episode return obtained by the parametrized policies.
Control We evaluate EO performance
Vision We consider set of vision classification and generation tasks including MNIST image generation (via a MLP VAF, Kingma and Welling 2013)), Mashion-MNIST (2 layer CNN), and CIFAR-10 (All-CNN-C variant) classification. The training fitness used the VAE and cross-entropy loss on the training set, while the evaluation used the test set and accuracy.
Sequence We make use of two sequence prediction tasks: The addition regression (L et al. 2015) task using GRU RNN and Sequential MNIST classification (LSTM, Hochreiter and Schmidhuber (1977). Both tasks test the ability of EO in optimizing systems with exploding/vanishing gradients.
Table 1: Summary of problems used in our benchmark experiments. Note that problems and consider surrogate optimization tasks/models (Hansen 2009) Arango et al., 202T), while all other tasks are concerned with the evolutionary optimization of neural network weights. Throughout the main text, we focus on such tasks but implement the BBO problems for completeness.
Data
Model
Task
Metric
Obj.
Pop.
Gens.
Time
Dim.
P1
НPО-B
HPO
Surrogate
Perf.
Perf.
4
100
5m
20
P2
BBOB
BBO
Synthetic
Perf.
Perf.
32
100
2m
< 50
P3
Ant
MLP
Control
Return
Return
512
2k
30m
4136
P4
Fetch
MLP
Control
Return
Return
512
2k
30m
4650
P5
Asterix
CNN
Control
Return
Return
256
1.5k
1h
51989
P6
Breakout
CNN
Control
Return
Return
256
1.5k
1h
51923
P7
MNIST
VAE
Generate
Loss
Loss
256
10k
7m
52984
P8
F-MNIST
CNN
Classify
Acc.
Loss
128
4k
5m
11274
P9
CIFAR-10
ABENNCC
Classify
Acc.
Loss
128
2.5k
1h
3994
P10
Addition
GRU
Regress
Loss
Loss
128
5k
5m
3425
P11
S-MNIST
LSTM
Classify
Acc.
Loss
512
3k
1h
10090
Our task selection is motivated by the observation that small-scale BBO benchmarks alone (e.g. BBOB/HPO-B] do not suffice in predicting the performance of EO methods on high-dimensional tasks requiring the optimization of network weights (see comparison of Figures T [nd TTp. Furthermore, the different task classes cover wide range of representative Deep Learning problems required for robust performance evaluation (see Figure
3.2 Optimizers and Fitnesses
Concretely, WeuroEvoBench focuses on ten competitive BO selected from the vlasses of ES and GA that represent different trade-off between exploration and exploitation in parameter space (Table 2
'The majority of considered tasks were chosen to be evaluated within less than 30 minutes (estimated on single V100S GPU). This allows for rapid evaluation of parameter tuning in <1 day of sequential execution time.

---

Table 2: Summary of Evolutionary Optimizers used in benchmark experiments.
ES1
ES2
ES3
ES4
ES5
EO Name
OpenAI- ES
PGPE
ARS
SNES
Sep -MM
ES
Reference
Salimans al
2017
Sehnke et al
2010
Mania et al.
2018
Schuul al 2011
Ros and Humen
2008
EO Type
FD-GD
FD-GD
FD-GD
FD-GD
boD
ES6
GAT
GA2
GA3
GA4
EO Name
LES
Gaussian-Ge
SAMR GA
GESMR
LGA
Reference
Lange 2002
Rechenberp 1976
Clunc 2008
Kumar
2022
al
3023
EO Type
Meta-Fol)
SA-GA
SA
Meta-SA GA
Finite Difference-based ES: A subset of ES random perturbations to MC estimate finite differ- ence gradient to the fitness function, F(.):
1
VoE F(0 + F ac)d
This estimate is then used along standard GD-based optimizers to refine the search distribution mean ES vary in their use of fitness transformation, anti-correlated noise, elite selection and covariance. Estimation-of Dsdimationoff-Distriuutio ES: A second class of ES does not rely on noise perturbations or low-dimensional approximations the fitness gradient. Instead, algorithms such as Sep-CMA-ES (Ros and Hansen 2008) rely on elite-weighted mean updates and iterative covariance estimation.
Gaussian Adapttidapt GA: We consider GAs that utilize Gaussian isotropic perturbations to the sampled children and do not use cross-over. We evaluate a simple GA with top-k elitist parent replacement (Rechenberg) 1978) as well two GAs that adapt the mutation rates based on the parent performance (SAMR-GA. Clune et al. (2008) and GESMR-GG --GA, Kumar et al. (2022)).
study the performance of two meta-learned EO algorithms (Learned ES, [Lange al. (2022) and Learned GA. Lange et al. (2023)). Instead of relying on manually designed update rules, learned EO leverage self-attention to parameterize novel families of EO al algorithms. The corresponding parameters are meta-evolved on small set of BBO problems.
Learned Evolutionary Optimization:
3.3 Experiment Types
Random search with space refinement. The core experiment of NeuroEvoBench is random search tuning run with different budgets. More specifically following SSchmidt al. (022)) we consider four settings: Default parameter configuration, small 20 trials, medium 40 trials and large - using 50 trials with search space refinement after 40 runs using the best hyperparameter settings. We provide additional information on the tuning ranges in the appendix.
Multi-seed re-evaluation of tuned hyperparameters. After completing the random search tuning, we re-evaluate the best configuration on multiple random seeds. This tests the robustness of the tuning procedure the stochasticity of the training run. Whenever applicable, we make use of the robust evaluation metrics introduced by the rliable library (Agarwal et al. 2021).
Grid search evaluation of core settings. Finally, we evaluate certain hyperparameter settings more explicitly by running a grid sweep. These include different fitness score transformations for the FD-GD ES methods, mean decay regularization, and resource settings (population/lvaluution)
3.4 MeuroEvoBench Task Evaluation API
from evosax import Strategies from neuroevabench. problems.....ppo.......o.....eee cifar import CifarPolicy from neuroevobench. problems cifar import CifarTask
from neuroevobench.c www.lleennchproolmssssssmm wroblems.ccie.. Define the task-specific network policy policy CifarPolicy()
import CifarEvaluator
Instantiate the train and test task train_task CifarTask test_task CifarTask (test_ dest_batch_siee test=True)
test=False)
5

---

Instantiate the task evaluator and run the search loop evaluator CifarEvaluator policy, train_task, test_task, popsize, strategy, ...) evaluator run (num_generations eval_every-gen
Listing 1: Example for NeuroEvoBench task evaluation API for CIFAR-10 Classification.
We provide easy-to-use API for NeuroEvoBench. An example of CIFAR-10 <Krizhevsky et al., 2009) classification is shown Listing 11 Our API expects three ingredients: The 'Policy' (neural network/agent architecture), the 'Task' (acceleraaee//disttibteed fitness evaluator) and the 'Evaluator' (the iterative training/generation loop). Plus, the EO algorithm only needs to follow the standard ask-tell API used for example in evosax (Lange 2022a
4
Results
FMNIST Classification [CNN]
MNIST Generation [VAE]
CIFAR1D Classification [CNN]
-200
Test Test Accuracy
Test Test Accuracy
0.6
-300
0.4
PGPP
Open
SAMB GA
SimpaeAA
Neg.
-400
0.2
ARS
GESMR .GA
0.1
1000
2000
3000
4000
5000
1000
2000
Ant Control [MLP]
Fetch Control [MLP]
Asterix Control [CNN]
4000
3000
Episode Return
Episade Return
Episode
1000
600
1000
1500
2000
500
1000
1500
2000
1000
Breakout Control [CNN]
[CNN]
Addition Regression [GRU]
SMNIST Classification [LSTM]
Episode Return
Neg. Test Loss
Test Accuracy
15
10
-2
1000
2000 Generations
4000
2000 Generations
Generations
Figure 2: NeuroEvoBench task evaluation after 50 trials of random search. We plot the tuned performance all 10 considered EO for the 9 neuroevolution tasks. GAs are generally outperformed by ES alternatives, but there is no clear winner across all considered ES. The results averaged over 5 independent runs & we plot standard error bars.
We conduct a set of experiments with MeuroEvoBench and discuss the lessons learned. For this, we focus the neuroevolution tasks (P3-P11) in order provide general insights to the ML community.
4.1 Given predefined tuning budget, which EO performs best?
there single EO algorithm that dominates all others across neuroevolution tasks? In Figure 2 we plot the performance of the 10 considered EOs for the 9 neuroevolution tasks after random search

---

tuning (see Section 2 for more details). We find that there is no single clear winner (see also Figure for the aggregated results) across all settings. Furthermore, we observe that the different Genetic Algorithm variants are generally outperformed by the ES competitors. Finally, while the meta-learned EO algorithms can outperform the manually designed alternatives on single tasks (e.g. Fetch and Ant control), they appear to be only capable of limited generalization beyond their meta-training setting.
FMNIST Classification [CNN]
MNIST Generation [VAE]
CIFAR10 Classification [CNN]
0.8
-200
0.4
Test Accuracy
OpenES
SimpWGG
Neg. Test Loss
Accuraac
0.6
POPE
BAWR-GA
-300
ARB
DEBAIR-DA
0.3
ONIS
LGA
0.4
SSSCMAIES
-400
0.2
0.2
0.1
Ant Control [MLP]
Fetch Control [MLP]
Asterix Control [CNN]
4000
20
Eplaode Return
3000
Episode Return
6
Episode Return
15
5
2000
10
1000
5
0
Breakout Control [CNN]
Addition Regression [GRU]
SMNIST Classification [LSTM]
30
0.0
0.6
Episode Return
Neg. Test Loss
Test Test Accuracy
20
0.4
0.2
20
40
20
40
20
40
Search lterations
# Search lterations
# Search Iterations
Figure 3: NeuroEvoBench best task performance across 50 trials of random search. We plot the maximum performance tracked across the random search procedure for all 10 considered EO and for the 9 neuroevolution benchmark tasks. We consider single random run for the tuning process.
4.2 How much tuning is required to achieve competitive performance?
Next, we investigated the impact of the tuning budget on the performance of the EO. Ideally, EO do not require too much tuning in order to perform well across different classes of tasks, In Figure we find that indeed most of the considered optimizers find their peak performance within the small tuming budget and less than 20 random search tuning trials. We note that for most tasks a single evaluation of the hyperparameters appears to provide robust performance estimate. Only for the Asterix and Breakout tasks we observed a significant drop in the estimated performance when re-evaluating the best performance over multiple random seeds (see Figure
4.3
What are general EO recommendations and their scaling properties?
We investigate a set of core considerations when performing Evolutionary Optimization: Resource allocation, the effect of stochastic evaluation fitness evaluation, fitness transformations and regular- ization techniques. More specifically for this exercise, we focus on OpenAI-ES (Salimans et al. 2017) as a popular representative of finite-differnne ES. EO rely on stochastic fitness evaluations of the individual population members. This introduces explicit choice of resource allocation: Shall
7

---

Noise Level: 0.0
Noise Level: 0.1
Noise Level: 0.2
MC Filness Evaluations
32
4000
16
Performance
-3500
-300
2
512
256
512
512
Population Size
Population Size
Population Size
Figure 4: Resource allocation for EO. For noisy Ant task find that candidate evaluations lead to better downstream performance but require more hardware memory. This is independent of the external noise level. The results averaged over 3 independent
Fetch - Optimizer
F-MNIST Optimizer
Addition Optimizer
0.86
0.0
Episode Retum
Test Accurcay
-0.2
0.82
Neg. Test
0.80
9CD
-0.6
Adam
-0.8
Cipppp
0.76
Adan
-1.0
500
1000
1500
2000
1000
2000
3000
4000
2000
4000
Fetch Mean Decay
F-MNIST Mean Decay
Addition Mean Decay
Episode Retum
Test Acouracy
0.8
Neg. Test Loss
-0.5
0.7
-0.15
2
0.6
500
1000
1500
2000
1000
2000
3000
2000 # Generations
Generations
Generations
Fetch Fitness Trafo
F-MNIST Fitness Trafo
Addition Fitness Trafo
0.00
Episode Retum
Test Test Accuracy
Neg. Test Loes
0.7
-0.10
Owtl Rank
2:5core
0.6
[-1.1
1-1.1/Karm
Roer
-0.20
500
1500 Generations
1000
2000
1000 2000
3000
4000
2000
4000
Generations
Generations
Figure 5: Impact of optimizer choice, mean decay and fitness transformation OpenAI-ES perfor- mance. OpenAI-ES largely robust to the optimizer choice, while decay and transformation are task-dependent. The results are averaged over 3 independent runs & we plot standard error bars.

---

one evaluate more parameter candidates at the cost of a potentially less informative performance estimate? Intuitively, this may related to the inherent noise associated with the considered task. Noiseless tasks trivially do not require multiple evaluations, but how to assign evaluations in the face of uncertainty? We set out to investigate this question by introducing additive Gaussian noise to the return evaluation in the Ant control task. In Figure we find that it is always beneficial to prefer larger population over an increased number of evaluations regardless the external noise level. How do the gradient optimizer choice, mean multiplicative decay regularization and fitness normal- ization transformation affect the performance of OpenAl-ES? We consider set of standard GD optimizers as well as ClipUp (Toklu et al. 2020h, which was designed for EO. For Fetch Adan (Xie) et al. 2022 outperforms the competitors by slight margin. Otherwise, in Figure we observe that OpenAT-E5 performs robustly across optimizers for three of the NEB tasks.
In line with GD-based optimization we see that too much mean decay (0e+1 -1 B regularization can be hurtful, while small values generally improve test performance. Centered rank fitness transformation (Salimans et al. 2017) provides good default setting, while for the Fetch task improvements can be achieved by using the raw return fitness score instead.
Finally, we investigate the scaling capabilities of OpenAI-ES for two key variables: Model and population size. In Figure / we evaluated the performance of control policies, CNN classifier and GRU networks for varying numbers of optimization parameters and increasing populations (given a fixed number of EO generations). We find that an increased population size always leads to better EO performance. An increase in model capacity, on the other hand, does not always lead to an improvement, which may be due the inherent required task solution and the curse of dimensionality.
Fetch
F-MNIST
Addition
6
Episode Return
Test Accuracy
0.88
Neg. Loss
-0.02
0.84
Unig Uvis:
-0.03
0.82
129
256
512
128
256
512
Population Size
Population Size
Population Size
Figure 6: Scaling results for EO. We consider the performance behavior of OpenAI-ES with varying number of population members and model size. Larger populations are always beneficial, while the required model capacity dependen... The results are averaged over 3 independent runs.
5 Discussion
Summary. We introduce NeuroEvoBench a benchmark targeting the rigorous evaluation of Evolutionary Optimization algorithms in the context of neural network training. It allows for easy, programmatic, and automatic investigation into wide range of different BBOOneuroevolution tasks, types of EO, and budgets. Our initial investigation answers several core scientific questions regarding resource allocation, fitness-shapie
regularization, and scalability of EO.
Limitations. We aim to support a representative selection of tasks and problems. Nonetheless, we acknowledge that the actual real-world workload can represent much more comprehensive range of scenarios. Going forward and as EO methods increase in capabilities, we envision a gradual enhancement and update of the task sweet and evaluation protocols.
Ethical Considerations. As a benchmark paper, we do not identify particular issues per se. Since we expect our work to be used by the community, call for range of considerations, including those of potential implicit bias in the task selection, evaluation protocols and result interpretation.
Future Work. We acknowledge that benchmarks have to be 'living projects' requiring constant refinement and updating with respect to the community's needs. We plan to continuously add more results for EO supported by evosax (Lange, 2022a) and to provide modular and easy to extend benchmark protocol. Further investigations may include strategy restarts, asynchronous evaluation methods the influence of shared randomness.

---

References
AGARWAL, R., M. SCHWARZER, P. S. CASTRO, A. C. COURVILLE, AND M. BELLEMARE (2021): "Deep reinforcement learning the edge of the statistical precipice," Advances in neural information processing systems, 34, 29304 29320.
ARANGO, S. P., H. S. JOMAA, M. WISTUBA, AND J. GRAROCKA (2021): "HPO-B: A large-scale reproducible benchmark for black-box HPO based on OpenML,," " arXiv preprint arXiv:2 106.06257.
BENNET, P., C. DOERR, A. MOREAU, J. RAPIN, F. TRYTAUD, AND O. TEYTAUD (2021): "Never-
grad: black-box optimization platform," ACM SIGEVOlution, 14, 8-15.
BRADBURY, J., R. FROSTIG, P. HAWKINS, M. J. JOHNSON, C. LEARY, D. MACLAURIN, G. NEC-
ULA, A. PASZKE, J. VANDER S. WANDERMAN-MILNE, AND Q. ZHANG (2018): "JAX: composable transformations of Python+NumPy programs,".
CHOO, K. B. VAN MERRIENBOER, D. BAHIDANAU, AND Y. BENGIO (2014); "On the properties of neural machine translation: Encoder-decoder approaches." " arXiv preprint arXiv: 1259.
CLUNE, J., D. MISEVIC, C. OFRIA, R. E. LENSKI, S. F. ELENA, AND R. SANJUÁN (2008): "Natural selection fails optimize mutation rates for long-term adaptation on rugged fitness landscapes," PLoS Computational Biology, 4, e1000187.
DENG, J., W. DONG, R. SOCHER, L.-J. LI, K. LI, AND L. FEI-FEI (2009): "Imagenet: large- scale hierarchical image database," in 2009 IEEE conference on computer vision and pattern recognition, Ieee, 4482555.
FREEMAN, C. D., E. FREY. A. RAICHUK, S. GIRGIN, I. MORDATCH, AND O. BACHEM (2021): "Brax-A Differentiable Physics Engine for Large Scale Rigid Body Simulation," " arXiv preprint
arXiv:2106.1322 13281.
GF, R., HUANG, C. JIN, AND Y. YUAN (2015): "Escaping from saddle points-online stochastic gradient for tensor decomposition." in Conference learning theory, PMLR, 797-842.
GENCOGLU, O., M. VAN GIL.S, E. GULDOGAN, C. MORIKAWA, M. SÚZEN, M. GRUBER,
J. LEINONEN, AND H. HUTTUNEN (2019): "HARK Side of Deep Learning-Fromt Grad Student Descent to Automated Machine Learning," arXiv preprint arXiv:1904. NTtV:19004076333 07633.
GOLOVIN, D., B. SOLNIK, S. MOITRA, G. KOCHANSKI, J. KARRO, AND D. SCULLEY (2017): "Google vizier: A service for black-box optimization," in Proceedings of the 23rd ACM SIGKDD international conference knowledge discovery and data mining, 1487-1495.
HANSEN, N. (2009): "Benchmarking a Bl-population CMA-ES on the BBOB 2009 function testbed," in Proceedings of the I 1th annual conference companion on genetic and evolutionary computation conference: late breaking papers, 2389-2396.
HARDT, M., B. RECHT, AND Y. SINGER (2016): "Train faster, generalize better: Stability of stochastic gradient descent," in International conference on machine learning, PMLR, 1225-1234.
HARRIS, C. R., K. MILLMAN, S. J. VAN DER WALT, R. GOMMERS, P. VIRTANEN, D. COUR- NAPEAU, E. WIESER, J. TAYLOR, S. BERG, N. J. SMITH, ET AL. (2020): "Array programming with NumPy," Nature, 585, 357-362.
HE, X., K. ZIIAO, AND X. CIIU (2021): "AutoML: A survey of the state-of-the-art," Knowledge-
Based Systems, 212, 106622.
HOCHREITER, S. AND J. SCHMIDHUBER (1997): "Long short-term memory," Neural computation, 9, 1735-1780.
HOOKER, S. (2021): "The hardware lottery," Communications of the ACM, 64, 58-65.
HUNTER, D. (2007): "Matplotlib: A 2D graphies environment," IEEE Annals of the History of Computing, 9, 90-95.
KINGMA, D. P. AND J. BA (2014): "Adam: A method for stochastic optimization," arXiv preprint 112.4.1412.6980.
KINGMA, P. AND M. WELLING (2013): "'Aut-encoding variational bayes," arXiv preprint 112.6.1312.6114
KLEINBERG, B., Y. LI, AND Y. YUAN (2018): "An alternative view: When does SGD escape local minima?" in International conference on machine learning, PMLR, 2698-2707, KRIZHEVSKY, A., G. HINTON, ET AL.. (2009): "Learning multiple layers of features from tiny images.".
10

---

KUMAR, A., B. LIU, R. MIKKULAINENN AND P. STONE (2022): "Effective Mutation Rate Adaptation through Group Elite Selection," arXiv preprint arXiv:2204.04817.
LANGE, R. T. (2021); "MLEEInfrrastrucuur:: A Set of Lightweight Tools for Distributed Machine Learning Experimentation."
(2022a): "evosax: JAX- -X-based Evolution Strategies," arXiv preprint arXIv:2212.044180
(2022b): "gymnax: A JAX-based Reinforcement Learning Environment Library."
LANGE, R. T. T. SCHAUL,, Y. CHEN, C. LU, T. ZAHAVY, V. DALIBARD, AND S. FLENNERHAG (2023): "Discovering Attention-Based Genetic Algorithms via Meta-Black- Box Optimization," arXiv preprint 33Xx:22300.03995.
LANGE, R. T., T. SCHAUL, Y. CHEN, T. ZAHAVY, V. DALLIBARD, C. LU, S. SINGH, AND
S. FLENNERHAG (2022): "Discovering Evolution Strategies via Meta-Black-Box Optimization." arXiv preprint arXiv:2211.11260.
LE, Q. V., JAITLY, AND G. E. HINTON (2015): "A simple way to initialize recurrent networks of rectified linear units," arXiv preprini arXiv: 1504.00941.
MANIA, H., A. GUY, AND B. RECIT (2018): "Simple random search of static linear policies competitive for reinforcement learning." Advances in Neural Information Processing Systems, 31.
METZ, L., C. D. FREEMAN, S. S. SCHOENHOLZ, AND T. KACHMAN (2021): "Gradients are not all you need," arXiv preprint arXiv:2111.05803.
METZ, L.,, N. MAHESWARANATHAN, R. SUN, C. D. FREEMAN, B. POOLE, AND J. SOHL- DICKSTEIN (2020): "Using thousand optimization tasks to learn hyperparameter search strate- gies," arXiv preprint 1/Xiv:2002.//88))
MORITZ, P., R. NISHIHARA, S. WANG, TUMANOV, R. LIAW, LIANG, M. ELIBOL, YANG, W. PAUL, M. I. JORDAN, ET AL. (2018); "Ray: A distributed framework for emerging {AI} applications," in 13th (USENIX) Symposium on Operating Systems Design and Implementation
((OSDI) 18), 561-577.
MOUSAVIRAD, S. J., SCHAEFER, M. J. JALALI, AND KOROVIN (2020): "A benchmark of recent population based metaheuristic algorithms for multi-layer neural network training," in Proceedings of the 2020 genetic and evolutionary computation conference companion, 1402-1408.
PEYRÉ, G., M. CUTURI, ET AL. (2017): "Computational optimal transport," Center for Research in Economics and Statistics Working Papers.
RECHENBERG, I. (1978): "Evolutionssttaategien" in Simulationssmethoden in der Medizin und Biologie, Springer, 83-114.
ROs, R. AND N. HANSEN (2008): "A simple modification in CMA-ES achieving linear time and space complexity," in International conference on parallel problem solving from nature, Springer, 296-305.
SALIMANS, T., J. Ho, X. CHEN, S. SIDOR, AND I. SUTSKEVER (2017): "Evolution strategies as scalable alternative reinforcement learning," arXiv preprint arXiv:1703.0338
SCHAUL, T., I. ANTONOGLOU, AND D. SILVER (2013): "Unit tests for stochastic optimization," arXiv preprini urXiv:1312.000
SCHAUL, T., T. GLASMACHERS, AND J. SCIIIIHHUBR (2011): "High dimensions and heavy tails for natural evolution strategies," in Proceedings of the 13th annual conference on Genetic and evolutionary computation, 845-852.
SCHMIDT, R. M., F. SCHNEIDER, AND P. HENNIG (2021): "Descending through a crowded belley-benchmarking deep learning optimizers," in International Conference Machine Learning, PMLR, 9367-9378
SCUNEIDER, F., L. BALLES, AND P. HENNIG (2019): "DeepOBS: deep learning optimizer benchmark suite." arXiv preprint arXiv:1903.05444
SEHNKE, F., OSENDORFER, T. RÜCKSTIESS, A. GRAVES, PETERS, AND SCHMIDHUBER (2010): "Parameter-exploring policy gradients," Neural Networks, 23, 551-559.
TANG, Y., TIAN, AND D. HA (2022): "EvolAX: Hardware-Accelerated Neuroevolution." arXiv
preprint arXiv:2202.05008
TIAN, AND D. HA (2022): "Modern evolution strategies for creativity: Fitting concrete images and abstract concepts," in Artificial Intelligence in Music, Sound, Art and Design: 11th International
11

---

Conference, EvoMUSART 2022, Held as Part of EvoStar 2022, Madrid, Spain, April 20-22, 2022, Proceedings, Springer, 275-291.
TOKLU, N. E., P. LISKOWSKI, AND R. K. SRIVASTAVA (2020): "Clipup: a simple and powerful optimizer for distribution-based policy evolution," International Conference Parallel Problem Solving from Nature, Springer, 515-527.
WASKOM, M. L. (2021): "Seaborn: statistical data visualization," Journal of Open Source Software, 6. 3021.
Jounaal
XIE, X., ZHOU, H. LI, Z. LIN, AND YAN (2022): "Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models," arXiv preprint arXiv:2208.00671
YOUNG, AND TIAN (2019): "MinAtar: An Atari-Inspired Testbed for Thorough and Repro- ducible Reinforcement Learning Experiments," arXiv preprint 103.017903
12

---

Checklist
http:///w//eaoooooee/e////erreer for the benchmark for 1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section and open-sourced benchmark code.
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments (e.g. for benchmarks)...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material See |attps://github. httpc////eeeerroeeeeee/eeeeeeeee for the benchmark code and https: //ghtbb. reproduction configurations and visualizations.
URL)? [Yes]
comph//urrerbbecc////ooooooooor. analysis for
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See bettp://github. ctt.tuub.ccmmmeuurrevbbeeecc neuroevobench- anis and supplementary information.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Figure captions.
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See https://github. com/ paarroevbbench/neurreeobobo
amalysis and supplementary information.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See supplementary
information.
(b) Did you mention the license of the assets? [Yes] See supplementary information.
(c) Did you include any new assets either in the supplemental material URL? [Yes] htttp:://ithhb. http:////t/////ggtaa././/////ora code and httlh:t:///// http:///htttcccceeeeeeeeeeeeeeta reproduction configurations.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "mGEEMvSVLQ", "reviewer": "Reviewer_YBAt", "summary": "", "questions": "", "limitations": "Please see the feedback on Opportunities For Improvement. The potential negative societal impact of the work is rather minor. ", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper provide a benchmarking study of evolutionary optimization (EO) methods for large-scale machine learning applications, covering a wide range of tasks, types of EO, and budgets.\n\nThe papers provide insights into the challenges and opportunities of using EO methods.\n\nThe benchmarking methodology is rigorous and well-documented.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "GzAia6yziI", "reviewer": "Reviewer_ezSc", "summary": "", "questions": "", "limitations": "Though enough experimentations were done but main concern is lack of insight in the data and detail investigation is needed \nspecifically for the control group. We see deviant behavior in the fetch control group in figures 2 and 5 and would like further insight\nReferring fig 4 did more experiments conducted on higher noise levels?\n", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "Numerous comprehensive experiments were conducted involving 11 different problems, various optimizers, and diverse fitness and experiment types. Despite being in its early stages, this research holds great potential for benefiting the scientific community in the future.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "0LCPwTwXpt", "reviewer": "Reviewer_1pB9", "summary": "", "questions": "", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "the experiments are broad. the authors repot on an interesting topic. copmaring across many tasks is nice, as well as the sensitivity analysis.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "tUmRvBSbAp", "reviewer": "Reviewer_geZa", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "--The work describes very well the important challenge of benchmarking evolutionary optimizations with real-world data and state-of-the-art optimization methods.\n--The paper is very well written and structured. \n--The introduction of the motivation is thorough. \n--The curation of the different datasets has been thoughtful and complete.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Neuroevolution is an interesting and promising research field. Whereas one reviewer's evaluation is negative, I think that this paper is acceptable (based on the other three reviewers' positive evaluations and my own evaluation about the potential future growth of this field). It would be an interesting future research topic to provide a benchmarking framework for multi-objective neuroevolution.
Decision: Accept (Poster)