Neural Exploratory Landscape Analysis for Meta-Black-Box-Optimization
Authors: Zeyuan Ma
Jiacheng Chen
Hongshu Guo
Yue-Jiao Gong
Venue: ICLR 2025 Poster (2025)
Keywords: Landscape Analysis
Black-Box Optimization
Meta-Black-Box-Optimization
Learning to Optimize
Primary area: optimization
TL;DR: A leanable neural netwok-based landscape analysis framework tailored for Meta-Black-Box Optimization methods.

Abstract: Recent research in Meta-Black-Box-Optimization (MetaBBO) have shown that meta-trained neural networks can effectively guide the design of black-box optimizers, significantly reducing the need for expert tuning and delivering robust performance across complex problem distributions. Despite their success, a paradox remains: MetaBBO still rely on human-crafted Exploratory Landscape Analysis features to inform the meta-level agent about the low-level optimization progress. To address the gap, this paper proposes Neural Exploratory Landscape Analysis (NeurELA), a novel framework that dynamically profiles landscape features through a two-stage, attention-based neural network, executed in an entirely end-to-end fashion. NeurELA is pre-trained over a variety of MetaBBO algorithms using a multi-task neuroevolution strategy. Extensive experiments show that NeurELA achieves consistently superior performance when integrated into different and even unseen MetaBBO tasks and can be efficiently fine-tuned for further performance boost. This advancement marks a pivotal step in making MetaBBO algorithms more autonomous and broadly applicable. The source code of NeurELA can be accessed at https://anonymous.4open.science/r/Neur-ELA-303C.

Published as a conference paper at ICLR 2025
NEURAL EXPLORATORY LANDSCAPE ANALYSIS FOR META- META-BLACK, B-L---LA--OOO BOX-OPTIMIZATION
Zeyuan Ma', Jiacheng Chen', Hongshu Guo', Yue-Jiao Gong
South China University of Technology (scut. crazynicolas, jackchan.935,, guohongshu369, gongyue 1 @gmail. com
ABSTRACT
Recent research in Meta-Black-Box Optimization (MetaBBO) have shown that meta-trained neural networks can effectively guide the design of black-box op- timizers, significantly reducing the need for expert tuning and delivering robust performance across complex problem distributions. Despite their success, paradox remains: MetaBBO still rely on human-crafted Exploratory Landscape Analysis features to inform the meta-level agent about the low-level optimization progress. To address the gap, this paper proposes Neural Exploratory Landscape Analy- sis (NeurELA), a novel framework that dynamically profiles landscape features through two-stage, attention-based neural network, executed in entirely end-to- end fashion. NeurELA is pre-trained over a variety of MetaBBO algorithms using a multi-task neuroevolution strategy. Extensive experiments show that NeurELA achieves consistently superior performance when integrated into different and even unseen MetaBBO tasks and can be efficiently fine-tuned for further performance boost. This advancement marks pivotal step in making MetaBBO algorithms more autonomous and broadly applicable. The source code of NeurELA can be Neurr EELA
accessed at hhtpt:::///.. // com
INTRODUCTION
Black-Box Optimization (BBO) a discipline in mathematical optimization characterized by the absence of both the closed-form expression of the target objective function and its derivative during the optimization process. Traditional BBO methods, such as Genetic Algorithm (Holland) 1975). Evolutionary Strategy (Hansen & Ostermeier 2001a), and Bayesian Optimization (Shahriari et al. 2015). have been extensively studied in the past half century. They are applied to address a broad range of black-box scenarios, including automated machine learning (Akiba al. 2011)) bioinfor- matics (Tsaban et al.) (022). prompt tuning in large language models (Guo et al. [022bb. However, these methods often require manual tuning with deep expertise to ensure the optimal performance on unseen tasks. Recent advances in Meta-Black--oo Optimization (MetaBBO) (Xue et 2022 Lange
al. 2023 Chen et al., 2024) Guo et al., 2024a Lange et al., 2024) leverage meta-learning (Finn ct al. 2017 streamline the traditional labour-intensin tuning process across different problem distributions. This effort not only reduces the need for expert-level knowledge, but also enhances generalization towards unseen problems. As depicted on the left side of Figure 1 MetaBBO generally employs a bi-level optimization paradigm (Ma et al.) 2023). In the meta level, a control policy (often implemented as neural network) is maintained to dictate timely algorithm configuration for the low-level BBO optimizer according to the low-level optimization status. Once obtained the algorithm configuration, the low-level BBO optimizer optimizes the target BBO problem by that configuration and returns feedback to the meta-level policy the performance gain feedback. The meta- objectivee of MetaBBO is to meta-learn an optimal control policy which could achieve maximum accumulated performance gain across problem distribution.
One of the core challenges to ensure the generalization performance of MetaBBO algorithms is to design an information-rich profiling system for the low-level optimization status (highlighted in yellow box on the left side the Figure I]. Firstly, the optimization status must include the
Jiao Gong the corresponding author.

---

Published as a conference paper at ICLR 2025
identification features to differentiate various optimization problems. This design is crucial to enable the meta-level neural policy to be aware of what is the target optimization problem. Secondly, the optimization status should align with the low-level optimization dynamics, which helps to inform the meta-level neural policy about where the low-level optimization progress has reached. Existing
The Meta-T evel Optimization
RB-DAS
meta train
BLLSO
RLEPSO
Opirization
Optimizatimn
impal
Meve-
Augregated Motn-Gly
Problems
starus
Nouni Policy
sampling
Profiling
ourput
DEDDQ
LINE
Lamiscape
Dyname
Meta-
analyser
cocfiguration
GLEET
GLUI
Target
BBO metaod
Optimize
problem
The Low-Level Optimization
Original
The general workflow of MetaBBO algorithms,, which follows a bi-level optimization paradigm. The landscape analyser timely profiles the low-level optimization progress, informing the meta-level neural policy to dynamically output desired configuration for the low-level BBO method. Right: The average optimization performance (larger is better) of integrating our pre- trained landscape analyser into diverse MetaBBO tasks (red line). The blue line denotes the original performance of the MetaBBO algorithms, while the green line denotes the performance of integrating the traditional exploratory landscape analysis features into the MetaBBO algorithms.
Figure 1: Left:
MetaBBO algorithms widely adopt Exploratory Landscape Analysis (ELA) (Wright et al. 1932
Mersmann et al. 201T) to profile the low-level optimization status. While the approaches provide comprehensive understanding of problem and algorithm characteristics, they still face several limitations. 1) Many features derived from ELA are highly correlated with each other (Renau al. 2019) and exhibit strong sensitivity random sampling strategies (Renau al., 2020). This necessitates an additional step of feature selection for the given MetaBBO scenario, requiring certain level of expert knowledge. 2) The landscape features given by ELA are more like an "offline" portray of the target optimization problem. This is somewhat incompatible with the dynamic nature of MetaBBO paradigm, which requires both the "offline" problem properties and the "online" optimization progress. 3) Computing some of these features, particularly those related to convexity and local optimum properties, consume additional function evaluations (FEs). The consumption of the computational resource, initially reserved for the low-level optimization, ultimately degrades the final performance the MetaBBO algorithms.
To address the above limitations, we propose Neural Exploratory Landscape Analysis (NeurELA), novel, learnable framework designed to dynamically profile optimization status for the MetaBBO algorithms in an end-to-en manner. Our NeurELA encounters certain challenges: on the hand, we need to carefully design universal neural network-based landscape analyser that is compatible with variety of MetaBBO algorithms. On the other hand, it is crucial enable effective learning within this analyser to ensure it robustly generalizes not only to unseen optimization problems but also previously unencountered MetaBBO algorithms. We address the challenges as follows.
1) Our research begins by designing a comprehensive operating space, Ω within which our landscape analyser A functions. This space encompasses a collection of MetaBBO tasks, each designated as T : (A, D). Here, A specifies a particular MetaBBO algorithm, and D represents the associated set of optimization problems meta-train and test the algorithm. The primary objective such that it enhances the performance of various A across their respective problem sets in Ω. Our introduction of A within the comprehensive operating space marks a significant advancement in MetaBBO methodologies. Unlike traditional approaches that treat each optimization algorithm and problem in isolation, A leverages a holistic view, enhancing performance across a diverse set of tasks.
optimize
2) We instantiate the landscape analyser A through neural network model parameterized by 0. Specifically, Ag is implemented using a two-stage attention-based neural network. Our design enables flexible integration with most existing MetaBBO algorithms an effective replacement for their traditional hand-crafted landscape analysis mechanisms. Given the con-differential nature of our objective in BBO, adopt Evolution Strategy (Hansen & Ostermeier 2001b Li et al. 2018) to
2

---

Published as a conference paper at ICLR 2025
train Ag. This training is carried out within multitask neuroevolution (Miikulaannen et al. 2024 paradigm, aiming to optimize 0 to maximize the expected performance across MetaBBO tasks.
3) We propose using the trained Ag in two distinct ways. First, having been trained on diverse range of tasks, Ag is capable of being directly applied to new, unseen MctaBBO algorithms, achieving zero- shot generalization. Second, Ag also supports fine-tuning in conjunction with the meta-training of new MetaBBO algorithms, allowing it to adapt its performance for specific optimization challenges.
Through extensive experiments, we validate the effectiveness of NeurELA, as illustrated on the right side of Figure [ These results demonstrate that it is possible to pre-train a universal neural landscape analyser As over a group of MetaBBO tasks and then generalize it to unseen ones. Our work pioneers the integration of automatic feature extraction in MetaBBO, transforming these methods into fully end-to-end systems. This advancement significantly streamlines optimization processes, enhancing efficiency and adaptability across related fields, and reducing the need for manual expertise.
2 RELATED WORKS
Meta-Black-Bon -Bxx Optimization. Existing MetaBBO paradigms can be classified into following categories with respect to role they play the low-level optimization process: 1) Hyper-parameters configurator: this line of works meta-trained neural network-
policy to dynamically configure the hyper-parameters of the backend BBO optimizer. e.g., the scale factor (F) and crossover rate (Cr) in Differential Evolution (DE) algorithm (Sun et al. 2021 Tan et al.] 2022). Initial researches (e.g..
RLPSO (Wu& & Wang 2022)) employed simple MLP structures to parameterize the meta-level
policy, which was enhanced by the latest works such as GLEET (Ma et al., 2024), which facilitated Transformer (Vaswani et al. (017) architecture to boost the sequential decision making along the low-level optimization. 2) Update rule selector: works in this line flexibly select desired update rules for the low-level optimization process. Several initial works learns to select one mutation operator from pre-defined DE mutation operator pool to attain diverse optimization behaviours (Sharma ctt al. 2019; Tan & Li 202T Tan et al. 2022). Then the research scope extends to the refinement of the operators pool for optimizing more complex problems (Lian et al. (024), and dynamically selecting the entire algorithm GGuo et al. 2024a). 3) Complete optimizer generator: works in this line aims meta-learning the neural network- based policy complete optimizer. Lange al. (Lange)
ee al. 2023 2024) proposed meta-training several attention blocks as the complete updating process of CMAES Hansen al., 2003) which was followed by Song al. (Song et al. (2024). proposing meta-training Transformer based language model as an entire BBO method. Such end-to-end deep learning systems encountered interpretability issue. To this end, SYMBOL (Chen et al. 2024) proposed generating symbolic update rules for the low-level optimization process, promoting the interpretable and automatic discovery of black-box method. GLHF (Li al. (024) carefully design Transformer-basse neural network that imitates the evolutionary operators in DE algorithm. As for the designs of optimization status extraction, a great majority of existing MetaBBO algorithms borrow the idea from well-known landscape analysis researches (Wright et al., 1932 Mersmann ct al. 2011). To make those landscape features compatible with the MetaBBO paradigm, existing BB algorithms more or less facilitated feature selection and feature aggregation procedure. Take a recent work GLEET (Ma et al. 2024) as an example, it aggregates the representative ELA features (Mersmann et al. 2011) into a 9-dimensional high-level profiling features, which helps reduce the training complexity and enhance the overall optimization performance. However, we have to note that such hand-crafted feature processing requires deep expertise in optimization.
Landscape Analysis. Landscape Analysis originates from the emergence of Automated Algorithm Selection (AAS) (Rice) [976; Kerschke et al.) 2019), which selects an optimal algorithm out of a collection of algorithms for any given optimization problem. Generally, Landscape analysis is formu- lated as a function: A (X, y) + 8, that maps the sampled candidates X and their corresponding objective values y to a feature vector each dimension of which indicates some problem-specific properties of the target optimization problem. The feature vector s can be subsequently fed into machine learning algorithms (e.g., Support Vector Machines (Cortes & Vapnik, 1995) and Ran- dom Forests (Breiman 200T)) to address some specific algorithmic tasks (Kerschke & Trautmann 2019a). Over the years, wide range of Landscape Analysis feature groups have been developed, and structurally summarized in i (Wrihht et al.) 1932 Pitzer & Affenzeller) 2012) Malan & Engelbrecht
2013 Kerschke & Trautmann] 2019b), of which the mostly utilized ones are: 1) Fitness Distance
3

---

Published as a conference paper at ICLR 2025
Correlation (Jones et al. 1995) includes six features profiling the distances of the sampled candidates both in decision space and the objective space, as well as the correlation in-between the two spaces. 2) Dispersion (Lunacek & Whitley, 2006) features describe the degree of dispersity for the best quantiles of the sampled candidates. 3) Classical ELA (Mersmann et al. 2011) proposed using six groups of low-level metrics such local search, skewness of objective space and approximated curvature in (second) first-order as comprchensive abstract of basic landscape properties (c.g., global structure, multimodality, separability). Generally, classic ELA requires adequate samples and enormous computational efforts to conclude all of its fifty low-level feature values. 4) Information Content (Muñoz et al., 2014) comprises five features about the smoothness, ruggedness and neu- trality of an optimization problem. It first applies random walk sampling (e.g., Metropolis Random Walk (Hastings) 1970)) for each of the sampled candidates, then calculates its features by statistic metrics. 5) Nearest Better Clustering (Kerschke et al., 2015) is collection of features that describe the distribution of the nearest better neighbors and the nearest neighbors of the sampled candidates.
While these hand-crafted landscape analysis features provide a systematic understanding of op- timization problems, they inevitably hold high computing complexity hence consume additional computational resources. Besides, the intricate correlations in-between these feature groups (Renau eet al. 2019) and their sensitivities to the random sampling strategy (Renau et al.. 2020) necessitate the expert-level feature selection and feature aggregation procedures. To address these issues, several early-stage studies have recently proposed learning neural networks (e.g., convolutional neural net- work (Prager et al., 2021a) and point cloud transformer (Seiler et al., (202aa) as the feature mapping function. Prager et al. bb first proposed using Deep-CNN to extract landscape features from fitness map of moderate initial sample points. The neural network is trained correctly predict algorithm configuration of modular CMA-ES algorithms. However, solely feeding the fitness map might introduce information loss. To address such issue, [Seiler et al. (2022b) propose a novel point cloud transformer (PCT) based approach, which embeds sample points information and the fitness map together as the input. This work generally achieves superior landscape feature prediction accuracy. Another technical bottleneck of (Prager al.) 202 is the supervised training, which might leads the learned landscape representation overfit to a certain AAS/AAC task. To refine this issue. vaa Stein al. (2023) propose self-supervised learning through AutoEncoder, which adopted fitness map as the target to be reconstructed. More recently, [Seilrr et al. (2024) propose DeepELA to adrress both the infomation loss and the supervised trainng issues. It introduces MHA and trains it by a contrastive learning loss (InfoNCE) to ensure aligned representation across different sampling strategies. In this paper, our NeurELA presents several novel contributions compared with the above baasedg-based ELA frameworks: a) NeurELA is designed for dynamic MetaBBO tasks. enabling dynamic algorithm configuration and selection, whereas previous studies focus on static profiling of given problem. b) NeurEL.A introduces a scalable embedding strategy that removes dimensional constraints in DeepEL.A and employs a two-stage attention mechanism for extracting features across sample points and problem dimensions, making it more generic and versatile for various MetaBBO tasks. For more detailed differences, please refer to Appendix B.2.
3 NEURAL EXPLORATORY LANDSCAPE ANALYSIS
3.1 FRAMEWORK
We introduce a multi-task operating space, denoted as 22 which serves as the foundational basis for our landscape analyser, A. The space Ω is mathematically defined as:
Ω (T& (As,, Dx) k 1,2..... .
(I)
where each To represents a task configuration consisting of a MetaBBO algorithm A and a cor- responding set of optimization problems Dz. This structured collection of tasks meticulously
designed to optimize our proposed landscape analyser Ag over variety of algorithms and problems.
In accordance with the MetaBBO protocol, each algorithm As is composed of three fundamental elements: the meta-level neural policy 1lg, the low-level optimizer I, and the initial landscape feature analyser, denoted as Ag- As illustrated in Figure [ the meta- meta-training of Aş adheres to a bi-level optimization paradigm. Given problem instance f sampled from Dt. at each optimization step t, the low-level BBO method I' interacts with f to attain a raw observation {X}, vf f of the current population of candidates solutions, denoted o², where is the population size in population-
4

---

Published as a conference paper at ICLR 2025
Fast Stage: Crossiolation
Secund Stage: Cruse-duncnoo
Dynuniic Optiziization Status
Layer Normalizaioo
Feed Layar
Food. Food-Forman Forward Layer
Layer Normaliin aation
Population Information Fabedding F
Pusitional Encooling
Self-Ansation
Maih-
ReLU
Bloek
Block
Impal
Atta Block
The Arthitecture of Ts-Amm
Figure 2: Left: The architecture of the basic attention block (Attn). Right: The computation graph
of the two-stage attention mechanism (Ts-Attt).
search methods Storn & Price [997); Kennedy & Eberharl (1995); Holland (1975). Subsequently, the observation O² is further processed by the original landscape analyser, Ag. to extract optimization status features s¹ Ao (o') Based these features, the meta-level neural policy II, dictates design choice ex from the configuration space C conditioned on the current optimization status: et (s*).. Then, the low-level BBO method T loads e to optimize for next time step t + 1. The meta-performance r² is calculated by evaluating the performance gain achieved by implementing C₁. The ultimate goal of the meta-training phase is to maximize the overall meta-objective I BBmBBBB defined as:
max//meeaBee By
t= where the expectation over all problem instances from Dy and spans the total length of the low-level optimization horizon.
In our NeurELA, we parameterize a neural network-based landscape analyser Ag (where & denote the learnable parameters), which substitutes the original landscape analyser Ag, to produce the optimization status features as s = (o Our aim is to develop a universal Ag for diverse MetaBBO algorithms that perform comparably to, better than, the original hand-crafted Ao in the corresponding MetaBBO tasks. To objectively measure the performance of Ag. we define an evaluation metric (AAg/TT as the relative performance Aş against Ag in the context of MetaBBO task T (As. Dz). Concretely, we first meta- train two versions of the MetaBBO algorithms on DD:: Ag (II,, 1. Ap) and & (II,, r, Ao), and then evaluate (AA (T&.) as:
T (Ao TA)
Zp.g where Zp. p=1q=1
p.g
Hp
P P
Op
where P = Dreal is the number of test problems (the dataset D3 is divided into distinct subsets: Daiin for training and Diest for testing), Q 50 represents the number of independent runs to obtain average performance, fi. denotes the objective value achieved by testing Ai on the p-th problem instance and q-th run, and Hpp and op are the mean and standard deviation of the objective values obtained by testing A on the p-th problem over 50 independent runs. Note that we follow the Z-score normalization design in Ma et al. (2023) to evaluate the performance gain Zp.a.
The meta-objective our NeurELA framework is defined as maximizing the expected relative performance of the neural network- landscape analyser, Ag, across all MetaBBO tasks defined within the multi-task space Mathematically, this is expressed as:
1 K
maxFm 0
-Eraa [Y (Ae Tx)]
Σ
T (Ao Tx)
(4)
3.2 ARCHITECTURE OF Ap
Our architecture is designed for compatibility with various MetaBBO algorithms. We introduce a Population Information Embedding (PIE) module that preprocesses observations o' (X), .87,200 v " allowing Ag to adapt to different search ranges and objective scales. Additionally, a two-stage attention-based encoder (Ts-Attn) enhances Ag information extraction capabilities and scalability.
5

---

Published as a conference paper at ICLR 2025
Algorithm Pseudo code of training NeurELA
1: Input: The MetaBBO task space S2 Evolution Strategy ES, optimization horizon mazGen.
2: Output: Optimal neural landscape analyser Ag
3: Initialize group of {Ae, J by iSSinit():
for generation
marGin do
5: for each As, do for cach MetaBBO task Ty E f2 do Construct A.. (T,,, r. A.) and (T,,,,, Ao));
6:
7:
8:
Meta-train Ag and A on Draanni
Test meta-trained As and A/. on Dusc: end for
10:
11:
Evaluate the fitness fa of As, by FImuFLA in Eq. end for
12:
13:
Sct Ag- as the one with the highest fitness so far;
14:
Update the distributional parameters in ES by ES.update KE,, (ft)))):
15:
Sample generation of (Ae, [ by ES.sample();
end for
PIE. PIE normalizes observation of using two min-max normalization operations: first on the candidate solutions {X against the search range. and second on the objective values { using the extremum values at time step This ensures unified representation and generalization by
scaling all values to [0. 1]. For a d-dimensional optimization problem, the normalized observations of are then reorganized into per-dimensional tuples { {((X),)
i,j (=))=))))= with a shape of d x m x 2. These tuples undergo a linear transformation via a mapping matrix Wemb E R2xh, producing the population information encoding ES, which has the shape d m Here, represents the hidden dimension used in the subsequent wto-stage attention module.
Ts-Attn. The computation graph of the Ts-Attn illustrated on the right side of Figure with basic component being the attention block (Attn) of hidden dimension h. As illustrated in the left side of Figure 2 the Alln block follows the design of the original Transformer Vaswani et al. (0177) except that the layer normalization Ba et al. (2016) is used instead of batch normalization loffe & Szegedy 2015). Ts-Attn receives F. and then advances the information sharing at both cross-solution and cross- dinsesiinnnion levels. 1) Cross-solution attention: We utilize Atm block to enable same dimensions shared by different candidate solutions within the population to exchange information. 2) Cross-dimension attention: A second Attn block is implemented to further promote the sharing of information across different dimensions within each candidate. To achieve this, the output from the cross-solution phase must be transposed into the shape positional encodings to maintain the dimensional order within a candidate. After these two phases of information processing, the resulting output s' provides comprehensive optimization status features, suitable for different MetaBBO algorithms. This highly parallelizable aatentiin-bbasd architecture significantly boosts the scalability of NeurELA as the number of candidates m or dimensions increases. Additional technical details about the architecture of As can be found in Appendix A.
d h and augmented with cosine/sine
3.3 TRAINING METHOD
Neurocvolution of Ag. The meta-objective of our NeurELA, as defined in Eq. is non- differentiable, since the evaluation of the relative performance T (Ag Tx)) in Eq. (3) is not directly derived from the parameter 0 of Ag. We in turn employ an Evolution Strategy, denoted as ES, to neuroevolve Ag towards maximizing its expected relative performance for multiple MetaBBO tasks in Ω. The pseudo code the training is presented in Algorithm [ The E.S first initializes a group of N landscape analysers, then iteratively searches for 0" through marGen generations of optimiza- tion (lines to 16). Note that the meta-training of the K MetaBBO tasks, which involves training their neural-policies over the training problem set, is nested within the optimization loop (lines 5 to 12) and is extremely consuminn.,. We hence employ Ray (Moritz et al. 2018). an open-source framework for parallel processing in machine learning applications, to distribute the meta-training pipelines to multiple CPUs. This parallelization significantly enhances the overall efficiency of training NeurELA.

---

Published as a conference paper at ICLR 2025
Zero-shot Generalization. The pre-trained Ag- demonstrates robust performance boosts across K representative MetaBBO tasks, enabling seamless integration into unseen MetaBBO tasks. Specif- ically, Ag- can replace the original Ao in an unseen MetaBBO algorithm, Auneenn This enables Aunseen to concentrate on refining its meta-level neural policy A during training on the designated problem set, utilizing the capabilities of Ag- without requiring any adaptation.
Fine-tuning Adaptation. If Ag- does not perform well in zero-shot generalization, we provide a flexible alternative: fine-tuning for unseen MetaBBO tasks. In this case, Ag- is integrated into Aunseen and co-trained with its IIe during its meta-training process. This method adjusts Ag. to better align with the specific characteristics of Tunseen.
4 EXPERIMENTAL RESULTS
In this section, we discuss the following research questions: RQI: Can NeurELA generalize the pre- trained landscape analyser Ag. to unseen MetaBBO tasks? RQ2: If the zero-shot performance is not as expected, is it possible to efficiently fine-tune NeurELA for the underperforming MetaBBO task? RQ3: What does NeurELA have learned? RQ4: How efficient is the pre-trained landscape analyser in computing landscape features? RQ5: How to choose the Evolutionary Strategy for training our NeurELA? RQ6: What is the relationship between the model complexity and the generalization performance? Below, we first introduce the experimental settings and then address RQI ~ RQ6.
The operating space Ω comprises K- MetaBBO tasks. In specific, we first select three representative MetaBBO algorithms LDE Sun et al. (2021), RLEPSO Yin et al. (2021 )and RL-DAS Guo al. (2024a) from existing MetaBBO literature. Then we the associated problem set Dy for these MetaBBO algorithms as the BBOB testsuites in COCO Hansen et al. 2021). which includes a variety of optimization problems with diverse landscape properties. We have to note that the selected algorithms cover diverse MetaBBO scenarios such as auto-configuration for control parameters and auto-selection for evolutionary operators, hence ensuring the generalization of our NeurELA. We follow the original settings of these MetaBBO algorithms during their meta-training, as suugested in their papers. For the settings about the neural network, we set the hidden dimension h 16 for the neural network modules in the landscape analyser Ag. With single- head attention in Attn, Ag possess a total number of 3296 learnable parameters. We adopt Fast C.M--ES Li let al. (2018) as FS for its searching efficiency and robust optimization performance. During the training, we employ a population of N = 10 Ags within a generation and optimize the population for maaGen = 50 generations. For each generation, pipelines across 30 independent CPU cores using Ray Moritz et al. (018). For each low-level optimization process, we set the maximum function evaluations as 20000. The experiments are run on computation node of a Slurm CPU cluster, with two Intel Sapphire Rapids 6458Q CPUs and 256 GB memories. Due to the limitation of the space, we present more technical details such as the train-test split (Dimin, Deaa) , the control parameters of Fast CMAES and the use of open-sourced software Appendix A.2 A.5.
Training Setup.
parallel the N x K 30 meta-training
Baselines. We consider three baselines for comparison: 1) Original, which denotes the original
hand-crafted landscape analyser functions (Ao) in the MetaBBO algorithms. 2) ELA, which denotes the traditional ELA feature groups summarized in Section and defined in (Mersmann et al. 2011). which we implement by the open-sourced Pflacco package (Kerschke & Trautmann 2019b). 3)
DeepELA (Seiler et al. 2024). which is recent work which trains attention- based neural network as an alternative of traditional ELA features through self-supervieed contrastive learning.
4.1 ZERO-SHOT PERFORMANCE (RQ1)
Recall that a MetaBBO task composed of an MetaBBO algorithm and an associated optimization problem set (i.e., TA (A+, IDx)). We hence generalize the pre-trained landscape analyser Ag-
towards three zero-shot cases: 1) five unseen MetaBBO algorithms DEDDQN Sharma et al. 2019).
RLPSO Wu & Wang (2022), GLEET et al. (2024), MelBA (Chaybouti et al. 2022) and GLHF LLi
et al. 2024); 2) two unseen problem set BBOB- BBOS-Noisy Hansen et al. (2021) and Protein- Protein-Docking benchmark Hwang et al. 22010); 3) both the unseen MetaBBO algorithms and problem sets. Those tested Me tasks cover a wide range of meta-learned black- box optimizers. Besides, the problem
sets include challenging realistic application, e.g., Protein-Docking. We evaluate the performance of
7

---

Published as a conference paper at ICLR 2025
BBCB
MrtaBOO tadks (mannin))
Undeen MeniBli (olgorthing)
Peebentee
Unseen MedalBOO uasks (peokiem
Uaseea MetaBao todks (olgantiuss prosien sens
Perlamero
't't'
PROTEIN
Performace
apeat
RLEPSO
RL-DAS
DEDDON
MELBA
Origiaal
BLA
Deep BLA
NeuBLA
Figure 3: Zero-shot performance of NeurELA unseen MetaBBO algorithms and problem sets.
LDE BBOB
RLPSO Trotein-Dcckin
GLEET BBOB
2.0
1.5
1.5
Performance
3.0
1.0
1.0
0.5
0.5
0.0
100
40
100
Epoch
Epoch
Epoch
Original
Figure 4: Performance gain curves of fine-tuning NeurELA for specific MetaBBO task.
the ELA, DeepBLA and our NeurBLA by calculating the relative optimization performance (Eq. against the Original baseline in the given MetaBBO tasks. Since the performance of Original baseline would be within this setting, report the exponential of the calculated relative performances in Figure 3 where the bars denote the average relative performance, the error bars denote the 95% bootstrapped confidence interval, for 10 runs of meta-training. For BBOB-Noisy set, maximum function evaluations as 20000, while for Protein-Docking set, we set it to 1000. The three zero-shot cases are split by the dashed lines. The results in Figure show that: Over all of the three zero-shot cases, NeurELA generally achieves superior performance against original feature extraction designs (Original) these MetaBBO algorithms, well as the traditional landscape analysis features (ELA) and DeepELA. 2) The results of integrating ELA into the evaluated MetaBBO tasks are quite noisy, which might due the intricate correlations in-between the feature groups. Such strong correlations might challenge the information processing ability of the neural policies II in some MetaBBO algorithms. 3) We observe that our NeurELA significantly boosts the performance of Original baseline unseen MetaBBO algorithms & problem sets case, which not only validates the effectiveness of our design, but also indicates that the specific feature extraction designs (Origin) in existing MetaBBO algorithms might overfits to the problem set is meta-trained with traditional ELA, DeepELA show similar performance, which in accordance with its the training objective serve as automated alternative of traditional ELA features. 5) Our NeurELA consistently outperforms the traditional ELA features and DeepBLA on the BBOB-Noisy testsuites and Protein- Protein-Docking benchmark, which comprises a wide range of optimization problems with either different noise models and levels (c.g.. Gaussian noise and Cauchy noise Hansen et al. (2009)) or intricate realistic problem features. indicates that NeurEL shows robust generalization ability to unseen MetaBBO tasks, which involve unseen MetaBBO algorithm & realistic problems.
the
4) Compared
4.2 FINE-TUNING PERFORMANCE (RQ2)
We seleet three representative MetaBBO task from the zero-shot performance analysis (Figure to validate fine-tuning adaption efficiency of our NeurEL.. The selected MetaBBO tasks includes:
8

---

Published as a conference paper at ICLR 2025
LDE on BBOB problem set, where NeurELA surpasses the Original baseline in LDE. 2) RLPSO on Protein- Protein-Docking problem set, where NeurELA equally performss against the Original baseline in RLPSO. 3) GLEET on BBOB problem set, where NeurELA does not achieves the expected zero- shot performance. Concretely, we integrate the pre-trained Ag- into the corresponding Me algorithm, then fine-tune it in conjunction with the meta-training over the given problem set. We depict the performance gain curves during the meta-training in Figure 4 We additionally plot the performance gain curves of the meta-training during the zero-shot generalization for comparison. The results showcase the flexible adaption potential of our NeurELA: 1) fine-tuning NeurELA on specific MetaBBO tasks where our NeurELA outperforms, equally performs or underperforms Original baseline would consistently introduce significant performance boosts. 2) the fine-tuning strategy efficient in adapting NeurEL. for new MetaBBO dynamies and problem structures, which only requires less than 20% meta-training epochs to achieve the similar zero-shot performance.
4.3 WHAT HAS NEURELA LEARNED? (RQ3)
In this section, we presents the advantage of the landscape fea- tures learned by NeurELA against traditional ELA features under the MetaBBO settings. Concretely, we take LDE (Sun et al., 202T) as a showcase. In LDE, the meta-level policy (a LSTM-based RL agent) outputs the mutation strength factor Fd E 1) for the low-level DE per optimization step t. F* indicates the exploration-exploittiii tradeoff during the optimization (larger value indicates exploration, vice versa). We first record the per step populations during the opti- mization process of LDE (10 runs on 10-D Rastrigin in CoCo-BBOB testsuites). We then simply classify the recorded populations as explo- ration population (F > 0.5) and exploitation population (F* < 0.5). We now illustrate the low-rank visualization of the ELA feature and our NeurELA feature for the recorded populations in Figure 5] The data points denoted the 2-dimensional features obtained by first feeding the populations into ELA/NeurELA and then using PCA (Shlens, (0014) transformation on the output landscape features. Red points and blue points represent the features for the exploration and exploitation popu- lation respectively. The results validate that NcurELA is more sensitive than traditional ELA features on low-level optimization dynamics. Tra- ditional ELA is mainly used for describing a problem, it might falls short in providing informative low-level optimization status for MetaBBO methods. This explains the consistent superior perfor- mance of our NeurELA (see Figure Notably, our NeurELA can be meta-trained on Me tasks to provide sensitive and accurate low-level optimization status for the meta-level policy. We also provide further interpretation analysis which reveals the correlation of features learned by NeurELA and traditional ELA features, refer Appendix B.3 for detailed discussion.
-10
405
0.00
6.00
6.5
Figure 5: Low-level opti- mization features obtained by traditional ELA and our NeurELA, through PCA di- mension reduction.
4.4 IN-DEPTH STUDY (RQ4 ~ RQ6)
Computational Efficiency Comparison (RQ4). We compare the computational efficiency of our NeurELA with Original and ELA as either the amount of candidate samples m or the dimension of the optimization problem di scales. We report the average wall-time required by each baseline and our NeurELA to produce the landscape features (for 1000 runs). The results show that: 1) Our NeurELA achieves comparative computational efficiency with the specific hand-crafted designs in existing MetaBBO algorithms, while introducing significant performance boosts; 2) When the sample size m increases, our NeurEL. becomes more time-efficient than the original feature extraction designs in existing MetaBBO algorithms. This is primarily owning to the two-stage attention architecture in our NeurELA, which can be efficiently computed on CPUs in parallel. 3) The traditional ELA features encounters the curse of dimensionality, which makes itself impractical for MetaBBO algorithms to solve high-dimensional optimization problems.
Evolution Strategy Selection (RQ5). Within this paper, we employ a neuroevolution process to optimize the parameters of the landscape analyzer Ag. which presents a large-scale optimization challenge as A, comprises 3296 learnable parameters. To identify an effective ES, we implement several candidate evolution strategy variants: Fast CMAES Li et al. (2018), SEP-CMAES Ros &

---

Published as a conference paper at ICLR 2025
Table 1:
The average wall time (in
Zene Ran--hoo Parfommace
(16,1.3g
(15.3
seconds) for computing features.
(16, Tike
d 10
d= 100
m
d 1000
Zeme-bon
NeafILA Lh (hmmm
Reul
Original
4.99F-03
5.61E403
6,38E003
ELA
1.49E-01
1,885+01
3000--02
64, 1870
1120
A
3.426403
6,245-03
7,118003
FOMARS
RAMSS
CMASS
d 10
Figure 6: Zero-shot perfor-
Figure 7: The influence of model complexity on the zero- shot performance.
Origiall
4,399--03
2.336/02
3.16E-02
mance when using different evolution strategies.
14
1.695.01
2.026.01
A
5.42E-03
7,97E-03
9,38E-03
Hansen (2008), RIES, Li & Zhang 2017), and the original CMAES Hansen & Ostermeier (2001b), and report the overall zero-shot performance (unseen algorithms & unseen problem sets) achieved by training our NeurELA with these candidate strategies for 10 runs in Figure The results reveal that Fast CMAES dominates the other candidates, and is hence selected as the neural network optimizer ES for NeurELA. The convergence of these ES baselines are shown in Appendix B.
Model Complexity (RQ6). We discuss the relationship between the model complexity and the zero-shot performance (unseen MetaBBO algorithm & problem sets) of our NeurELA. Concretely, We pre-train NeurELA under 6 different model complexities, with various hidden dimensions, i.e., (16, 64), and the number of the Ts-Attn module, i.e., / (1, 5). We additionally pre-train three MLP baselines, which substitute the Ts-Attn module in NeurELA with a linear feed-forward layer, which holds a shape of h x h, h (16, 64, 128). We report both the zero-shot performance (y- axis) and the computational efficiency (r-axis, presented as the consumed wall time for computing the landscape features) in Figure 7 where the dashed lines denotes the performance and wall-time of the Original baseline. #apara denotes the number of the learnable parameters. The results show that: 1) significant performance gap observed between the MLP baselines and our Ts-Attn module (h 16, / 1). It validates the effectiveness of our Ts-Attn design, which enhance the feature extraction of our NeurELA by encouraging the information sharing both the cross-solution and cross- dooss-dimnnioo levels; 2) As the model complexity increases, the performance of the Ts-Attn module drops rapidly. It reveals that the increased number of learnable parameters challenges the optimization ability of the backend FSS Given the limited computational resources, it is difficult to identify the optimal parameters l*
Limitations. As pioneer work using deep neural network for landscape analysis in MetaBBO, our NeurELA show possibility of learning an universal landscape analyser with minimal expertise requirement and appealing optimization performance. However, major limitation that the training efficiency of NeurELA is not very satisfactory. As discussed in the Model Complexity part, the zero-shot performance drops rapidly when the model complexity increases. This due to the relatively limited large-scale optimization ability of existing evolution strategies. Unfortunately, the meta-objective of NeurEL. (Eq. 4 is not differentiable hence cannot benefit from back propagation methods. We denote this limitation as an important future work to address.
CONCLUSION
In this paper, introduce NeurELA to complement the last mile in making the MetaBBO paradigms entirely end-to-end: the automatic extraction of optimization status features. To this end, we propose landscape analyser parameterized by two-stage attention-bassd neural network. The analyser enables flexible replacement for the original hand-crafted landscape analysis mechanisms in existing MetaBBO algorithms. We employ a neuroevolution paradigm optimize by maximizing performance over a multi-task MetaBBO operating space. The experimental results showcase that our NeurBLA serves two key functions: 1) can seamlessly integrated into unseen MetaBBO algorithms for impressive zero-shot performance, and 2) it can also be fine-tuned together with new MetaBBO algorithms to adapt its performance for specific optimization challenges. While possesses certain limitation (c.g., low training efficiency), work represents significant step toward realizing a fully automatic MetaBBO algorithm- paradigm that learns to optimize without the need for manual expertise.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China No. 62276100, in part by the Guangdong Provincial Natural Science Foundation for Outstanding Youth Team Project No. 2024B151 in part by the Guangdong Natural Science Funds for Distinguished Young Scholars No. 2022 202281515020049, and in part by the TCL Young Scholars Program.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In International Conference on Knowledge Discovery & Data mining, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rXiv:6607006455 2016.
Lco Breiman. Random forests. Machine Learning, 2001.
Sofian Chaybouti, Ludovic Dos Santos, Cedric Malherbe, and Aladin Virmaux. Meta-learning of black-box solvers using deep reinforcement learning. In NeurIPS 2022, MetaLearn Workshop, 2022.
Jiacheng Chen, Zeyuan Ma, Hongshu Guo, Yining Ma, Jie Zhang, and Yue-Jiao Gong. SYMBOL: Generating flexible black-box optimizers through symbolic equation learning. In International Conference on Learning Representations 2024.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 1995.
Qiqi Duan, Guochen Zhou, Chang Shao, Zhuowei Wang, Mingyang Feng. Yijun Yang, Qi Zhao, and Yuhui Shi. Pypop7: A pure-python library for population-basee black-box optimization. arXiv 2022.
preprint arXiv:22
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning. 2017.
Hongshu Guo, Yining Ma, Zeyuan Ma, Jiacheng Chen, Xinglin Zhang, Zhiguang Cao, Jun Zhang, and Yue-Jiao Gong. Deep reinforcement learning for dynamic algorithm selection: A proof-of-principe study on differential evolution. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2024a.
Qingyan Guo, Rui Wang. Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In International Conference on Learning Representations, 2024b.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Compulation, a.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 2001
Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary Computation, 2003.
Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. Real-paraneter black-box opti- mization benchmarking 2009: Noiseless functions definitions. PhD thesis, INRIA, 2009.
Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. Coco: A platform for comparing continuous optimizers a black-box box setting. Optimization Methods and Software, 2021.
W Keith Hastings. Monte carlo sampling methods using markov chains and their applications. 1970.
John H Holland. Adaptation in Natural and Artificial Systems. 1975.
11

---

Published as a conference paper at ICLR 2025
Howook Hwang, Thom Vreven, Joël Janin, and Zhiping Weng. Protein- version 4.0. Proteins: Structure, Function, and Bioinformatics, 2010.
Protein-protedd proteinrottneee docking benchmark
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015,
Terry Jones, Stephanie Forrest, et al. Fitness distance correlation as a measure of problem difficulty for genetic algorithms. In ICGA, 1995.
James Kennedy and Russell Eberhart. Particle swarm optimization. In Proceedings of the Interna- tional Conference on Neural Networks, 1995.
Pascal Kerschke and Heike Trautmann. Automated algorithm selection on continuous black-box problems by combining exploratory landscape analysis and machine learning. Evolutionary Computation, 2019a.
Pascal Kerschke and Heike Trautmann. Comprehensive feature-based landscape analysis of continu- ous and constrained optimization problems using the r-package flacco. Applications Statistical Computing, 2019b.
Pascal Kerschke, Mike Preuss, Simon Wessing, and Heike Trautmann. Detecting funnel structures by means of exploratory landscape analysis. In Annual Conference on Genetic and Evolutionary Computation, 2015.
Pascal Kerschke, Holger H Hoos, Frank Neumann, and Heike Trautmann. Automated algorithm selection: Survey and perspectives. Evolutionary Computation, 2019.
Jakub Kudela. A critical problem in benchmarking and analysis of evolutionary computation methods. Nature Machine Intelligence, 2022.
Robert Lange. Tom Schaul, Yutian Chen, Tom Zahavy, Valentin Dalibard, Chris Lu, Satinder Singh, and Sebastian Flennerhag. Discovering evolution strategies via sataa-llak-booo optimization. In Companion Conference on Genetic and Evolutionary Computation, 2023.
Robert Tjarko Lange. Yingtao Tian, and Yujin Tang. Evolution transformer: In-context evolutionary optimization. arXiv preprint arXiv:2403.02985 2024.
Xiaobin Li, Kai Wu, Yujian Betterest Li, Xiaoyu Zhang, Handing Wang, and Jing Liu, Glhf: General learned evolutionary algorithm via hyper functions. arXiv preprint arXiv:2405.03778, 2024.
Zhenhua Li and Qingfu Zhang. A simple yet efficient evolution strategy for large-scale black-box optimization. IEEE Transactions on Evolutionary Computation, 2017.
Zhenhua Li, Qingfu Zhang, Xi Lin, and Hui-Ling Zhen. Fast covariance matrix adaptation for large-scale black-box optimization. IEEE Transactions on Cybernetics, 2018.
Hongqiao Lian, Zeyuan Ma, Hongshu Guo, Ting Huang, and Yue-Jiao Gong. Rlemmo: Evolutionary
multimodal optimization assisted by deep reinforcement learning. In Genetic and Evolutionary Computation Conference Companion, 2024.
Monte Lunacek and Darrell Whitley. The dispersion metric and the ema evolution strategy. In Annual Conference on Genetic and Evolutionary Computation, 2006.
Zeyuan Ma, Hongshu Guo, Jiacheng Chen, Zhenrui Li, Guojun Peng, Yue-Jiao Gong, Yining Ma, and Zhiguang Cao. Metabox: A benchmark platform for meta-black-box optimization with reinforcement learning. In Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Zeyuan Ma, Jiacheng Chen, Hongshu Guo, Yining Ma, and Yue-Jiao Gong. Auto-configuring exploration-exploittiion tradeoff in evolutionary computation via deep reinforcement learning. In Genetic and Evolutionary Computation Conference Companion, 2024.
Katherine M Malan and Andries P Engelbrecht, A survey of techniques for characterising fitness landscapes and some possible ways forward. Information Sciences, 2013.
12

---

Published as a conference paper at ICLR 2025
Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. Exploratory landscape analysis. In Annual Conference on Genetic and Evolutionary Computation, 2011.
Risto Miikkulaneen, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing. 2024.
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael Jordan, et al. Ray: A distributed framework for emerging ai applications. In Symposium Operating Systems Design and Implementation. 2018.
Mario A Muñoz, Michael Kirley, and Saman K Halgamuge. Exploratory landscape analysis of contin- uous space optimization problems using information content. IEEE transactions on evolutionary computation, 2014.
Erik Pitzer and Michael Affenzeller. A comprehensive survey on fitness landscape analysis. Recent Advances in Intelligent Engineering Systems, 2012.
Raphael Patrick Prager, Moritz Vinzent Seiler, Heike Trautmann, and Pascal Kerschke. Towards feature-free automated algorithm selection for single-objective continuous black-box optimization. In IEEE Symposium Series on Computational Intelligence, 2021a.
Raphael Patrick Prager, Moritz Vinzent Seiler, Heike Trautmann, and Pascal Kerschke. Towards feature- free automated algorithm selection for single-objective continuous black-box optimization. In 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1-8, 2021b. doi: 10.1 /00////SC150455..221.06666
Quentin Renau, Johann Dreo, Carola Doerr, and Benjamin Doerr. Expressiveness and robustness of landscape features. In Genetic and Evolutionary Computation Conference Companion, 2019.
Quentin Renau, Carola Doerr, Johann Dreo, and Benjamin Doerr. Exploratory landscape analysis is strongly sensitive to the sampling strategy. In International Conference Parallel Problem Solving from Nature, 2020.
John R Rice. The algorithm selection problem. In Advances in Computers. 1976.
Raymond Ros and Nikolaus Hansen. A simple modification ema-es achieving linear time and space complexity. In International Conference on Parallel Problem Solving from from Nature, 2008.
Moritz Vinzent Seiler, Raphael Patrick Prager, Pascal Kerschke, and Heike Trautmann. A collection of deep learning-based feature-free approaches for characterizing single-objective continuous fitness landscapes. In Genetic and Evolutionary Computation Conference Companion, 2022a.
Moritz Vinzent Seiler, Raphael Patrick Prager, Pascal Kerschke, and Heike Trautmann. A collection of deep learning-basee feature-free approaches for characterizing single-objective continuous fitness landscapes. In Proceedings of the Companion Conference Genetic and Evolutionary Computation, 2022b.
Moritz Vinzent Seiler, Pascal Kerschke, and Heike Trautmann. Deep-ela: Deep exploratory landscape analysis with self-supervised pretrained transformers for single-and multi-objective continuous optimization problems. arXiv preprint arXiv:2401.09) 192, 2024.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 2015.
Mudita Sharma, Alexandros Komninos, Manuel López-Ibáñed and Dimitar Kazakov. Deep rein- forcement learning based parameter control in differential evolution. In Proceedings of the Genetic and Evolutionary Computation Conference, 2019.
Jonathon Shlens. A tutorial on principal component analysis. arXiv preprint arXiv: 1100, 2014.
13

---

Published as a conference paper at ICLR 2025
Lei Song, Chenxiao Gao, Ke Xue, Chenyang Wu, Dong Li, Jianye Hao, Zongzhang Zhang, and Chao Qian. Reinforced in-context black-box optimization. arXiv preprint arXiv:2402.17423, 2024.
Rainer Storn and Kenneth Price. Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization, 1997.
Jianyong Sun, Xin Liu, Thomas Bäck, and Zongben Xu. Learning adaptive differential evolution algorithm from optimization experiences by policy gradient. IEEE Transactions on Evolutionary Computaaioo, 2021.
Zhiping Tan and Kangshun Li. Differential evolution with mixed mutation strategy based on deep reinforcement learning- Applied Soft Computing, 2021.
Zhiping Tan, Yu Tang, Kangshun Li, Huasheng Huang, and Shaoming Luo. Differential evolution with hybrid parameters and mutation strategies based on reinforcement learning. Swarm and Evolutionary Computation, 2022.
Tomer Tsaban, Julia K Varga, Orly Avraham, Ziv Ben-Aharon, Alisa Khramushin, and Ora Schueler- Furman. Harnessing protein folding neural networks for peptide-protein docking. Nature Commu- nications, 2022.
Bas van Stein, Fu Xing Long, Moritz Frenzel, Peter Krause, Markus Gitterle, and Thomas Bäck. Doe2vec: Deep-leanning based features for exploratory landscape analysis. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation, GECCO '23 Companion, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Sewall Wright et al. The roles of mutation, inbreeding, crossbreeding, and selection evolution. 1932.
Di Wu and G Gary Wang. Employing reinforcement learning to enhance partiele swarm optimization methods. Engineering Optimization, 2022.
Ke Xue. Jiacheng Xu, Lei Yuan, Miqing Li, Chao Qian, Zongzhang Zhang, and Yang Yu. Multi-agent dynamic algorithm configuration. Advances Neural Information Processing Systems, 2022.
Shiyuan Yin, Yi Liu, Guoliang Gong, Huaxiang Lu, and Wenchang Li. Rlepso: Reinforcement learning based ensemble particle swarm optimizer. In International Conference on Algorithms, Computing and Artificial Intelligence, 2021.
14

---

Published as a conference paper at ICLR 2025
A TECHNICAL DETAILS
A.1 TECHNICAL DETAILS OF Aa
Within this paper, the aim of the landscape analysis is to profile the dynamic optimization status of the current optimization process. That is, given d-dimensional target optimization problem f. at any time step I, the optimization process maintains a population of m candidate solutions {X} E [Rd) and their corresponding objective values {uf = [X)))= We consider end-to- end neural network structure that receives the candidates population and their corresponding objective values as input, and then outputs h-dimensional dynamic optimization status se for each X This optimization status feature aggregates the information of the optimization problem and the current candidate population, hence can be used for dynamic landscape analysis MetaBBO algorithms. We have to note that the core challenges in designing such neural landscape analyser locate at: 1) generalizability: should be able to handle optimization problems with different searching ranges and objective value scales; 2) scalability: it should be capable of computing the dynamic optimization status efficiently as the amount of the sampled candidates or the dimensions of the problem scales. We address the above two challenges by designing two-stage attention based neural network structure as the landscape analyser (Ao) in NeurELA. We now introduce the architecture of the Ag and establish its overall computation graph step by step. For the convenience of writing, we omit superfix for time step é.
Pre-processing Module. To make NeurELA generalizable across different problems with various searching ranges and objective value scales, we apply min-max normalization over the searching space and the objective value space. Concretely, for a specific d-dimensional optimization problem ff (suppose a minimization problem), we acquire its searching range {(bb) ub"] 1bb])}==> where lb and ubd represent the lower bound and the upper bound at j-th dimension. Then we normalize each X; in
the candidate population by X
X:- a63-157 where X denotes the j-th dimension of X; After the min-max normalization over the searching space, we min-max normalize the objective values within -erin_. where ymmn and ymar denotes the lowest and highest achieved this time step, Y/ Smay -mmin
objective values in this time step. We have to note that by normalizing the X, and yi within the range of [0, 1]. we attain universal representation for different optimization problems, ensuring the generalizability of the subsequent neural network modules. The normalized { and {3s im are then re- organized as a collection of meta data {{(}), 3.) 3 with the shape of d X m x 2. We then embed the meta data with linear mapping Wemb E 2 as the final input encoding s, of which the shape d x m x h. h denotes the hidden dimension of the subsequent two-stage attention module.
Two-stage Attention Block, optimization status information across candidate solutions and across each dimension of the decision variables. The overall computation graph of the Ts-Attn is illustrated in the right of Figure 2 in the main body, of which a basic component the attention block (Attn). As illustrated in the left of Figure 2 in the main body, the Attn block mainly follows designs of the original Transformer Vaswani et al. (2017). except that the layer normalization| Ba et al. (2016) is used instead of batch normalization loffe & SSzeeedy (2015). Given group of L ippt encoding vectors Xin E R lxxh Eq. (5) details the computation of the Atnn block.
We construct a two-stage attention block (Ts-Attn) to aggregate
9 LN(X... MHSS(X...)
U = FF (ReLU(FF())))) (g))
LN(g + +
where MHSA, LN and FF denote the multi-head self-attention Vaswani et al. (2017) (with the hidden dimension of f)), layer normalization Ba et al. (0166) and linear feed forward layer respectively. The output holds identical shape with the input X In our Ts-Attn block, we employ Attn block Attrinnter for the first cross-solution information sharing stage, and the other Attn block Attraattre for the second cross-dimension information sharing stage (illustrated in the right of Figure in the main body. The Ts-Attn receives the input encoding s of the current candidate population, and then advances the information sharing in both cross-solution and cross-dimension level. The computation
15

---

Published as a conference paper at ICLR 2025
is detailed in Eq. O
H Att (S)
Transposel H d x m x h + + m m x h) + PE
H out Attmantra (H)
(6)
F Findie MeanPooling Hou:; mn x d x h + m mx h
F Ppop MeanPooling( indiv), h h)
At the first stage, we let the input encoding (attained from the pre-processing module) pass through Attm Since we group the encodings of the same dimension of all candidates in 8, the Atter promotes the optimization information sharing across candidates in current population. From the first stage, we obtain group of hidden features И with the shape of d we first transpose into the shape of m x x h to regroup all dimensions of a candidate together. We then add coslsin positional encoding (PE) over the transposed H inform the order of different dimensions in candidate. We then let H pass through Alimamtro to advance the information sharing among the different dimensions within the same candidate. The output of Allnintra holds the shape of m x d x h. At last, we apply McanPooling on Hout to get the landscape feature for each candidate Findie in the population, and apply a second MeanPooling on Findie to get the landscape feature for the whole candidate population Fpop- We have to note that we calculate both Findio and Foo to make our NeurEL.A compatible with diverse MetaBBO algorithms, which either require the landscape feature of the whole population (e.g.. Wu & Wang (2022)) or require separate landscape feature for each candidate (e.g., Sun et al. (2021)). The highly parallelizable attention- basod neural-network architecture ensure the scalability of our method as the amount of the sampled candidates or the
X h. At the second stage,
dimensions of the problem increases.
Now summarize the end-to-end workflow of the neural landscape analyser (As) in our NeurELA. At any time-step t within the optimization process, the pre- pre-processing module transforms the infor- mation of the candidate population (i.e., {X]] and {u() )==)) into the input encoding 8. Then the Ts-Attn module transforms s into the dynamic landscape features F!. inidiv and F pop
A.2 TRAIN-TEST SPLIT OF BBOB TESTSUITES
BBOB contains 24 synthetic problems owning various landscape properties Mersmann et al. (001 (e.g. multi-modality, global structure, separability and etc.). Table 2 list all of the 24 problems according to their category. Due the diversity of properties, how to split these problems into train-test set becomes key issue to ensure the training performance and its generalization ability. Our fundamental principle to split is maximize the inclusion of representative landscape properties as possible. Specifically we visualize these 24 problems under 21) setting, and then select 12 representative problems into train set. We also provide contour map of problems in train set in Figure & and test set in Figure Moreover, to avoid possible issue Kudela (2022) coming from fixed optima which often located in (0.....)) 0] in current benchmark problems (this might facilitate model to overfit to this fixed point), we thus add random offset O into each problems, that is to convert y f(x) into y = f(r O). This operation is inserted into both train set Dimin and test set Destt-
A.3 TRAIN-TEST SPLIT OF BBOB-NOIST AND
TESTSUITES
We summarize some key characteristic of this two testsuits as follows.
this testsuits contains 30 noisy problems from COCO Hansen et al. (021). They are obtained by further inserting noise with different models and levels into problems in BBOB testsuits. BBOB-Noisy is characterized by its noisy nature and often used to examine robustness of certain optimizers.
BBOB-Noisy:
Protein-Docking: this testsuits contains 280 instances of different protein-protein com- plexes/Hwang. al. (2010). These problems characterized by rugged objective landscapes and are computationally expensive to evaluate.
We follow train-test split for these two testsuites defined in MetaBox Ma et al. (203). Under easy mode in MetaBox, 75% of instances are allocated into training and the remaining 25% are used in
16

---

Published as a conference paper at ICLR 2025
Table Overview of the BBOB testsuits.
No.
Functions
Sphere l'unction
Ellipsoidal Function
Sepurable functions
3
Kastrigin Function
Buche-Rasttije Rastrig Function
5
Linear Slope
Altraclive Secior Fuunciion
Step Ellipsoidal Function
with low or moderate condirioning
Rosenbeock Function original
Function, rolated
10
Ellipsidal l'unction
Functions with high conditioning and unimodal
Discus Function
12
Bent Cigar Function
13
Sharp Ridge Function
Different Powers
Multi Multi-modal modal functions
15
Rastrigin Function (non-s:parable counterpant of F3)
Funciion
global structure
with adequate
17
18
Schallers F7 Function, moderately ill condiiieeeee
Schafters F7 Function
19
Composiic Orjowank-Kesenbros Function F8F2
Multi-modal functions
20
Gallagher's Gaussian 10]
Schweeel Function
Function
with weak
22
Gallagher' Gaussian 21 Peaks Function
global structure
23
Katsunra Function
24
Lunacek bi Rustrigin Function
Detault range: [-5, 51
testing. Training or further fine-tume fine-tuning on these two testsuites in our experiments are executed in the train set, and then validate performance corresponding MetaBBO algorithms the test set Drest
(a) Function
(b) Function 2
(c) Function
(d) Function
(e) Function 13
(I) Function 16
Function 17
(h) Function 18
(i) Function 21
(j) Function 22
(k) Function 23
(l) Function 24
Figure 8: Fitness landscapes functions BBOB train set when dimension is set to 2.
(a) Function 3
(b) Function 4
(c) Function 6
(d) Function 8
(c) Function 9
(f) Function 10
(g) Function
(h) Function 12
Function
(j) Function 15
(k) Function (0) Function 20
Figure 9: Fitness landscapes of functions in BBOB test set when dimension is set to 2.
17

---

Published as a conference paper at ICLR 2025
A.4
LICENSE OF USED OPEN-SOURCED ASSETS
Our codebase
can
be
accessed at https://anonyymous 4open. wloen.sccience.. science/r/ listed several open-sourced assets used in our work and
Neur-ELA 303C In Table
their corresponding licenses.
Table 3: Used open-sourced tools and their licenses.
Used scenario
Asset
License GPI.-300 license
Top-level optimizer
PyPop7 Duan al.
(0022)
MetaBBO algorithms implementation
MetaBox Ma et al.
(0023)
BSD-3-Clause license
Low-level train-test workflow
Parallel processing
Ray Moritz et al.
2018
Apache-2.0 license
ELA feature calculation
pflacco Kerschke & Trautmann
(0019b)
MIT license
A.5 CONTROL-PARAMETER OF ES
Fast CMAES We grid-search three key hyper-parannn parameters in Fast CMAES, including the mean value j and sigma value of the initial Gaussian distribution used for sampling, learning rate of evolution path update c. We list the grid search options in Table and choose the best setting according to training performance on BBOB, Besides, for other control-parann control-parameters of the Fast CMAES, we follow the default settings listed in its original paper Li et al. (2018).
Table 4: Grid-scarch of paraoe--arrmeterss of Fast CMAES.
Control-parameters
Grid options
Selected setting
Initial mean value p
[0 R"T
RD
Initial sigma value
[0.1, 0.3]
Learning rate of cvolution path update
[2.0/(D 5.0). (D00(D+
2.0/(D 5.0)
D represents the searching dimension of Fast CMAES. More specifically, as the top-level uptiminer to neural-evolve our neural landscape analyser As. D specifies the dimension of As
which is 3296 under default settings in our main experiment.
Other candidate evolution strategy variants We follows the default settings as implementations in PyPop7 Duan al. (2022) for other candidate top-level optimizers. We made comparision study among SEP-CMAES Ros & Hansen (2008), RIES, RMES Li & Zhang (217)), original CMAES Hansen & Ostermeier (2001b) and Fast CMAES under their default settings and finally select Fast CMAES as the default top-level optimizer of this work.
B ADDITIONAL DISCUSSION
B.1 TRAINING CONVERGENCE
In NeurELA, the meta-objective as defined in Eq. (4), is cnndiifeerentibbe. Hence, we train the neural network in NeurELA through neuroevolution. Such paradigm requires effective evolutionary optimizers which maintain a population of neural networks and reproduce elite offsprings iteratively according to the training objective of the neural networks. In NeurELA, we adopt Evolution Strategy (ES) since it is claimed to be more effective then other optimizers. There are many modern variants of ES method, of which we select five: Fast CMAES (Li et al. 2018), Sep-CMAES (Ros & Hansen] 2008), RIES (Li & Zhang 2017). RMES (Li & Zhang (017) and CMAES (Hansen & Ostermeier 2001b) as
HOMARS
CEAES
KIFS
-2--
-4-
46
5
10
30
15
58
Trchniig Step
Figure 10: Training curves of dif- ferent ES baselines when training NeurELA
candidates. We present the training curves of all five optional ES baselines under our training settings in Figure [0] The
results demonstrate that the Fast CMAES adopted for
18

---

Published as a conference paper at ICLR 2025
training NeurELA converges and achieves superior training effectiveness to other ES baselines.
B.2 DIFFERENCE BETWEEN NEURELA AND DEFP-ELA
Although previous work Deep-ELA (Seiler al. 2024) also proposed using attention-based architecture for landscape analysis, there are significant differences between our NeurELA and Deep-ELA, which we listed below:
1. Target scenario. NeurELA is explicitly designed for MetaBBO tasks, where dynamic optimization status is critical for providing timely and accurate decision-making at the meta level. In contrast, Beep-ELA serves as static profiling tool for global optimization problem properties and is not tailored for dynamic scenarios. NeurELA supports dynamic algorithm configuration, algorithm selection, and operator selection. In contrast, Deep-ELA's features are restricted to static algorithm selection and configuration, limiting adaptability in dynamic MetaBBO workflows.
2. Feature extraction workflow. First, NeurELA addresses the limited scalability of Deep-ELA -LA for high dimensional problem. Concretely, the embedding in Deep-ELA is dependent on the problem dimension and hence the authors of Deep-ELA pre-defined maximum dimension (50 in the original paper). To address this, NeurELA proposes a novel embedding strategy which re-organizes the sample points and their objective values to make the last dimension of the input tensor is 2 (Section 3.2). This embedding format has a sigmificant advantage: the neural network of NeurELA is hence capable of processing any dimensional problem and any number of sample points. NeurELA enhances the information extraction through its two-stage attention-based neural network. Specifically, when processing the embedded data, Deep-ELA leverages its self-attention layers for information sharing across sample points only. In contrast, NeurELA incorporates a two-stage attention mechanism, enabling the neural network first extract comprehensive and useful features across sample points ccross-solution attention) and then across problem dimensions pross-dimension attention). This design helps mitigate computational bias and improve feature representation.
3. Training method. The training objective and training methodology in NeurELA and Deep- ELA are fundamentally different. Deep-ELAA aims to learn neural network that could serve as an alternative of traditional ELA. Its training objective is to minimize the contrastive loss (InfoNCE) between the outputs of its two prediction heads (termed as student head and teacher head) by gradient descent, in order to achieve invariance across different landscape augmentation instance. In contrast, the training objective of NeurELA is to learn a neural network that could provide dynamic landscape features for various MetaBBO tasks. Specifically, its objective is to maximize the expected relative performance improvement when integrated into different MetaBBO methods. Since such relative performance improvement is not differentiable, NeurEL LA employs neuroevolution its training methodology. Neuroevolution is recognized as an effective alternative to gradient descent, offering robust global optimization capabilities.
the same problem
In summary, NeurELA and Deep-ELA are two totally different works with different target operating scenarios, algorithm design tasks, neural network designs and workflows, and training methodologies.
B.3 FURTHER INTERPRETATION ANALYSIS
To further interpret what features have been learned by our NeurELA, we have conducted following experimental analysis to further interpret the relationship between NeurELA features and traditional ELAA features, where we uses Pearson Correlation analysis to quantify the correlation between each Ae feature and each traditional ELLA Teature.Belone we explain our experimental methodology step by step:
We select three MetaBBO methods (LDE, RLEPSO and RL-DAS) from our training task set and employ their pre-trained models to optimize the 24 problem instances in CoCo BBOB- suite. Each MetaBBO method performed 10 independent runs per problem instance, with each run consisting of 500 optimization steps. Now obtain **24* 10 720 optimization trajectories, each with length 500, and the data at each step of a trajectory is the population and the corresponding objective values (Xs, Ys).
19

---

Published as a conference paper at ICLR 2025
ela meta.lin, simple.adi_r
ela_med metameta.lin www.mmaliinimippeiieee....o iitpeiiiei//eeeeeep
0.8
ela meta.lin, lin, simple.coef.min
ela meta. senttaiiinssmple...eee simple.coef.mar
ela meta simple. coef.max by
0.6
ela meta Jii___n _______tee w__nteract.ad___
wwtmmmta.qqu setiiuuudu_aaaa catiaauuudslll.eaaeo simple.ad__r
ela [ta_mmtta.qu sead__iplleecod
meta.quad _nteract..dj_rr
ela pla_conv.conv_pe _robb
ela www.coooovinncccc __rob_rbb..
ela convolonndeee _e__dde__rrgg
ela ela_coo www..linndeeeeeeeeeeeee eee__de____eee _bbs
ela je_eeeel. mmcce..mcc_.lleleeeeeee _________________
ela level.mmce qda
ELA
ela level.lda
ela level. Ida 50
Jevel.mmce _da__00
ela level.lda _da_50
mecell.mcceeeeeeemmmmmmooo [da_75
ela la_level.mcee Baciee__d_______ _5
ela level.lda qda ela local.n loc opt.abs
0..4
ela local.n, loc opt.
ela al_local. ocal.bbstt2mea _oottoooannooooorggggog
ela local local www.bbeteeeennomeeeee contr.ratio
basisassieeee..v ela Jocal. sizes.avg, best
-0.6
ela local basin, wizes.avg_nn.. aegeeaaggooonno joi_oon_bett best
local. basin sizes.avg_ve worst
ela distreskewness distrkurtosis
-0.8
ela distr.numberrrr..m.. of peakks
F1
F2
F4
F5
F6
F7 F8
F9 F10F11F122133114PSSI
NeurELA
Figure 11: Correlatinn score of NeurELA features and traditional ELA features
Based on the obtained trajectories, we use the pre-trained NeurELA model (outputs 16 features) and the traditional ELA (we choose 32 ELA features from the traditional ELA including the Meta- model group, Convexity group, Level-Set group, Local landscape group and Distribution group) to calculate landscape features for each optimization step. After the computation, obtain 720 landscape features time series for NeurELA and traditional ELA respectively.
3. For each pair of landscape features time series, measure the relationship between the i-th feature in NeurELA features and the j-th feature traditional ELA features by computing the Pearson Correlation Coefficient rt.j of the time series of these features: (F NerrerELL ELA and POELA...ELLA (P j,500
4. We obtain the final correlation scores of each feature pair between NeurELA and traditional ELA by averaging Vs of this feature pair over the 720 time series data, i [1, 2,..,... 16). Finally obtain correlation matrix with 32 rows and 16 columns. We illustrate this correlation matrix by the heatmap in Figure [I The x-axis denote 16 NeurELA features and y-axis denote 32 ELA features, larger value denotes the two features closely related.
21,
From the correlation results in that Figure, we could find some relationship patterns between our NeurELA features and the traditional ELA features: a) four NeurELA features F1, F4, F8 and F16 are novel features learned by NeurELA which show weak correlation (< 0.6) with all ELA features. b) some NeurEL features show strong correlation with one particular feature group in traditional
20

---

Published as a conference paper at ICLR 2025
ELA, such as F3 with the Meta-model group. c) some NeurELA features show strong correlation with multiple feature groups in traditional ELA, such as F10 with Distribution group and Meta-model group. d) all NeurELA features show weak correlation with the Convexity group and Local landscape group, which might reveals these two group features are less useful for addressing MetaBBO tasks.
21


Reviews:
[{"review_id": "W3ffbexefK", "reviewer": "Reviewer_RDeR", "summary": "This paper proposes an automatic construction method of landscape features for meta black-box optimization (MetaBBO). The proposed approach, termed neural exploratory landscape analysis (NeurELA), trains the attention-based neural network extracting landscape features to improve the MetaBBO algorithms. While existing MetaBBO methods rely on human-crafted landscape features, the proposed NeurELA can automate the design process for landscape features in MetaBBO. The experimental result demonstrates that the proposed NeurELA outperforms existing MetaBBO methods on unseen MetaBBO tasks. In addition, the authors show that fine-tuning process can lead to further performance improvement.", "questions": "1. I suppose that the computational cost of the proposed approach is larger than the existing MetaBBO because it requires the training of a landscape feature extractor as the outer loop for MetaBBO. What is the exact computational time/cost for the proposed method compared to existing MetaBBO methods?\n1. If the baseline methods can use the same computational budgets, how is the performance gain of the proposed approach? For instance, extra budgets may be used to optimize hyperparameters or to select traditional ELA features in the baseline MetaBBO.\n1. Training the landscape feature extractor in the current setting seems challenging because it should optimize the high-dimensional parameters by the evolution strategy. Is there any empirical evidence for the convergence of ES as the outer loop optimizer?\n1. What kind of BBO or MetaBBO algorithms can be used with the proposed NeurELA? The authors might assume the population-based BBO algorithms or evolutionary algorithms. Is it possible to combine the NeurELA with other kinds of BBO methods?\n1. What is the exact number of dimensions of the NeurELA features in the experiment? What is the impact of the dimensionality of the EeurELA features on the performance?\n1. I could not find the definition of the performance metric in the experimental evaluation, e.g., in Figure 3. Could you provide a detailed definition of the performance metric in the experimental results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The motivation for optimizing the feature extractor in MetaBBO is reasonable.\n- The technical novelty of this paper is to present automating approach for extracting landscape features in MetaBBO.\n- The effectiveness of the proposed NeurELA is experimentally demonstrated for several MetaBBO and BBO problems.", "weaknesses": "- The proposed formulation seems to be a tri-level optimization problem of training landscape feature extractor, training meta-level policy, and optimizing a target objective function. Therefore, using the proposed NeurELA increases the whole computational cost compared to existing MetaBBO methods.\n- Training the landscape feature extractor is performed in a neuroevolution manner. It seems hard to scale for a large neural network as the feature extractor. In addition, it is not clear that the current setting, i.e., optimizing 3,296 parameters for 500 evaluations by the evolution strategy, is sufficient for convergence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UHDHcPPkRK", "reviewer": "Reviewer_PkFr", "summary": "The paper introduces Neural Exploratory Landscape Analysis (NeurELA), a novel framework designed to improve Meta-Black-Box Optimization (MetaBBO) by dynamically profiling optimization landscape features through a two-stage, attention-based neural network. Unlike traditional approaches that rely on human-crafted features, NeurELA learns these features automatically in an end-to-end manner. This is aimed at overcoming the limitations of existing Exploratory Landscape Analysis (ELA) methods, such as computational overhead and reliance on expert knowledge. The authors propose a novel neural-network based landscape analyzer, and propose a two-stage attention mechanism: this architecture facilitates robust feature extraction, capable of generalizing to various MetaBBO algorithms without manual adjustments. The novel framework is tested in three MetaBBO tasks against several baselines and ablations.", "questions": "See weaknesses..", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The new end-to-end pipeline is (imho) novel and interesting\n- NeurELA consistently outperforms the baselines\n- Many ablations are performed and thus we better understand the effectiveness of the approach and its parts\n- The sharing of the code is appreciated", "weaknesses": "**Weaknesses before author's rebuttal. I believe that most of my comments have been (at least partly) by the new version/reply by the authors.**\n\n- The presentation of the paper needs quite some work. Many typos are present and a few paragraphs are hard to read. Overall, the authors need to spend a bit more time in improving the presentation.\n- I believe that a more detailed description of the MetaBBO tasks would greatly help the reader understand and appreciate the performance of the proposed framework.\n- In Section 4.2 and Fig. 4 it is not clear to what \"one epoch\" corresponds to. For example, I do not understand how we can compare \"ZeroShot\" with \"Finetune\" in the same scale. If I understand correctly, the \"ZeroShot\" variant just uses the learned $\\Lambda_{\\theta}$ to select at each generation of the low-level optimizer the appropriate feature vector and configuration. On the contrary, the \"Finetune\" variant should run the whole meta-learning pipeline. Thus, how can we compare \"epochs\" of one to the other? We need more explanation here to appreciate the results.\n- The computational overhead of NeurELA framework is quite big. Also compared to the original variant ($\\Lambda_0$) the gains are not big even when a big number of samples ($m$) is used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E179ySMGpf", "reviewer": "Reviewer_ARe5", "summary": "The paper introduces Neural Exploratory Landscape Analysis (NeurELA), a framework for Meta-Black-Box Optimization (MetaBBO) that replaces traditional, human-crafted Exploratory Landscape Analysis (ELA) features with a fully neural network-based approach. NeurELA employs a two-stage, attention-based neural network trained via a neuroevolution to dynamically profile optimization landscapes, adapting its features in real time. The authors show that NeurELA enhances performance across various MetaBBO algorithms and generalizes effectively to unseen tasks for zero-shot generalization and fine-tuning.", "questions": "1. What's the major difference between the proposed method and Deep-ELA? A more detailed explanation and corresponding experiments and/or ablation studies will help better support the paper's novelty and contribution.\n\n2. Line 479: \"This is primarily owning to the two-stage attention architecture in our NeurELA, which can be efficiently computed on GPUs in parallel.\" Was the training done on GPU? It was mentioned in line 356 that the training uses a Slurm CPU cluster. \n\n3. The wall time is too short to really tell a difference between algorithms since programming/system processing could have a larger effect if it's in milliseconds. Please consider using a more complicate benchmark or increasing the dimensions.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. Quality: The paper validates the proposed method in detail by answering research questions, not only on the method's performance but also its adaptability, computational efficiency, and generalization capacity. \n\n2. Clarity: The paper is well-structured, with clear explanations of NeurELA’s architecture, training, and integration within MetaBBO tasks.", "weaknesses": "1. Originality: The proposed work is very similar to Seiler et al., 2024 (Deep-ELA), which also uses multi-head attention as the main component in the architecture. The only difference seems to be that Deep-ELA uses kNN embedding, while the proposed method uses a linear transformation to encode the population information, which is widely used in LLMs to generate embedding from tokens. \n\n2. Limited comparisons in experiments: The proposed work does not compare to any recent methods, e.g., Deep-ELA. \n\n3. Limited tasks: Although NeurELA is tested across a variety of MetaBBO algorithms and optimization problems, the experiments lack a detailed analysis of its performance in higher-dimensional optimization scenarios, where many MetaBBO algorithms struggle.\n\n4. Interpretability and feature analysis: Although NeurELA shows promise in dynamically adapting landscape features, there is limited discussion on the interpretability of these features in relation to traditional ELA metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U2JTMknaUf", "reviewer": "Reviewer_C3y6", "summary": "The paper proposes Neural Exploratory Landscape Analysis (NeurELA), a framework to replace hand-crafted landscape analysis features in Meta Black-Box Optimization (MetaBBO) with a learned neural network approach.\n\nThe key contributions are:\n- making the MetaBBO paradigm entirely end-to-end\n- A two-stage attention-based neural network architecture for dynamic landscape feature extraction\n- A multi-task training framework to learn from multiple MetaBBO algorithms simultaneously\n- Demonstration of zero-shot generalization to unseen algorithms and problem sets\n- Ability to fine-tune the pre-trained analyzer for specific tasks", "questions": "- What is the minimum number of training tasks needed for good zero-shot generalization?\n- Could you provide more insight into what features the network learns compared to traditional ELA?\n\n### Nitpicking\n- L110: an universal neural landscape analyser... -> a universal\n- L176: (Prager et al., 2021b) first proposed... -> wrong citation type, should use \\citet instead of \\citep\n- L318: We hence introduce Ray Moritz et al. (2018), an open-source... -> wrong citation type + you don't introduce Ray, you \"employ\" Ray\n- L350: functiona... -> funtion\n- L367: Recall that an MetaBBO task is... -> that a MetaBBO", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The approach is well-motivated and builds on established MetaBBO literature\n- The experimental methodology is rigorous with proper baselines and ablations\n- Results demonstrate clear improvements over traditional ELA features\n- Comprehensive experiments across multiple MetaBBO algorithms\n- Testing on both synthetic benchmarks and real-world problems (protein docking)", "weaknesses": "- Could better analyze when/why zero-shot generalization fails\n- The authors acknowledge training efficiency issues with larger models\n- Lacks theoretical justification for why the two-stage attention architecture works well", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)